2022-05-18T04:12:11.2791111Z Requested labels: linux.8xlarge.nvidia.gpu
2022-05-18T04:12:11.2791194Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master
2022-05-18T04:12:11.2791217Z Waiting for a runner to pick up this job...
2022-05-18T04:14:10.2623731Z Job is about to start running on the runner: i-0e12f07c7a192d642 (repository)
2022-05-18T04:14:14.4762509Z Current runner version: '2.291.1'
2022-05-18T04:14:14.4770318Z Runner name: 'i-0e12f07c7a192d642'
2022-05-18T04:14:14.4771062Z Runner group name: 'Default'
2022-05-18T04:14:14.4771779Z Machine name: 'ip-10-0-4-221'
2022-05-18T04:14:14.4774720Z ##[group]GITHUB_TOKEN Permissions
2022-05-18T04:14:14.4775588Z Actions: write
2022-05-18T04:14:14.4776045Z Checks: write
2022-05-18T04:14:14.4776414Z Contents: write
2022-05-18T04:14:14.4776922Z Deployments: write
2022-05-18T04:14:14.4777361Z Discussions: write
2022-05-18T04:14:14.4777731Z Issues: write
2022-05-18T04:14:14.4778143Z Metadata: read
2022-05-18T04:14:14.4778607Z Packages: write
2022-05-18T04:14:14.4778979Z Pages: write
2022-05-18T04:14:14.4779421Z PullRequests: write
2022-05-18T04:14:14.4779934Z RepositoryProjects: write
2022-05-18T04:14:14.4780414Z SecurityEvents: write
2022-05-18T04:14:14.4780896Z Statuses: write
2022-05-18T04:14:14.4781325Z ##[endgroup]
2022-05-18T04:14:14.4785597Z Secret source: Actions
2022-05-18T04:14:14.4786417Z Prepare workflow directory
2022-05-18T04:14:14.6077600Z Prepare all required actions
2022-05-18T04:14:14.6298288Z Getting action download info
2022-05-18T04:14:14.8144180Z Download action repository 'pytorch/pytorch@master' (SHA:7b8cf1f7366bff95e9954037a58a8bb0edaaebd3)
2022-05-18T04:14:17.9403084Z Download action repository 'nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a' (SHA:71062288b76e2b6214ebde0e673ce0de1755740a)
2022-05-18T04:14:18.0459148Z Download action repository 'seemethere/upload-artifact-s3@v4' (SHA:c1c31f57581a11fe6d4d052da6276adb2df71f1e)
2022-05-18T04:14:18.3280521Z Getting action download info
2022-05-18T04:14:18.4638472Z Download action repository 'malfet/checkout@silent-checkout' (SHA:f63e9e15406be6060f159846cd2e098f759c5246)
2022-05-18T04:14:18.6424327Z Getting action download info
2022-05-18T04:14:18.9406696Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-05-18T04:14:18.9407082Z with:
2022-05-18T04:14:18.9407330Z   submodules: recursive
2022-05-18T04:14:18.9407594Z   fetch-depth: 0
2022-05-18T04:14:18.9407835Z env:
2022-05-18T04:14:18.9408053Z   IN_CI: 1
2022-05-18T04:14:18.9408278Z   IS_GHA: 1
2022-05-18T04:14:18.9408499Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:14:18.9408773Z ##[endgroup]
2022-05-18T04:14:18.9715131Z ##[group]Run echo "${GITHUB_WORKSPACE}"
2022-05-18T04:14:18.9715505Z echo "${GITHUB_WORKSPACE}"
2022-05-18T04:14:18.9715811Z if [ -z "${NO_SUDO}" ]; then
2022-05-18T04:14:18.9716110Z   sudo rm -rf "${GITHUB_WORKSPACE}"
2022-05-18T04:14:18.9716361Z else
2022-05-18T04:14:18.9716615Z   rm -rf "${GITHUB_WORKSPACE}"
2022-05-18T04:14:18.9716864Z fi
2022-05-18T04:14:18.9717113Z mkdir "${GITHUB_WORKSPACE}"
2022-05-18T04:14:18.9735380Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:14:18.9735716Z env:
2022-05-18T04:14:18.9735925Z   IN_CI: 1
2022-05-18T04:14:18.9736147Z   IS_GHA: 1
2022-05-18T04:14:18.9736396Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:14:18.9736637Z   NO_SUDO: 
2022-05-18T04:14:18.9736867Z ##[endgroup]
2022-05-18T04:14:18.9966408Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T04:14:19.1062576Z ##[group]Run malfet/checkout@silent-checkout
2022-05-18T04:14:19.1062882Z with:
2022-05-18T04:14:19.1063156Z   ref: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T04:14:19.1063427Z   fetch-depth: 0
2022-05-18T04:14:19.1063689Z   submodules: recursive
2022-05-18T04:14:19.1063951Z   quiet-checkout: true
2022-05-18T04:14:19.1064211Z   repository: pytorch/pytorch
2022-05-18T04:14:19.1064667Z   token: ***
2022-05-18T04:14:19.1064915Z   ssh-strict: true
2022-05-18T04:14:19.1065182Z   persist-credentials: true
2022-05-18T04:14:19.1065428Z   clean: true
2022-05-18T04:14:19.1065656Z   lfs: false
2022-05-18T04:14:19.1065916Z   set-safe-directory: true
2022-05-18T04:14:19.1066149Z env:
2022-05-18T04:14:19.1066359Z   IN_CI: 1
2022-05-18T04:14:19.1066578Z   IS_GHA: 1
2022-05-18T04:14:19.1066807Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:14:19.1067224Z ##[endgroup]
2022-05-18T04:14:19.2595701Z Syncing repository: pytorch/pytorch
2022-05-18T04:14:19.2597608Z ##[group]Getting Git version info
2022-05-18T04:14:19.2598155Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-05-18T04:14:19.2598744Z [command]/usr/bin/git version
2022-05-18T04:14:19.2599018Z git version 2.32.0
2022-05-18T04:14:19.2601707Z ##[endgroup]
2022-05-18T04:14:19.2624064Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/522b7a35-06c3-4c5e-b114-9f4e60f1517c' before making global git config changes
2022-05-18T04:14:19.2624966Z Adding repository directory to the temporary git global config as a safe directory
2022-05-18T04:14:19.2632609Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T04:14:19.2675055Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-05-18T04:14:19.2680483Z ##[group]Initializing the repository
2022-05-18T04:14:19.2687093Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T04:14:19.2719703Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-05-18T04:14:19.2720678Z hint: is subject to change. To configure the initial branch name to use in all
2022-05-18T04:14:19.2721351Z hint: of your new repositories, which will suppress this warning, call:
2022-05-18T04:14:19.2721651Z hint: 
2022-05-18T04:14:19.2722027Z hint: 	git config --global init.defaultBranch <name>
2022-05-18T04:14:19.2722320Z hint: 
2022-05-18T04:14:19.2722897Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-05-18T04:14:19.2724028Z hint: 'development'. The just-created branch can be renamed via this command:
2022-05-18T04:14:19.2724399Z hint: 
2022-05-18T04:14:19.2724831Z hint: 	git branch -m <name>
2022-05-18T04:14:19.2725382Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-05-18T04:14:19.2735377Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-05-18T04:14:19.2769035Z ##[endgroup]
2022-05-18T04:14:19.2770264Z ##[group]Disabling automatic garbage collection
2022-05-18T04:14:19.2775519Z [command]/usr/bin/git config --local gc.auto 0
2022-05-18T04:14:19.2808351Z ##[endgroup]
2022-05-18T04:14:19.2809233Z ##[group]Setting up auth
2022-05-18T04:14:19.2819547Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-05-18T04:14:19.2856284Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-05-18T04:14:19.3293219Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-05-18T04:14:19.3325714Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-05-18T04:14:19.3627246Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-05-18T04:14:19.3672282Z ##[endgroup]
2022-05-18T04:14:19.3673205Z ##[group]Fetching the repository
2022-05-18T04:14:19.3681796Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-05-18T04:15:00.7606785Z [command]/usr/bin/git rev-parse --verify --quiet 3b2375291aab7b48442f2e6fb1ef66cebc761e24^{object}
2022-05-18T04:15:00.7635512Z 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T04:15:00.7643403Z ##[endgroup]
2022-05-18T04:15:00.7643920Z ##[group]Determining the checkout info
2022-05-18T04:15:00.7644372Z ##[endgroup]
2022-05-18T04:15:00.7644819Z ##[group]Checking out the ref
2022-05-18T04:15:00.7650002Z [command]/usr/bin/git checkout --quiet --force 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T04:15:02.3364717Z ##[endgroup]
2022-05-18T04:15:02.3365506Z ##[group]Setting up auth for fetching submodules
2022-05-18T04:15:02.3373306Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-05-18T04:15:02.3427597Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-05-18T04:15:02.3461438Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-05-18T04:15:02.3494365Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-05-18T04:15:02.3525103Z ##[endgroup]
2022-05-18T04:15:02.3525562Z ##[group]Fetching submodules
2022-05-18T04:15:02.3532331Z [command]/usr/bin/git submodule sync --recursive
2022-05-18T04:15:02.3863740Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-05-18T04:15:02.4174947Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-05-18T04:15:02.4178198Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-05-18T04:15:02.4181895Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-05-18T04:15:02.4185902Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-05-18T04:15:02.4189952Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-05-18T04:15:02.4194062Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-05-18T04:15:02.4198498Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-05-18T04:15:02.4202824Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-05-18T04:15:02.4207269Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-05-18T04:15:02.4212722Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-05-18T04:15:02.4217226Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-05-18T04:15:02.4222132Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-05-18T04:15:02.4227489Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-05-18T04:15:02.4232719Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-05-18T04:15:02.4239154Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-05-18T04:15:02.4245365Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:15:02.4251670Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-05-18T04:15:02.4259408Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-05-18T04:15:02.4265794Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-05-18T04:15:02.4272367Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-05-18T04:15:02.4279330Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-05-18T04:15:02.4286628Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-05-18T04:15:02.4294243Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-05-18T04:15:02.4301209Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-05-18T04:15:02.4308486Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-05-18T04:15:02.4315813Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-05-18T04:15:02.4323514Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-05-18T04:15:02.4331832Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-05-18T04:15:02.4339673Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-05-18T04:15:02.4347664Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-05-18T04:15:02.4355904Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-05-18T04:15:02.4364287Z Submodule 'third_party/python-peachpy' (https://github.com/Maratyszcza/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-05-18T04:15:02.4373855Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-05-18T04:15:02.4382315Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-05-18T04:15:02.4391056Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-05-18T04:15:02.4399976Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-05-18T04:15:02.4409163Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-05-18T04:15:02.4474698Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-05-18T04:15:02.7587775Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-05-18T04:15:03.0348235Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-05-18T04:15:03.2524780Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-05-18T04:15:03.5060578Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-05-18T04:15:03.7865043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-05-18T04:15:07.3438633Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-05-18T04:15:07.6781323Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-05-18T04:15:08.1376708Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-05-18T04:15:09.2802600Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-05-18T04:15:10.4603894Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-05-18T04:15:15.0241532Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-05-18T04:15:15.6160427Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-05-18T04:15:16.5472364Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-05-18T04:15:17.4244808Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-05-18T04:15:17.6483922Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-05-18T04:15:18.0837111Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-05-18T04:15:18.3523215Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-05-18T04:15:19.1742721Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-05-18T04:15:19.4822788Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-05-18T04:15:19.7626154Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-05-18T04:15:21.2375416Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-05-18T04:15:21.5796289Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-05-18T04:15:21.9140272Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-05-18T04:15:23.0100854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-05-18T04:15:23.3567493Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-05-18T04:15:23.5628358Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-05-18T04:15:27.7913323Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-05-18T04:15:27.9984115Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-05-18T04:15:28.1989327Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-05-18T04:15:28.8476420Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-05-18T04:15:29.2837364Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-05-18T04:15:29.5326843Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-05-18T04:15:29.7909766Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-05-18T04:15:30.4644537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-05-18T04:15:32.2288866Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-05-18T04:15:32.6513362Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-05-18T04:15:34.3553089Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-05-18T04:15:34.3934771Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-05-18T04:15:34.4285667Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-05-18T04:15:34.4831962Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-05-18T04:15:34.5367169Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-05-18T04:15:35.3269575Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-05-18T04:15:35.3798104Z Submodule path 'third_party/benchmark': checked out 'e991355c02b93fe17713efe04cbc2e278e00fdbd'
2022-05-18T04:15:35.5282219Z Submodule path 'third_party/cpuinfo': checked out '5916273f79a21551890fd3d56fc5375a78d1598d'
2022-05-18T04:15:35.5950852Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-05-18T04:15:36.0214990Z Submodule path 'third_party/cudnn_frontend': checked out '43709ab96c47e26eebcdac72f93f946d44ceffa8'
2022-05-18T04:15:36.3471984Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-05-18T04:15:36.4275289Z Submodule path 'third_party/fbgemm': checked out '2e9be65810107a9595da717f95d21924b73be833'
2022-05-18T04:15:36.4326400Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:15:36.4329338Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:15:36.4333432Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:15:36.4378924Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-05-18T04:15:37.0668726Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-05-18T04:15:37.5266122Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-05-18T04:15:38.4252548Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out '8b35b4cffb62ecb58a903bf91cb7537d7a672211'
2022-05-18T04:15:38.5741390Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-05-18T04:15:38.6685212Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-05-18T04:15:38.8004070Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-05-18T04:15:38.8670504Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2022-05-18T04:15:38.9026660Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-05-18T04:15:38.9744268Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-05-18T04:15:39.0284007Z Submodule path 'third_party/gloo': checked out 'c22a5cfba94edf8ea4f53a174d38aa0c629d070f'
2022-05-18T04:15:39.1101617Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-05-18T04:15:39.1479921Z Submodule path 'third_party/ideep': checked out '02b17c5748c9349dcc586c359af800c684d9b1ab'
2022-05-18T04:15:39.1529960Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-05-18T04:15:39.1573951Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-05-18T04:15:44.6655861Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46'
2022-05-18T04:15:44.6719733Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:15:44.6768104Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-05-18T04:15:49.7848859Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815'
2022-05-18T04:15:49.8254676Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-05-18T04:15:49.9639729Z Submodule path 'third_party/kineto': checked out 'b2b48c00c6e5bd8e807e2231adb229db6a1d1c22'
2022-05-18T04:15:49.9690712Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:15:49.9694097Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:15:49.9739737Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-05-18T04:15:50.8815163Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-05-18T04:15:51.7750296Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-05-18T04:15:51.8636583Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-05-18T04:15:51.9127451Z Submodule path 'third_party/nccl/nccl': checked out '7e515921295adaab72adf56ea71a0fafb0ecb5f3'
2022-05-18T04:15:51.9542339Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-05-18T04:15:52.2765574Z Submodule path 'third_party/onnx': checked out '96046b8ccfb8e6fa82f6b2b34b3d56add2e8849c'
2022-05-18T04:15:52.2832180Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-05-18T04:15:52.2836001Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-05-18T04:15:52.2894669Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-05-18T04:15:52.6425083Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-05-18T04:15:53.4503462Z Submodule path 'third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-05-18T04:15:53.5137402Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '59a2ac2745d8a57ac94c6accced73620d59fb844'
2022-05-18T04:15:53.5571864Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-05-18T04:15:53.5621209Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:15:53.5663454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-05-18T04:15:54.8985007Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-05-18T04:15:54.9050230Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:15:54.9053157Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:15:54.9105727Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-05-18T04:15:55.3736820Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-05-18T04:15:56.0723482Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-05-18T04:15:56.1737815Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-05-18T04:15:56.1795356Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:15:56.1839867Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-05-18T04:15:56.4015333Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-05-18T04:15:56.4393033Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-05-18T04:15:56.7811110Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-05-18T04:15:56.7859748Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:15:56.7862632Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-05-18T04:15:56.7913408Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-05-18T04:15:57.1191084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-05-18T04:15:57.9959193Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-05-18T04:15:58.1025445Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-05-18T04:15:58.1393694Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-05-18T04:15:58.1774252Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-05-18T04:15:58.2384197Z Submodule path 'third_party/pybind11': checked out '8de7772cc72daca8e947b79b83fea46214931604'
2022-05-18T04:15:58.2734586Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-05-18T04:15:58.3331365Z Submodule path 'third_party/python-peachpy': checked out '07d8fde8ac45d7705129475c0f94ed8925b93473'
2022-05-18T04:15:58.3690713Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-05-18T04:15:58.4472852Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-05-18T04:15:58.6062862Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-05-18T04:15:58.6637335Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-05-18T04:15:58.6687150Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:15:58.6690596Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:15:58.6694022Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:15:58.6697151Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:15:58.6742053Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-05-18T04:15:59.4631669Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-05-18T04:15:59.7093096Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-05-18T04:16:00.6872885Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-05-18T04:16:01.4455434Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-05-18T04:16:01.4901392Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-05-18T04:16:01.5952930Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-05-18T04:16:01.6544441Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-05-18T04:16:01.6601879Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:16:01.6647026Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-05-18T04:16:01.9197805Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-05-18T04:16:02.1034145Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-05-18T04:16:02.1123925Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-05-18T04:16:02.1447078Z Entering 'android/libs/fbjni'
2022-05-18T04:16:02.1488098Z Entering 'third_party/FP16'
2022-05-18T04:16:02.1529419Z Entering 'third_party/FXdiv'
2022-05-18T04:16:02.1571646Z Entering 'third_party/NNPACK'
2022-05-18T04:16:02.1614246Z Entering 'third_party/QNNPACK'
2022-05-18T04:16:02.1655496Z Entering 'third_party/XNNPACK'
2022-05-18T04:16:02.1708471Z Entering 'third_party/benchmark'
2022-05-18T04:16:02.1750273Z Entering 'third_party/cpuinfo'
2022-05-18T04:16:02.1792459Z Entering 'third_party/cub'
2022-05-18T04:16:02.1834297Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:16:02.1881882Z Entering 'third_party/eigen'
2022-05-18T04:16:02.1925145Z Entering 'third_party/fbgemm'
2022-05-18T04:16:02.1966155Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:16:02.2007164Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:16:02.2048336Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:16:02.2091909Z Entering 'third_party/flatbuffers'
2022-05-18T04:16:02.2136545Z Entering 'third_party/fmt'
2022-05-18T04:16:02.2178799Z Entering 'third_party/foxi'
2022-05-18T04:16:02.2219258Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:16:02.2260752Z Entering 'third_party/gloo'
2022-05-18T04:16:02.2302124Z Entering 'third_party/googletest'
2022-05-18T04:16:02.2342468Z Entering 'third_party/ideep'
2022-05-18T04:16:02.2382383Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:16:02.2425032Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:16:02.2471214Z Entering 'third_party/ios-cmake'
2022-05-18T04:16:02.2513105Z Entering 'third_party/kineto'
2022-05-18T04:16:02.2553420Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:16:02.2594210Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:16:02.2637252Z Entering 'third_party/nccl/nccl'
2022-05-18T04:16:02.2677738Z Entering 'third_party/neon2sse'
2022-05-18T04:16:02.2717637Z Entering 'third_party/onnx'
2022-05-18T04:16:02.2770114Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:16:02.2810306Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:16:02.2853331Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:16:02.2895325Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:16:02.2940616Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:16:02.2981247Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:16:02.3020943Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:16:02.3067218Z Entering 'third_party/pocketfft'
2022-05-18T04:16:02.3107997Z Entering 'third_party/protobuf'
2022-05-18T04:16:02.3153815Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:16:02.3194678Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:16:02.3236758Z Entering 'third_party/psimd'
2022-05-18T04:16:02.3277279Z Entering 'third_party/pthreadpool'
2022-05-18T04:16:02.3319434Z Entering 'third_party/pybind11'
2022-05-18T04:16:02.3360826Z Entering 'third_party/python-enum'
2022-05-18T04:16:02.3400506Z Entering 'third_party/python-peachpy'
2022-05-18T04:16:02.3440862Z Entering 'third_party/python-six'
2022-05-18T04:16:02.3481069Z Entering 'third_party/sleef'
2022-05-18T04:16:02.3522905Z Entering 'third_party/tbb'
2022-05-18T04:16:02.3566319Z Entering 'third_party/tensorpipe'
2022-05-18T04:16:02.3607927Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:16:02.3647814Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:16:02.3689299Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:16:02.3731434Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:16:02.3771913Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:16:02.3815910Z Entering 'third_party/zstd'
2022-05-18T04:16:02.3867533Z ##[endgroup]
2022-05-18T04:16:02.3871543Z ##[group]Persisting credentials for submodules
2022-05-18T04:16:02.3876934Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-05-18T04:16:02.4196865Z Entering 'android/libs/fbjni'
2022-05-18T04:16:02.4239989Z Entering 'third_party/FP16'
2022-05-18T04:16:02.4280369Z Entering 'third_party/FXdiv'
2022-05-18T04:16:02.4321155Z Entering 'third_party/NNPACK'
2022-05-18T04:16:02.4361974Z Entering 'third_party/QNNPACK'
2022-05-18T04:16:02.4404249Z Entering 'third_party/XNNPACK'
2022-05-18T04:16:02.4456125Z Entering 'third_party/benchmark'
2022-05-18T04:16:02.4496863Z Entering 'third_party/cpuinfo'
2022-05-18T04:16:02.4538304Z Entering 'third_party/cub'
2022-05-18T04:16:02.4580032Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:16:02.4624961Z Entering 'third_party/eigen'
2022-05-18T04:16:02.4667073Z Entering 'third_party/fbgemm'
2022-05-18T04:16:02.4707264Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:16:02.4746915Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:16:02.4787612Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:16:02.4828831Z Entering 'third_party/flatbuffers'
2022-05-18T04:16:02.4871414Z Entering 'third_party/fmt'
2022-05-18T04:16:02.4912223Z Entering 'third_party/foxi'
2022-05-18T04:16:02.4952027Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:16:02.4992535Z Entering 'third_party/gloo'
2022-05-18T04:16:02.5034216Z Entering 'third_party/googletest'
2022-05-18T04:16:02.5074893Z Entering 'third_party/ideep'
2022-05-18T04:16:02.5114451Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:16:02.5156503Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:16:02.5203431Z Entering 'third_party/ios-cmake'
2022-05-18T04:16:02.5243873Z Entering 'third_party/kineto'
2022-05-18T04:16:02.5284937Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:16:02.5325455Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:16:02.5367363Z Entering 'third_party/nccl/nccl'
2022-05-18T04:16:02.5408349Z Entering 'third_party/neon2sse'
2022-05-18T04:16:02.5448395Z Entering 'third_party/onnx'
2022-05-18T04:16:02.5500623Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:16:02.5540399Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:16:02.5582696Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:16:02.5622709Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:16:02.5667427Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:16:02.5707923Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:16:02.5749924Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:16:02.5795069Z Entering 'third_party/pocketfft'
2022-05-18T04:16:02.5835574Z Entering 'third_party/protobuf'
2022-05-18T04:16:02.5880401Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:16:02.5919974Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:16:02.5963591Z Entering 'third_party/psimd'
2022-05-18T04:16:02.6004475Z Entering 'third_party/pthreadpool'
2022-05-18T04:16:02.6044550Z Entering 'third_party/pybind11'
2022-05-18T04:16:02.6085695Z Entering 'third_party/python-enum'
2022-05-18T04:16:02.6125388Z Entering 'third_party/python-peachpy'
2022-05-18T04:16:02.6166013Z Entering 'third_party/python-six'
2022-05-18T04:16:02.6207188Z Entering 'third_party/sleef'
2022-05-18T04:16:02.6247849Z Entering 'third_party/tbb'
2022-05-18T04:16:02.6292215Z Entering 'third_party/tensorpipe'
2022-05-18T04:16:02.6332505Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:16:02.6372788Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:16:02.6413241Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:16:02.6453624Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:16:02.6492788Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:16:02.6535708Z Entering 'third_party/zstd'
2022-05-18T04:16:02.6590600Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-05-18T04:16:02.6902594Z Entering 'android/libs/fbjni'
2022-05-18T04:16:02.6939329Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-05-18T04:16:02.6956721Z Entering 'third_party/FP16'
2022-05-18T04:16:02.6994979Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-05-18T04:16:02.7011663Z Entering 'third_party/FXdiv'
2022-05-18T04:16:02.7048819Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-05-18T04:16:02.7066009Z Entering 'third_party/NNPACK'
2022-05-18T04:16:02.7105523Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-05-18T04:16:02.7121840Z Entering 'third_party/QNNPACK'
2022-05-18T04:16:02.7159263Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-05-18T04:16:02.7176295Z Entering 'third_party/XNNPACK'
2022-05-18T04:16:02.7214724Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-05-18T04:16:02.7241727Z Entering 'third_party/benchmark'
2022-05-18T04:16:02.7281436Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T04:16:02.7298291Z Entering 'third_party/cpuinfo'
2022-05-18T04:16:02.7337213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-05-18T04:16:02.7353930Z Entering 'third_party/cub'
2022-05-18T04:16:02.7392407Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-05-18T04:16:02.7409268Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:16:02.7446861Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-05-18T04:16:02.7469296Z Entering 'third_party/eigen'
2022-05-18T04:16:02.7507328Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-05-18T04:16:02.7526361Z Entering 'third_party/fbgemm'
2022-05-18T04:16:02.7565167Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-05-18T04:16:02.7581988Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:16:02.7619849Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-05-18T04:16:02.7636624Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:16:02.7674793Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-05-18T04:16:02.7692236Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:16:02.7729718Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-05-18T04:16:02.7747875Z Entering 'third_party/flatbuffers'
2022-05-18T04:16:02.7786041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-05-18T04:16:02.7805081Z Entering 'third_party/fmt'
2022-05-18T04:16:02.7842653Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-05-18T04:16:02.7860771Z Entering 'third_party/foxi'
2022-05-18T04:16:02.7898843Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-05-18T04:16:02.7915808Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:16:02.7953036Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-05-18T04:16:02.7969811Z Entering 'third_party/gloo'
2022-05-18T04:16:02.8007363Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-05-18T04:16:02.8024096Z Entering 'third_party/googletest'
2022-05-18T04:16:02.8061873Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-05-18T04:16:02.8078743Z Entering 'third_party/ideep'
2022-05-18T04:16:02.8116883Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-05-18T04:16:02.8134371Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:16:02.8171816Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-05-18T04:16:02.8190404Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:16:02.8228148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-05-18T04:16:02.8251221Z Entering 'third_party/ios-cmake'
2022-05-18T04:16:02.8288691Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-05-18T04:16:02.8305848Z Entering 'third_party/kineto'
2022-05-18T04:16:02.8343408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-05-18T04:16:02.8360002Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:16:02.8397567Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-05-18T04:16:02.8414853Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:16:02.8454338Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-05-18T04:16:02.8472380Z Entering 'third_party/nccl/nccl'
2022-05-18T04:16:02.8511298Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-05-18T04:16:02.8527873Z Entering 'third_party/neon2sse'
2022-05-18T04:16:02.8564817Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-05-18T04:16:02.8582665Z Entering 'third_party/onnx'
2022-05-18T04:16:02.8620599Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-05-18T04:16:02.8648005Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:16:02.8685983Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T04:16:02.8703797Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:16:02.8741748Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T04:16:02.8760383Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:16:02.8798315Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-05-18T04:16:02.8814714Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:16:02.8852626Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-05-18T04:16:02.8874137Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:16:02.8912017Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T04:16:02.8928981Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:16:02.8966933Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T04:16:02.8983415Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:16:02.9021942Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-05-18T04:16:02.9043063Z Entering 'third_party/pocketfft'
2022-05-18T04:16:02.9081349Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-05-18T04:16:02.9098123Z Entering 'third_party/protobuf'
2022-05-18T04:16:02.9135215Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-05-18T04:16:02.9155134Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:16:02.9193061Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T04:16:02.9209578Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:16:02.9246946Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-05-18T04:16:02.9265707Z Entering 'third_party/psimd'
2022-05-18T04:16:02.9303113Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-05-18T04:16:02.9319287Z Entering 'third_party/pthreadpool'
2022-05-18T04:16:02.9356380Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-05-18T04:16:02.9373773Z Entering 'third_party/pybind11'
2022-05-18T04:16:02.9410513Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T04:16:02.9427606Z Entering 'third_party/python-enum'
2022-05-18T04:16:02.9465560Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-05-18T04:16:02.9482081Z Entering 'third_party/python-peachpy'
2022-05-18T04:16:02.9519478Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-05-18T04:16:02.9537436Z Entering 'third_party/python-six'
2022-05-18T04:16:02.9575329Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-05-18T04:16:02.9591586Z Entering 'third_party/sleef'
2022-05-18T04:16:02.9629416Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-05-18T04:16:02.9645759Z Entering 'third_party/tbb'
2022-05-18T04:16:02.9683238Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-05-18T04:16:02.9703435Z Entering 'third_party/tensorpipe'
2022-05-18T04:16:02.9740671Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-05-18T04:16:02.9757145Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:16:02.9794290Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-05-18T04:16:02.9811136Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:16:02.9847638Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-05-18T04:16:02.9865124Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:16:02.9901941Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-05-18T04:16:02.9918661Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:16:02.9956448Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T04:16:02.9972732Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:16:03.0009646Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-05-18T04:16:03.0029826Z Entering 'third_party/zstd'
2022-05-18T04:16:03.0068207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-05-18T04:16:03.0903406Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-05-18T04:16:03.1217459Z Entering 'android/libs/fbjni'
2022-05-18T04:16:03.1259696Z Entering 'third_party/FP16'
2022-05-18T04:16:03.1301388Z Entering 'third_party/FXdiv'
2022-05-18T04:16:03.1342177Z Entering 'third_party/NNPACK'
2022-05-18T04:16:03.1383968Z Entering 'third_party/QNNPACK'
2022-05-18T04:16:03.1426222Z Entering 'third_party/XNNPACK'
2022-05-18T04:16:03.1478854Z Entering 'third_party/benchmark'
2022-05-18T04:16:03.1520413Z Entering 'third_party/cpuinfo'
2022-05-18T04:16:03.1562443Z Entering 'third_party/cub'
2022-05-18T04:16:03.1603910Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:16:03.1650757Z Entering 'third_party/eigen'
2022-05-18T04:16:03.1694612Z Entering 'third_party/fbgemm'
2022-05-18T04:16:03.1736225Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:16:03.1776800Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:16:03.1818390Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:16:03.1860130Z Entering 'third_party/flatbuffers'
2022-05-18T04:16:03.1903556Z Entering 'third_party/fmt'
2022-05-18T04:16:03.1944724Z Entering 'third_party/foxi'
2022-05-18T04:16:03.1985580Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:16:03.2026689Z Entering 'third_party/gloo'
2022-05-18T04:16:03.2068191Z Entering 'third_party/googletest'
2022-05-18T04:16:03.2111425Z Entering 'third_party/ideep'
2022-05-18T04:16:03.2152030Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:16:03.2194689Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:16:03.2243450Z Entering 'third_party/ios-cmake'
2022-05-18T04:16:03.2284574Z Entering 'third_party/kineto'
2022-05-18T04:16:03.2327200Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:16:03.2367752Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:16:03.2411325Z Entering 'third_party/nccl/nccl'
2022-05-18T04:16:03.2452241Z Entering 'third_party/neon2sse'
2022-05-18T04:16:03.2494327Z Entering 'third_party/onnx'
2022-05-18T04:16:03.2547290Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:16:03.2589119Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:16:03.2633010Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:16:03.2673946Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:16:03.2720495Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:16:03.2762168Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:16:03.2804645Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:16:03.2851076Z Entering 'third_party/pocketfft'
2022-05-18T04:16:03.2893546Z Entering 'third_party/protobuf'
2022-05-18T04:16:03.2938339Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:16:03.2980650Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:16:03.3025370Z Entering 'third_party/psimd'
2022-05-18T04:16:03.3068862Z Entering 'third_party/pthreadpool'
2022-05-18T04:16:03.3112640Z Entering 'third_party/pybind11'
2022-05-18T04:16:03.3154999Z Entering 'third_party/python-enum'
2022-05-18T04:16:03.3196886Z Entering 'third_party/python-peachpy'
2022-05-18T04:16:03.3240663Z Entering 'third_party/python-six'
2022-05-18T04:16:03.3282406Z Entering 'third_party/sleef'
2022-05-18T04:16:03.3324348Z Entering 'third_party/tbb'
2022-05-18T04:16:03.3368657Z Entering 'third_party/tensorpipe'
2022-05-18T04:16:03.3411721Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:16:03.3454384Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:16:03.3496786Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:16:03.3538328Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:16:03.3579048Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:16:03.3623411Z Entering 'third_party/zstd'
2022-05-18T04:16:03.3680198Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-05-18T04:16:03.4002032Z Entering 'android/libs/fbjni'
2022-05-18T04:16:03.4044149Z Entering 'third_party/FP16'
2022-05-18T04:16:03.4086265Z Entering 'third_party/FXdiv'
2022-05-18T04:16:03.4128382Z Entering 'third_party/NNPACK'
2022-05-18T04:16:03.4170604Z Entering 'third_party/QNNPACK'
2022-05-18T04:16:03.4212522Z Entering 'third_party/XNNPACK'
2022-05-18T04:16:03.4265751Z Entering 'third_party/benchmark'
2022-05-18T04:16:03.4308257Z Entering 'third_party/cpuinfo'
2022-05-18T04:16:03.4350603Z Entering 'third_party/cub'
2022-05-18T04:16:03.4392335Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:16:03.4439429Z Entering 'third_party/eigen'
2022-05-18T04:16:03.4484018Z Entering 'third_party/fbgemm'
2022-05-18T04:16:03.4525406Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:16:03.4566979Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:16:03.4609273Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:16:03.4653079Z Entering 'third_party/flatbuffers'
2022-05-18T04:16:03.4696459Z Entering 'third_party/fmt'
2022-05-18T04:16:03.4737705Z Entering 'third_party/foxi'
2022-05-18T04:16:03.4780070Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:16:03.4822483Z Entering 'third_party/gloo'
2022-05-18T04:16:03.4863854Z Entering 'third_party/googletest'
2022-05-18T04:16:03.4905784Z Entering 'third_party/ideep'
2022-05-18T04:16:03.4946744Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:16:03.4990100Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:16:03.5038036Z Entering 'third_party/ios-cmake'
2022-05-18T04:16:03.5079937Z Entering 'third_party/kineto'
2022-05-18T04:16:03.5121902Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:16:03.5163444Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:16:03.5205772Z Entering 'third_party/nccl/nccl'
2022-05-18T04:16:03.5246703Z Entering 'third_party/neon2sse'
2022-05-18T04:16:03.5287790Z Entering 'third_party/onnx'
2022-05-18T04:16:03.5342166Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:16:03.5383511Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:16:03.5426594Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:16:03.5467402Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:16:03.5514130Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:16:03.5556516Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:16:03.5598107Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:16:03.5643963Z Entering 'third_party/pocketfft'
2022-05-18T04:16:03.5685113Z Entering 'third_party/protobuf'
2022-05-18T04:16:03.5729748Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:16:03.5770754Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:16:03.5814466Z Entering 'third_party/psimd'
2022-05-18T04:16:03.5855958Z Entering 'third_party/pthreadpool'
2022-05-18T04:16:03.5898181Z Entering 'third_party/pybind11'
2022-05-18T04:16:03.5940346Z Entering 'third_party/python-enum'
2022-05-18T04:16:03.5981377Z Entering 'third_party/python-peachpy'
2022-05-18T04:16:03.6023074Z Entering 'third_party/python-six'
2022-05-18T04:16:03.6064044Z Entering 'third_party/sleef'
2022-05-18T04:16:03.6105752Z Entering 'third_party/tbb'
2022-05-18T04:16:03.6148827Z Entering 'third_party/tensorpipe'
2022-05-18T04:16:03.6192097Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:16:03.6233171Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:16:03.6275860Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:16:03.6317236Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:16:03.6358076Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:16:03.6401605Z Entering 'third_party/zstd'
2022-05-18T04:16:03.6453227Z ##[endgroup]
2022-05-18T04:16:03.6499598Z [command]/usr/bin/git log -1 --format='%H'
2022-05-18T04:16:03.6526788Z '3b2375291aab7b48442f2e6fb1ef66cebc761e24'
2022-05-18T04:16:03.6672872Z Prepare all required actions
2022-05-18T04:16:03.6703795Z ##[group]Run ./.github/actions/setup-linux
2022-05-18T04:16:03.6704072Z env:
2022-05-18T04:16:03.6704293Z   IN_CI: 1
2022-05-18T04:16:03.6704501Z   IS_GHA: 1
2022-05-18T04:16:03.6704751Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:16:03.6705009Z ##[endgroup]
2022-05-18T04:16:03.6721533Z ##[group]Run set -euo pipefail
2022-05-18T04:16:03.6721846Z set -euo pipefail
2022-05-18T04:16:03.6722134Z function get_ec2_metadata() {
2022-05-18T04:16:03.6722459Z   # Pulled from instance metadata endpoint for EC2
2022-05-18T04:16:03.6722953Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-05-18T04:16:03.6723366Z   category=$1
2022-05-18T04:16:03.6723696Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-05-18T04:16:03.6723988Z }
2022-05-18T04:16:03.6724302Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-05-18T04:16:03.6724641Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-05-18T04:16:03.6725019Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-05-18T04:16:03.6725361Z echo "system info $(uname -a)"
2022-05-18T04:16:03.6738575Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:16:03.6738856Z env:
2022-05-18T04:16:03.6739070Z   IN_CI: 1
2022-05-18T04:16:03.6739288Z   IS_GHA: 1
2022-05-18T04:16:03.6739517Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:16:03.6739774Z ##[endgroup]
2022-05-18T04:16:03.6838985Z ami-id: ami-096198a0bccc6bad4
2022-05-18T04:16:03.6900485Z instance-id: i-0e12f07c7a192d642
2022-05-18T04:16:03.6963022Z instance-type: g3.8xlarge
2022-05-18T04:16:03.6971664Z system info Linux ip-10-0-4-221.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-05-18T04:16:03.6989599Z ##[group]Run if systemctl is-active --quiet docker; then
2022-05-18T04:16:03.6990005Z if systemctl is-active --quiet docker; then
2022-05-18T04:16:03.6990346Z     echo "Docker daemon is running...";
2022-05-18T04:16:03.6990603Z else
2022-05-18T04:16:03.6990920Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-05-18T04:16:03.6991229Z fi
2022-05-18T04:16:03.7002889Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:16:03.7003185Z env:
2022-05-18T04:16:03.7003406Z   IN_CI: 1
2022-05-18T04:16:03.7003612Z   IS_GHA: 1
2022-05-18T04:16:03.7003859Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:16:03.7004119Z ##[endgroup]
2022-05-18T04:16:03.7055324Z Docker daemon is running...
2022-05-18T04:16:03.7072430Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-05-18T04:16:03.7072897Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-05-18T04:16:03.7073279Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-05-18T04:16:03.7073833Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-05-18T04:16:03.7074284Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-05-18T04:16:03.7085715Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:16:03.7086009Z env:
2022-05-18T04:16:03.7086204Z   IN_CI: 1
2022-05-18T04:16:03.7086425Z   IS_GHA: 1
2022-05-18T04:16:03.7086669Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:16:03.7086916Z   AWS_RETRY_MODE: standard
2022-05-18T04:16:03.7087167Z   AWS_MAX_ATTEMPTS: 5
2022-05-18T04:16:03.7087432Z   AWS_DEFAULT_REGION: us-east-1
2022-05-18T04:16:03.7087681Z ##[endgroup]
2022-05-18T04:16:04.6488574Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-05-18T04:16:04.6489067Z Configure a credential helper to remove this warning. See
2022-05-18T04:16:04.6490159Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-05-18T04:16:04.6490459Z 
2022-05-18T04:16:04.6491133Z Login Succeeded
2022-05-18T04:16:04.6578987Z ##[group]Run env | grep '^GITHUB' > "/tmp/github_env_${GITHUB_RUN_ID}"
2022-05-18T04:16:04.6579456Z env | grep '^GITHUB' > "/tmp/github_env_${GITHUB_RUN_ID}"
2022-05-18T04:16:04.6592245Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:16:04.6592616Z env:
2022-05-18T04:16:04.6592906Z   IN_CI: 1
2022-05-18T04:16:04.6593161Z   IS_GHA: 1
2022-05-18T04:16:04.6593479Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:16:04.6593857Z ##[endgroup]
2022-05-18T04:16:04.6658323Z Prepare all required actions
2022-05-18T04:16:04.6658706Z Getting action download info
2022-05-18T04:16:04.8384237Z Download action repository 'seemethere/add-github-ssh-key@v1' (SHA:1ecffedb1e192a50aa67dba2f0e048e5d3bfa144)
2022-05-18T04:16:04.9680059Z ##[group]Run ./.github/actions/setup-ssh
2022-05-18T04:16:04.9680323Z with:
2022-05-18T04:16:04.9680773Z   github-secret: ***
2022-05-18T04:16:04.9680999Z env:
2022-05-18T04:16:04.9681209Z   IN_CI: 1
2022-05-18T04:16:04.9681425Z   IS_GHA: 1
2022-05-18T04:16:04.9681650Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:16:04.9681902Z ##[endgroup]
2022-05-18T04:16:04.9706142Z ##[group]Run seemethere/add-github-ssh-key@v1
2022-05-18T04:16:04.9706427Z with:
2022-05-18T04:16:04.9706807Z   GITHUB_TOKEN: ***
2022-05-18T04:16:04.9707073Z   activate-with-label: false
2022-05-18T04:16:04.9707317Z   label: with-ssh
2022-05-18T04:16:04.9707575Z   remove-existing-keys: true
2022-05-18T04:16:04.9707820Z env:
2022-05-18T04:16:04.9708009Z   IN_CI: 1
2022-05-18T04:16:04.9708259Z   IS_GHA: 1
2022-05-18T04:16:04.9708487Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:16:04.9708742Z ##[endgroup]
2022-05-18T04:16:05.0405172Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2022-05-18T04:16:05.0453605Z Prepare all required actions
2022-05-18T04:16:05.0473757Z ##[group]Run ./.github/actions/pull-docker-image
2022-05-18T04:16:05.0474042Z with:
2022-05-18T04:16:05.0474538Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:16:05.0474989Z env:
2022-05-18T04:16:05.0475203Z   IN_CI: 1
2022-05-18T04:16:05.0475424Z   IS_GHA: 1
2022-05-18T04:16:05.0475653Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:16:05.0475913Z ##[endgroup]
2022-05-18T04:16:05.0491272Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-05-18T04:16:05.0491619Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-05-18T04:16:05.0491933Z retry docker pull "${DOCKER_IMAGE}"
2022-05-18T04:16:05.0504003Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:16:05.0504283Z env:
2022-05-18T04:16:05.0504484Z   IN_CI: 1
2022-05-18T04:16:05.0504707Z   IS_GHA: 1
2022-05-18T04:16:05.0504940Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:16:05.0505438Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:16:05.0505909Z ##[endgroup]
2022-05-18T04:16:05.2843808Z 6deab82db6a72ca54cd3e3322ee4f13864536734: Pulling from pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7
2022-05-18T04:16:05.2844739Z 58690f9b18fc: Pulling fs layer
2022-05-18T04:16:05.2845330Z b51569e7c507: Pulling fs layer
2022-05-18T04:16:05.2845897Z da8ef40b9eca: Pulling fs layer
2022-05-18T04:16:05.2846439Z fb15d46c38dc: Pulling fs layer
2022-05-18T04:16:05.2847006Z e0d2c5aceba3: Pulling fs layer
2022-05-18T04:16:05.2847581Z 9c4425f4b8cb: Pulling fs layer
2022-05-18T04:16:05.2848120Z 3c5d24e8ef06: Pulling fs layer
2022-05-18T04:16:05.2848703Z 79c5859701ff: Pulling fs layer
2022-05-18T04:16:05.2849276Z d81828418d08: Pulling fs layer
2022-05-18T04:16:05.2850131Z f256bb6f705c: Pulling fs layer
2022-05-18T04:16:05.2850744Z cc2d8a95a2e5: Pulling fs layer
2022-05-18T04:16:05.2851179Z b9db730d0400: Pulling fs layer
2022-05-18T04:16:05.2851429Z 49f9027dc2e7: Pulling fs layer
2022-05-18T04:16:05.2851695Z d60308a752bd: Pulling fs layer
2022-05-18T04:16:05.2851958Z 624ec6d4936f: Pulling fs layer
2022-05-18T04:16:05.2852225Z b815c3dfeb5c: Pulling fs layer
2022-05-18T04:16:05.2852477Z 4a9f9b66af25: Pulling fs layer
2022-05-18T04:16:05.2852729Z 9c4425f4b8cb: Waiting
2022-05-18T04:16:05.2852971Z fb15d46c38dc: Waiting
2022-05-18T04:16:05.2853208Z b6d963fbdb11: Pulling fs layer
2022-05-18T04:16:05.2853462Z 3c5d24e8ef06: Waiting
2022-05-18T04:16:05.2853752Z 4486d2823377: Pulling fs layer
2022-05-18T04:16:05.2854016Z 68d34a18a767: Pulling fs layer
2022-05-18T04:16:05.2854264Z 1c478b5d7dcd: Pulling fs layer
2022-05-18T04:16:05.2854517Z f256bb6f705c: Waiting
2022-05-18T04:16:05.2854764Z 49f9027dc2e7: Waiting
2022-05-18T04:16:05.2855001Z 1d14eefa2afe: Pulling fs layer
2022-05-18T04:16:05.2855274Z cd1fd540bef8: Pulling fs layer
2022-05-18T04:16:05.2855546Z fdc2f33cd3f0: Pulling fs layer
2022-05-18T04:16:05.2855794Z 0626725f1e19: Pulling fs layer
2022-05-18T04:16:05.2856043Z cc2d8a95a2e5: Waiting
2022-05-18T04:16:05.2856282Z b9db730d0400: Waiting
2022-05-18T04:16:05.2856516Z 60b6e4baae49: Pulling fs layer
2022-05-18T04:16:05.2856764Z 79c5859701ff: Waiting
2022-05-18T04:16:05.2857012Z a9f25937ad89: Pulling fs layer
2022-05-18T04:16:05.2857242Z d81828418d08: Waiting
2022-05-18T04:16:05.2857471Z d60308a752bd: Waiting
2022-05-18T04:16:05.2857718Z 341c51541e6b: Pulling fs layer
2022-05-18T04:16:05.2857965Z 31a8b7b678c7: Pulling fs layer
2022-05-18T04:16:05.2858212Z b6d963fbdb11: Waiting
2022-05-18T04:16:05.2858451Z 624ec6d4936f: Waiting
2022-05-18T04:16:05.2858684Z ad8c1f2236e5: Pulling fs layer
2022-05-18T04:16:05.2858947Z f8f22be640a6: Pulling fs layer
2022-05-18T04:16:05.2859198Z 4486d2823377: Waiting
2022-05-18T04:16:05.2859419Z fdc2f33cd3f0: Waiting
2022-05-18T04:16:05.2859838Z 68d34a18a767: Waiting
2022-05-18T04:16:05.2860094Z 1c478b5d7dcd: Waiting
2022-05-18T04:16:05.2860312Z 60b6e4baae49: Waiting
2022-05-18T04:16:05.2860564Z 0b6a6636bca7: Pulling fs layer
2022-05-18T04:16:05.2860816Z a9f25937ad89: Waiting
2022-05-18T04:16:05.2861033Z ad8c1f2236e5: Waiting
2022-05-18T04:16:05.2861268Z 31a8b7b678c7: Waiting
2022-05-18T04:16:05.2861500Z 4a9f9b66af25: Waiting
2022-05-18T04:16:05.2861731Z 9c50e79f8e38: Pulling fs layer
2022-05-18T04:16:05.2861992Z 37c76e461e24: Pulling fs layer
2022-05-18T04:16:05.2862257Z 84c1af12bf7e: Pulling fs layer
2022-05-18T04:16:05.2862493Z 0b6a6636bca7: Waiting
2022-05-18T04:16:05.2862748Z 30d627f75fb9: Pulling fs layer
2022-05-18T04:16:05.2863002Z 9c50e79f8e38: Waiting
2022-05-18T04:16:05.2863236Z 0d7a717fbbe1: Pulling fs layer
2022-05-18T04:16:05.2863485Z 37c76e461e24: Waiting
2022-05-18T04:16:05.2863721Z cd1fd540bef8: Waiting
2022-05-18T04:16:05.2864069Z 4c42c8b107a9: Pulling fs layer
2022-05-18T04:16:05.2864303Z 84c1af12bf7e: Waiting
2022-05-18T04:16:05.2864546Z 0d7a717fbbe1: Waiting
2022-05-18T04:16:05.2864798Z dcb77576adf6: Pulling fs layer
2022-05-18T04:16:05.2865031Z 4c42c8b107a9: Waiting
2022-05-18T04:16:05.2865275Z 547da1897fee: Pulling fs layer
2022-05-18T04:16:05.2865540Z e31574ad02fc: Pulling fs layer
2022-05-18T04:16:05.2865791Z a9dad096f89d: Pulling fs layer
2022-05-18T04:16:05.2866058Z 2c6e0c416cd7: Pulling fs layer
2022-05-18T04:16:05.2866321Z 9642ca476af3: Pulling fs layer
2022-05-18T04:16:05.2866566Z 23d19ef2f74c: Pulling fs layer
2022-05-18T04:16:05.2866817Z 2c6e0c416cd7: Waiting
2022-05-18T04:16:05.2867070Z 097f8fc9708d: Pulling fs layer
2022-05-18T04:16:05.2867305Z e31574ad02fc: Waiting
2022-05-18T04:16:05.2867553Z 905dc4a4a899: Pulling fs layer
2022-05-18T04:16:05.2867802Z 23d19ef2f74c: Waiting
2022-05-18T04:16:05.2868035Z d38486135f29: Pulling fs layer
2022-05-18T04:16:05.2868276Z 097f8fc9708d: Waiting
2022-05-18T04:16:05.2868532Z db1065d40131: Pulling fs layer
2022-05-18T04:16:05.2868779Z 22de453da86e: Pulling fs layer
2022-05-18T04:16:05.2869048Z dc725e8f0593: Pulling fs layer
2022-05-18T04:16:05.2869302Z 22de453da86e: Waiting
2022-05-18T04:16:05.2869540Z a0bccb87b633: Pulling fs layer
2022-05-18T04:16:05.2869795Z a0bccb87b633: Waiting
2022-05-18T04:16:05.2870030Z dc725e8f0593: Waiting
2022-05-18T04:16:05.3663681Z b51569e7c507: Verifying Checksum
2022-05-18T04:16:05.3663988Z b51569e7c507: Download complete
2022-05-18T04:16:05.3882487Z da8ef40b9eca: Verifying Checksum
2022-05-18T04:16:05.3882925Z da8ef40b9eca: Download complete
2022-05-18T04:16:05.4557746Z fb15d46c38dc: Verifying Checksum
2022-05-18T04:16:05.4558299Z fb15d46c38dc: Download complete
2022-05-18T04:16:05.5385119Z e0d2c5aceba3: Verifying Checksum
2022-05-18T04:16:05.5385685Z e0d2c5aceba3: Download complete
2022-05-18T04:16:05.6331732Z 3c5d24e8ef06: Verifying Checksum
2022-05-18T04:16:05.6332275Z 3c5d24e8ef06: Download complete
2022-05-18T04:16:05.6751213Z 9c4425f4b8cb: Verifying Checksum
2022-05-18T04:16:05.6751714Z 9c4425f4b8cb: Download complete
2022-05-18T04:16:05.7307249Z 79c5859701ff: Download complete
2022-05-18T04:16:05.8154927Z 58690f9b18fc: Verifying Checksum
2022-05-18T04:16:05.8155701Z 58690f9b18fc: Download complete
2022-05-18T04:16:05.8186567Z f256bb6f705c: Verifying Checksum
2022-05-18T04:16:05.8186908Z f256bb6f705c: Download complete
2022-05-18T04:16:05.9011695Z b9db730d0400: Download complete
2022-05-18T04:16:06.1831085Z 49f9027dc2e7: Download complete
2022-05-18T04:16:07.9285370Z d60308a752bd: Verifying Checksum
2022-05-18T04:16:07.9285732Z d60308a752bd: Download complete
2022-05-18T04:16:08.0257753Z 624ec6d4936f: Verifying Checksum
2022-05-18T04:16:08.0258105Z 624ec6d4936f: Download complete
2022-05-18T04:16:08.1080800Z b815c3dfeb5c: Download complete
2022-05-18T04:16:08.1913093Z 4a9f9b66af25: Download complete
2022-05-18T04:16:08.2884564Z 58690f9b18fc: Pull complete
2022-05-18T04:16:08.4275653Z b51569e7c507: Pull complete
2022-05-18T04:16:08.5585331Z da8ef40b9eca: Pull complete
2022-05-18T04:16:08.5970666Z b6d963fbdb11: Verifying Checksum
2022-05-18T04:16:08.5971402Z b6d963fbdb11: Download complete
2022-05-18T04:16:08.6627307Z fb15d46c38dc: Pull complete
2022-05-18T04:16:08.6761321Z 4486d2823377: Download complete
2022-05-18T04:16:08.7603054Z 68d34a18a767: Download complete
2022-05-18T04:16:08.9460870Z e0d2c5aceba3: Pull complete
2022-05-18T04:16:09.2846621Z 9c4425f4b8cb: Pull complete
2022-05-18T04:16:09.4139777Z 3c5d24e8ef06: Pull complete
2022-05-18T04:16:09.5344456Z 79c5859701ff: Pull complete
2022-05-18T04:16:14.4702490Z d81828418d08: Verifying Checksum
2022-05-18T04:16:14.4702821Z d81828418d08: Download complete
2022-05-18T04:16:14.5931752Z 1d14eefa2afe: Verifying Checksum
2022-05-18T04:16:14.5932405Z 1d14eefa2afe: Download complete
2022-05-18T04:16:14.6599553Z cd1fd540bef8: Verifying Checksum
2022-05-18T04:16:14.6600173Z cd1fd540bef8: Download complete
2022-05-18T04:16:15.1984496Z fdc2f33cd3f0: Verifying Checksum
2022-05-18T04:16:15.1985369Z fdc2f33cd3f0: Download complete
2022-05-18T04:16:15.2859201Z 0626725f1e19: Verifying Checksum
2022-05-18T04:16:15.2859573Z 0626725f1e19: Download complete
2022-05-18T04:16:15.3634898Z 60b6e4baae49: Verifying Checksum
2022-05-18T04:16:15.3635569Z 60b6e4baae49: Download complete
2022-05-18T04:16:15.4532013Z a9f25937ad89: Verifying Checksum
2022-05-18T04:16:15.4532616Z a9f25937ad89: Download complete
2022-05-18T04:16:16.4042252Z 341c51541e6b: Verifying Checksum
2022-05-18T04:16:16.4042612Z 341c51541e6b: Download complete
2022-05-18T04:16:16.5036667Z 31a8b7b678c7: Verifying Checksum
2022-05-18T04:16:16.5037281Z 31a8b7b678c7: Download complete
2022-05-18T04:16:16.5844594Z ad8c1f2236e5: Verifying Checksum
2022-05-18T04:16:16.5844950Z ad8c1f2236e5: Download complete
2022-05-18T04:16:16.6653262Z f8f22be640a6: Verifying Checksum
2022-05-18T04:16:16.6653846Z f8f22be640a6: Download complete
2022-05-18T04:16:16.7147432Z cc2d8a95a2e5: Verifying Checksum
2022-05-18T04:16:16.7147934Z cc2d8a95a2e5: Download complete
2022-05-18T04:16:16.7439096Z 0b6a6636bca7: Verifying Checksum
2022-05-18T04:16:16.7439392Z 0b6a6636bca7: Download complete
2022-05-18T04:16:16.8026371Z 9c50e79f8e38: Download complete
2022-05-18T04:16:16.8646487Z 84c1af12bf7e: Verifying Checksum
2022-05-18T04:16:16.8647082Z 84c1af12bf7e: Download complete
2022-05-18T04:16:16.9450875Z 30d627f75fb9: Verifying Checksum
2022-05-18T04:16:16.9451714Z 30d627f75fb9: Download complete
2022-05-18T04:16:17.2001159Z 0d7a717fbbe1: Verifying Checksum
2022-05-18T04:16:17.2001531Z 0d7a717fbbe1: Download complete
2022-05-18T04:16:17.2850126Z 4c42c8b107a9: Download complete
2022-05-18T04:16:17.7795679Z 37c76e461e24: Verifying Checksum
2022-05-18T04:16:17.8522527Z 37c76e461e24: Download complete
2022-05-18T04:16:17.8523050Z 547da1897fee: Verifying Checksum
2022-05-18T04:16:17.8523566Z 547da1897fee: Download complete
2022-05-18T04:16:17.8963250Z dcb77576adf6: Verifying Checksum
2022-05-18T04:16:17.8963645Z dcb77576adf6: Download complete
2022-05-18T04:16:17.9190251Z e31574ad02fc: Download complete
2022-05-18T04:16:17.9914956Z 2c6e0c416cd7: Download complete
2022-05-18T04:16:18.0472054Z 9642ca476af3: Verifying Checksum
2022-05-18T04:16:18.0472671Z 9642ca476af3: Download complete
2022-05-18T04:16:18.1169947Z 23d19ef2f74c: Download complete
2022-05-18T04:16:18.1867909Z 097f8fc9708d: Verifying Checksum
2022-05-18T04:16:18.1868250Z 097f8fc9708d: Download complete
2022-05-18T04:16:18.3372109Z 905dc4a4a899: Verifying Checksum
2022-05-18T04:16:18.3372767Z 905dc4a4a899: Download complete
2022-05-18T04:16:18.4087821Z d38486135f29: Verifying Checksum
2022-05-18T04:16:18.4088509Z d38486135f29: Download complete
2022-05-18T04:16:19.0017332Z db1065d40131: Verifying Checksum
2022-05-18T04:16:19.0017919Z db1065d40131: Download complete
2022-05-18T04:16:19.0994516Z 22de453da86e: Verifying Checksum
2022-05-18T04:16:19.0995270Z 22de453da86e: Download complete
2022-05-18T04:16:21.5384063Z a9dad096f89d: Verifying Checksum
2022-05-18T04:16:21.5384716Z a9dad096f89d: Download complete
2022-05-18T04:16:21.6257658Z a0bccb87b633: Verifying Checksum
2022-05-18T04:16:21.6257988Z a0bccb87b633: Download complete
2022-05-18T04:16:22.8476102Z 1c478b5d7dcd: Verifying Checksum
2022-05-18T04:16:22.8476853Z 1c478b5d7dcd: Download complete
2022-05-18T04:16:25.6488934Z d81828418d08: Pull complete
2022-05-18T04:16:25.7822399Z f256bb6f705c: Pull complete
2022-05-18T04:16:42.4221838Z cc2d8a95a2e5: Pull complete
2022-05-18T04:16:44.3977211Z b9db730d0400: Pull complete
2022-05-18T04:16:46.3982624Z 49f9027dc2e7: Pull complete
2022-05-18T04:16:49.0910073Z dc725e8f0593: Verifying Checksum
2022-05-18T04:16:49.0910433Z dc725e8f0593: Download complete
2022-05-18T04:16:53.0637867Z d60308a752bd: Pull complete
2022-05-18T04:16:54.9413616Z 624ec6d4936f: Pull complete
2022-05-18T04:16:56.8203415Z b815c3dfeb5c: Pull complete
2022-05-18T04:16:58.6986748Z 4a9f9b66af25: Pull complete
2022-05-18T04:17:02.3788104Z b6d963fbdb11: Pull complete
2022-05-18T04:17:04.3354573Z 4486d2823377: Pull complete
2022-05-18T04:17:06.4964245Z 68d34a18a767: Pull complete
2022-05-18T04:17:31.3379843Z 1c478b5d7dcd: Pull complete
2022-05-18T04:17:33.3563483Z 1d14eefa2afe: Pull complete
2022-05-18T04:17:35.2031138Z cd1fd540bef8: Pull complete
2022-05-18T04:17:38.3548307Z fdc2f33cd3f0: Pull complete
2022-05-18T04:17:40.6264597Z 0626725f1e19: Pull complete
2022-05-18T04:17:42.5353001Z 60b6e4baae49: Pull complete
2022-05-18T04:17:44.4108834Z a9f25937ad89: Pull complete
2022-05-18T04:17:48.5904693Z 341c51541e6b: Pull complete
2022-05-18T04:17:51.0174363Z 31a8b7b678c7: Pull complete
2022-05-18T04:17:53.6274928Z ad8c1f2236e5: Pull complete
2022-05-18T04:17:53.7931652Z f8f22be640a6: Pull complete
2022-05-18T04:17:53.8900379Z 0b6a6636bca7: Pull complete
2022-05-18T04:17:54.0230699Z 9c50e79f8e38: Pull complete
2022-05-18T04:17:56.5286597Z 37c76e461e24: Pull complete
2022-05-18T04:17:56.6423220Z 84c1af12bf7e: Pull complete
2022-05-18T04:17:56.7556459Z 30d627f75fb9: Pull complete
2022-05-18T04:17:57.1317373Z 0d7a717fbbe1: Pull complete
2022-05-18T04:17:57.2336041Z 4c42c8b107a9: Pull complete
2022-05-18T04:17:58.4720553Z dcb77576adf6: Pull complete
2022-05-18T04:17:58.5823915Z 547da1897fee: Pull complete
2022-05-18T04:17:58.7042556Z e31574ad02fc: Pull complete
2022-05-18T04:18:06.4003386Z a9dad096f89d: Pull complete
2022-05-18T04:18:08.2477229Z 2c6e0c416cd7: Pull complete
2022-05-18T04:18:10.4941424Z 9642ca476af3: Pull complete
2022-05-18T04:18:12.4021893Z 23d19ef2f74c: Pull complete
2022-05-18T04:18:14.3111009Z 097f8fc9708d: Pull complete
2022-05-18T04:18:17.4078261Z 905dc4a4a899: Pull complete
2022-05-18T04:18:19.3155850Z d38486135f29: Pull complete
2022-05-18T04:18:21.7403342Z db1065d40131: Pull complete
2022-05-18T04:18:21.8404622Z 22de453da86e: Pull complete
2022-05-18T04:19:03.7145219Z dc725e8f0593: Pull complete
2022-05-18T04:19:05.7192450Z a0bccb87b633: Pull complete
2022-05-18T04:19:07.0670326Z Digest: sha256:66b56fbc2d0d8bf75af01c4976aba15f28c9802507dc01f27e71a55f8ffc13e0
2022-05-18T04:19:07.5680902Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:19:07.8513289Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:19:07.8599667Z ##[group]Run nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a
2022-05-18T04:19:07.8600011Z with:
2022-05-18T04:19:07.8600226Z   timeout_minutes: 10
2022-05-18T04:19:07.8600471Z   max_attempts: 3
2022-05-18T04:19:07.8600857Z   command: set -ex
bash .github/scripts/install_nvidia_utils_linux.sh
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2022-05-18T04:19:07.8601220Z   retry_wait_seconds: 10
2022-05-18T04:19:07.8601487Z   polling_interval_seconds: 1
2022-05-18T04:19:07.8601747Z   warning_on_retry: true
2022-05-18T04:19:07.8601986Z   continue_on_error: false
2022-05-18T04:19:07.8602222Z env:
2022-05-18T04:19:07.8602433Z   IN_CI: 1
2022-05-18T04:19:07.8602635Z   IS_GHA: 1
2022-05-18T04:19:07.8602877Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:19:07.8603143Z ##[endgroup]
2022-05-18T04:19:07.9041020Z 
2022-05-18T04:19:07.9114774Z == Installing nvidia container toolkit for amzn2 ==
2022-05-18T04:19:07.9117701Z + bash .github/scripts/install_nvidia_utils_linux.sh
2022-05-18T04:19:07.9118133Z + sudo yum install -y yum-utils
2022-05-18T04:19:08.3805688Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:19:09.1715337Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2022-05-18T04:19:09.1715751Z Nothing to do
2022-05-18T04:19:09.1914964Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-05-18T04:19:09.8194287Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:19:09.8472568Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-05-18T04:19:09.8473241Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2022-05-18T04:19:09.8473992Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2022-05-18T04:19:09.8620641Z + sudo yum install -y nvidia-docker2
2022-05-18T04:19:10.3396174Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:19:10.3797908Z Retrieving key from https://nvidia.github.io/libnvidia-container/gpgkey
2022-05-18T04:19:10.3868998Z Importing GPG key 0xF796ECB0:
2022-05-18T04:19:10.3869379Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-05-18T04:19:10.3869760Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-05-18T04:19:10.3870237Z  From       : https://nvidia.github.io/libnvidia-container/gpgkey
2022-05-18T04:19:12.2007171Z Retrieving key from https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-05-18T04:19:12.2076037Z Importing GPG key 0xF796ECB0:
2022-05-18T04:19:12.2076444Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-05-18T04:19:12.2076869Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-05-18T04:19:12.2079108Z  From       : https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-05-18T04:19:12.4316792Z Retrieving key from https://nvidia.github.io/nvidia-docker/gpgkey
2022-05-18T04:19:12.4384997Z Importing GPG key 0xF796ECB0:
2022-05-18T04:19:12.4385702Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-05-18T04:19:12.4386200Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-05-18T04:19:12.4386655Z  From       : https://nvidia.github.io/nvidia-docker/gpgkey
2022-05-18T04:19:32.6552229Z Resolving Dependencies
2022-05-18T04:19:32.6557944Z --> Running transaction check
2022-05-18T04:19:32.6558574Z ---> Package nvidia-docker2.noarch 0:2.10.0-1 will be installed
2022-05-18T04:19:32.6583942Z --> Processing Dependency: nvidia-container-toolkit >= 1.9.0-1 for package: nvidia-docker2-2.10.0-1.noarch
2022-05-18T04:19:32.6905724Z --> Running transaction check
2022-05-18T04:19:32.6906240Z ---> Package nvidia-container-toolkit.x86_64 0:1.9.0-1 will be installed
2022-05-18T04:19:32.6915063Z --> Processing Dependency: libnvidia-container-tools < 2.0.0 for package: nvidia-container-toolkit-1.9.0-1.x86_64
2022-05-18T04:19:32.7032740Z --> Processing Dependency: libnvidia-container-tools >= 1.9.0-1 for package: nvidia-container-toolkit-1.9.0-1.x86_64
2022-05-18T04:19:32.7033269Z --> Running transaction check
2022-05-18T04:19:32.7033696Z ---> Package libnvidia-container-tools.x86_64 0:1.9.0-1 will be installed
2022-05-18T04:19:32.7062112Z --> Processing Dependency: libnvidia-container1(x86-64) >= 1.9.0-1 for package: libnvidia-container-tools-1.9.0-1.x86_64
2022-05-18T04:19:32.7091320Z --> Processing Dependency: libnvidia-container.so.1(NVC_1.0)(64bit) for package: libnvidia-container-tools-1.9.0-1.x86_64
2022-05-18T04:19:32.7092054Z --> Processing Dependency: libnvidia-container.so.1()(64bit) for package: libnvidia-container-tools-1.9.0-1.x86_64
2022-05-18T04:19:32.7092523Z --> Running transaction check
2022-05-18T04:19:32.7093299Z ---> Package libnvidia-container1.x86_64 0:1.9.0-1 will be installed
2022-05-18T04:19:32.9926307Z --> Finished Dependency Resolution
2022-05-18T04:19:33.0542589Z 
2022-05-18T04:19:33.0542879Z Dependencies Resolved
2022-05-18T04:19:33.0554283Z 
2022-05-18T04:19:33.0554825Z ================================================================================
2022-05-18T04:19:33.0555354Z  Package                     Arch     Version       Repository             Size
2022-05-18T04:19:33.0557748Z ================================================================================
2022-05-18T04:19:33.0558351Z Installing:
2022-05-18T04:19:33.0558916Z  nvidia-docker2              noarch   2.10.0-1      libnvidia-container   8.7 k
2022-05-18T04:19:33.0559283Z Installing for dependencies:
2022-05-18T04:19:33.0559745Z  libnvidia-container-tools   x86_64   1.9.0-1       libnvidia-container    48 k
2022-05-18T04:19:33.0560249Z  libnvidia-container1        x86_64   1.9.0-1       libnvidia-container   1.0 M
2022-05-18T04:19:33.0560976Z  nvidia-container-toolkit    x86_64   1.9.0-1       libnvidia-container   1.5 M
2022-05-18T04:19:33.0561222Z 
2022-05-18T04:19:33.0561337Z Transaction Summary
2022-05-18T04:19:33.0561635Z ================================================================================
2022-05-18T04:19:33.0561931Z Install  1 Package (+3 Dependent packages)
2022-05-18T04:19:33.0562130Z 
2022-05-18T04:19:33.0562251Z Total download size: 2.5 M
2022-05-18T04:19:33.0566344Z Installed size: 7.4 M
2022-05-18T04:19:33.0566642Z Downloading packages:
2022-05-18T04:19:33.1533421Z --------------------------------------------------------------------------------
2022-05-18T04:19:33.1533886Z Total                                               26 MB/s | 2.5 MB  00:00     
2022-05-18T04:19:33.1577423Z Running transaction check
2022-05-18T04:19:33.1638508Z Running transaction test
2022-05-18T04:19:33.1788176Z Transaction test succeeded
2022-05-18T04:19:33.1791357Z Running transaction
2022-05-18T04:19:37.7908380Z   Installing : libnvidia-container1-1.9.0-1.x86_64                          1/4 
2022-05-18T04:19:38.9899931Z   Installing : libnvidia-container-tools-1.9.0-1.x86_64                     2/4 
2022-05-18T04:19:39.0108555Z   Installing : nvidia-container-toolkit-1.9.0-1.x86_64                      3/4 
2022-05-18T04:19:39.0480069Z   Installing : nvidia-docker2-2.10.0-1.noarch                               4/4 
2022-05-18T04:19:39.0577536Z   Verifying  : libnvidia-container-tools-1.9.0-1.x86_64                     1/4 
2022-05-18T04:19:39.0676610Z   Verifying  : nvidia-container-toolkit-1.9.0-1.x86_64                      2/4 
2022-05-18T04:19:39.0772758Z   Verifying  : nvidia-docker2-2.10.0-1.noarch                               3/4 
2022-05-18T04:19:39.1405483Z   Verifying  : libnvidia-container1-1.9.0-1.x86_64                          4/4 
2022-05-18T04:19:39.1405996Z 
2022-05-18T04:19:39.1406179Z Installed:
2022-05-18T04:19:39.1406907Z   nvidia-docker2.noarch 0:2.10.0-1                                              
2022-05-18T04:19:39.1407304Z 
2022-05-18T04:19:39.1407553Z Dependency Installed:
2022-05-18T04:19:39.1408419Z   libnvidia-container-tools.x86_64 0:1.9.0-1                                    
2022-05-18T04:19:39.1409864Z   libnvidia-container1.x86_64 0:1.9.0-1                                         
2022-05-18T04:19:39.1410790Z   nvidia-container-toolkit.x86_64 0:1.9.0-1                                     
2022-05-18T04:19:39.1411249Z 
2022-05-18T04:19:39.1411457Z Complete!
2022-05-18T04:19:39.2384286Z + sudo systemctl restart docker
2022-05-18T04:19:39.7660939Z == Installing nvidia driver NVIDIA-Linux-x86_64-510.60.02.run ==
2022-05-18T04:19:39.7662703Z + sudo yum groupinstall -y 'Development Tools'
2022-05-18T04:19:40.2176019Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:19:40.2339641Z Existing lock /var/run/yum.pid: another copy is running as pid 19813.
2022-05-18T04:19:40.2340054Z Another app is currently holding the yum lock; waiting for it to exit...
2022-05-18T04:19:40.2348157Z   The other application is: yum
2022-05-18T04:19:40.2348453Z     Memory :  91 M RSS (308 MB VSZ)
2022-05-18T04:19:40.2349152Z     Started: Wed May 18 04:19:39 2022 - 00:01 ago
2022-05-18T04:19:40.2349554Z     State  : Running, pid: 19813
2022-05-18T04:19:42.2374926Z Another app is currently holding the yum lock; waiting for it to exit...
2022-05-18T04:19:42.2380975Z   The other application is: yum
2022-05-18T04:19:42.2381272Z     Memory : 155 M RSS (373 MB VSZ)
2022-05-18T04:19:42.2381813Z     Started: Wed May 18 04:19:39 2022 - 00:03 ago
2022-05-18T04:19:42.2382183Z     State  : Running, pid: 19813
2022-05-18T04:19:45.2547426Z Resolving Dependencies
2022-05-18T04:19:45.2551880Z --> Running transaction check
2022-05-18T04:19:45.2554859Z ---> Package autoconf.noarch 0:2.69-11.amzn2 will be installed
2022-05-18T04:19:45.2786397Z --> Processing Dependency: m4 >= 1.4.14 for package: autoconf-2.69-11.amzn2.noarch
2022-05-18T04:19:45.3111387Z --> Processing Dependency: perl(Data::Dumper) for package: autoconf-2.69-11.amzn2.noarch
2022-05-18T04:19:45.3112897Z ---> Package automake.noarch 0:1.13.4-3.1.amzn2 will be installed
2022-05-18T04:19:45.3176025Z --> Processing Dependency: perl(Thread::Queue) for package: automake-1.13.4-3.1.amzn2.noarch
2022-05-18T04:19:45.3183692Z --> Processing Dependency: perl(TAP::Parser) for package: automake-1.13.4-3.1.amzn2.noarch
2022-05-18T04:19:45.3195422Z ---> Package bison.x86_64 0:3.0.4-6.amzn2.0.2 will be installed
2022-05-18T04:19:45.3304617Z ---> Package byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 will be installed
2022-05-18T04:19:45.3316193Z ---> Package cscope.x86_64 0:15.8-10.amzn2.0.2 will be installed
2022-05-18T04:19:45.3368431Z ---> Package ctags.x86_64 0:5.8-13.amzn2.0.2 will be installed
2022-05-18T04:19:45.3381411Z ---> Package diffstat.x86_64 0:1.57-4.amzn2.0.2 will be installed
2022-05-18T04:19:45.3392434Z ---> Package doxygen.x86_64 1:1.8.5-4.amzn2 will be installed
2022-05-18T04:19:45.3486911Z ---> Package elfutils.x86_64 0:0.176-2.amzn2 will be installed
2022-05-18T04:19:45.3650387Z ---> Package flex.x86_64 0:2.5.37-3.amzn2.0.3 will be installed
2022-05-18T04:19:45.3681366Z ---> Package gcc.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:19:45.3840528Z --> Processing Dependency: cpp = 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:19:45.3856632Z --> Processing Dependency: libsanitizer >= 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:19:45.3904709Z --> Processing Dependency: libquadmath >= 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:19:45.3958287Z --> Processing Dependency: libmpx >= 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:19:45.4007613Z --> Processing Dependency: libitm >= 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:19:45.4053848Z --> Processing Dependency: libcilkrts >= 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:19:45.4102296Z --> Processing Dependency: libatomic >= 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:19:45.4152008Z --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:19:45.4286790Z --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:19:45.4310081Z --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:19:45.4333682Z ---> Package gcc-c++.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:19:45.4370640Z ---> Package gcc-gfortran.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:19:45.4415277Z --> Processing Dependency: libgfortran.so.4()(64bit) for package: gcc-gfortran-7.3.1-14.amzn2.x86_64
2022-05-18T04:19:45.4474329Z ---> Package indent.x86_64 0:2.2.11-13.amzn2.0.2 will be installed
2022-05-18T04:19:45.4497817Z ---> Package intltool.noarch 0:0.50.2-7.amzn2 will be installed
2022-05-18T04:19:45.4551905Z --> Processing Dependency: perl(XML::Parser) for package: intltool-0.50.2-7.amzn2.noarch
2022-05-18T04:19:45.4568548Z --> Processing Dependency: gettext-devel for package: intltool-0.50.2-7.amzn2.noarch
2022-05-18T04:19:45.4589494Z ---> Package libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 will be installed
2022-05-18T04:19:45.4626917Z ---> Package patch.x86_64 0:2.7.1-12.amzn2.0.2 will be installed
2022-05-18T04:19:45.4668395Z ---> Package patchutils.x86_64 0:0.3.3-4.amzn2.0.1 will be installed
2022-05-18T04:19:45.4699788Z ---> Package rcs.x86_64 0:5.9.0-5.amzn2.0.2 will be installed
2022-05-18T04:19:45.4742890Z ---> Package rpm-build.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-05-18T04:19:45.4978198Z --> Processing Dependency: /usr/bin/gdb-add-index for package: rpm-build-4.11.3-48.amzn2.0.2.x86_64
2022-05-18T04:19:45.4998837Z ---> Package rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-05-18T04:19:45.5034693Z ---> Package subversion.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-05-18T04:19:45.5196888Z --> Processing Dependency: subversion-libs(x86-64) = 1.7.14-16.amzn2.0.1 for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5217658Z --> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5218730Z --> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5219342Z --> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5220079Z --> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5220700Z --> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5221331Z --> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5222092Z --> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5222722Z --> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5223339Z --> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5223954Z --> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5224686Z --> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5225305Z --> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5225917Z --> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5226542Z --> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5227144Z --> Processing Dependency: libneon.so.27()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5247742Z --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5271033Z --> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:19:45.5297666Z ---> Package swig.x86_64 0:3.0.12-11.amzn2.0.3 will be installed
2022-05-18T04:19:45.5324618Z ---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.13 will be installed
2022-05-18T04:19:45.5375872Z --> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.13.noarch
2022-05-18T04:19:45.5394532Z --> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.13.noarch
2022-05-18T04:19:45.5408643Z --> Processing Dependency: go-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.13.noarch
2022-05-18T04:19:45.5577795Z ---> Package systemtap.x86_64 0:4.4-1.amzn2.0.2 will be installed
2022-05-18T04:19:45.5594175Z --> Processing Dependency: systemtap-devel = 4.4-1.amzn2.0.2 for package: systemtap-4.4-1.amzn2.0.2.x86_64
2022-05-18T04:19:45.5606480Z --> Processing Dependency: systemtap-client = 4.4-1.amzn2.0.2 for package: systemtap-4.4-1.amzn2.0.2.x86_64
2022-05-18T04:19:45.5618066Z --> Running transaction check
2022-05-18T04:19:45.5621285Z ---> Package apr.x86_64 0:1.7.0-9.amzn2 will be installed
2022-05-18T04:19:45.5704636Z ---> Package apr-util.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-05-18T04:19:45.5752354Z --> Processing Dependency: apr-util-bdb(x86-64) = 1.6.1-5.amzn2.0.2 for package: apr-util-1.6.1-5.amzn2.0.2.x86_64
2022-05-18T04:19:45.5767591Z ---> Package cpp.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:19:45.5853117Z ---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed
2022-05-18T04:19:45.5868923Z ---> Package gdb.x86_64 0:8.0.1-36.amzn2.0.1 will be installed
2022-05-18T04:19:45.5949471Z ---> Package gettext-devel.x86_64 0:0.19.8.1-3.amzn2 will be installed
2022-05-18T04:19:45.6014561Z --> Processing Dependency: gettext-common-devel = 0.19.8.1-3.amzn2 for package: gettext-devel-0.19.8.1-3.amzn2.x86_64
2022-05-18T04:19:45.6023588Z ---> Package glibc-devel.x86_64 0:2.26-58.amzn2 will be installed
2022-05-18T04:19:45.6140973Z --> Processing Dependency: glibc-headers = 2.26-58.amzn2 for package: glibc-devel-2.26-58.amzn2.x86_64
2022-05-18T04:19:45.6165404Z --> Processing Dependency: glibc-headers for package: glibc-devel-2.26-58.amzn2.x86_64
2022-05-18T04:19:45.6165986Z ---> Package go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.1 will be installed
2022-05-18T04:19:45.6172050Z ---> Package libatomic.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:19:45.6194836Z ---> Package libcilkrts.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:19:45.6234557Z ---> Package libgfortran.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:19:45.6281559Z ---> Package libitm.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:19:45.6307553Z ---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed
2022-05-18T04:19:45.6327311Z ---> Package libmpx.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:19:45.6351984Z ---> Package libquadmath.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:19:45.6388450Z ---> Package libsanitizer.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:19:45.6449422Z ---> Package m4.x86_64 0:1.4.16-10.amzn2.0.2 will be installed
2022-05-18T04:19:45.6474435Z ---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed
2022-05-18T04:19:45.6505571Z ---> Package neon.x86_64 0:0.30.0-3.amzn2.0.2 will be installed
2022-05-18T04:19:45.6590618Z --> Processing Dependency: libgnutls.so.28(GNUTLS_2_12)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-05-18T04:19:45.6628883Z --> Processing Dependency: libgnutls.so.28(GNUTLS_1_4)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-05-18T04:19:45.6629994Z --> Processing Dependency: libproxy.so.1()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-05-18T04:19:45.6651390Z --> Processing Dependency: libpakchois.so.0()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-05-18T04:19:45.6671190Z --> Processing Dependency: libgnutls.so.28()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-05-18T04:19:45.6678590Z ---> Package perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 will be installed
2022-05-18T04:19:45.6734123Z ---> Package perl-Test-Harness.noarch 0:3.28-3.amzn2 will be installed
2022-05-18T04:19:45.6873634Z ---> Package perl-Thread-Queue.noarch 0:3.02-2.amzn2 will be installed
2022-05-18T04:19:45.6888558Z ---> Package perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 will be installed
2022-05-18T04:19:45.6911507Z ---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed
2022-05-18T04:19:45.6912611Z ---> Package subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-05-18T04:19:45.6960648Z ---> Package systemtap-client.x86_64 0:4.4-1.amzn2.0.2 will be installed
2022-05-18T04:19:45.7208542Z --> Processing Dependency: mokutil for package: systemtap-client-4.4-1.amzn2.0.2.x86_64
2022-05-18T04:19:45.7224240Z --> Processing Dependency: libavahi-common.so.3()(64bit) for package: systemtap-client-4.4-1.amzn2.0.2.x86_64
2022-05-18T04:19:45.7251827Z --> Processing Dependency: libavahi-client.so.3()(64bit) for package: systemtap-client-4.4-1.amzn2.0.2.x86_64
2022-05-18T04:19:45.7252391Z ---> Package systemtap-devel.x86_64 0:4.4-1.amzn2.0.2 will be installed
2022-05-18T04:19:45.7376741Z --> Processing Dependency: kernel-devel-uname-r for package: systemtap-devel-4.4-1.amzn2.0.2.x86_64
2022-05-18T04:19:45.8285120Z --> Running transaction check
2022-05-18T04:19:45.8286377Z ---> Package apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-05-18T04:19:45.8301433Z ---> Package avahi-libs.x86_64 0:0.6.31-20.amzn2 will be installed
2022-05-18T04:19:45.8336521Z ---> Package gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 will be installed
2022-05-18T04:19:45.8337590Z ---> Package glibc-headers.x86_64 0:2.26-58.amzn2 will be installed
2022-05-18T04:19:45.8380601Z --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-58.amzn2.x86_64
2022-05-18T04:19:45.9354466Z --> Processing Dependency: kernel-headers for package: glibc-headers-2.26-58.amzn2.x86_64
2022-05-18T04:19:45.9355312Z ---> Package gnutls.x86_64 0:3.3.29-9.amzn2.0.1 will be installed
2022-05-18T04:19:45.9433514Z --> Processing Dependency: trousers >= 0.3.11.2 for package: gnutls-3.3.29-9.amzn2.0.1.x86_64
2022-05-18T04:19:45.9462387Z ---> Package kernel-devel.x86_64 0:4.14.276-211.499.amzn2 will be installed
2022-05-18T04:19:45.9492050Z --> Processing Dependency: elfutils-libelf-devel for package: kernel-devel-4.14.276-211.499.amzn2.x86_64
2022-05-18T04:19:45.9513444Z ---> Package libproxy.x86_64 0:0.4.11-10.amzn2.0.3 will be installed
2022-05-18T04:19:45.9553471Z --> Processing Dependency: libmodman.so.1()(64bit) for package: libproxy-0.4.11-10.amzn2.0.3.x86_64
2022-05-18T04:19:45.9573802Z ---> Package mokutil.x86_64 1:0.3.0-10.amzn2.0.1 will be installed
2022-05-18T04:19:45.9622154Z --> Processing Dependency: libefivar.so.1(libefivar.so.0)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-05-18T04:19:45.9644634Z --> Processing Dependency: libefivar.so.1(LIBEFIVAR_0.24)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-05-18T04:19:45.9645701Z --> Processing Dependency: libefivar.so.1()(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-05-18T04:19:45.9646283Z ---> Package pakchois.x86_64 0:0.4-10.amzn2.0.2 will be installed
2022-05-18T04:19:45.9667060Z --> Running transaction check
2022-05-18T04:19:45.9667968Z ---> Package efivar-libs.x86_64 0:31-4.amzn2.0.4 will be installed
2022-05-18T04:19:45.9694426Z ---> Package elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 will be installed
2022-05-18T04:19:45.9707521Z --> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.amzn2.x86_64
2022-05-18T04:19:45.9735064Z ---> Package kernel-headers.x86_64 0:4.14.276-211.499.amzn2 will be installed
2022-05-18T04:19:45.9736174Z ---> Package libmodman.x86_64 0:2.0.1-8.amzn2.0.2 will be installed
2022-05-18T04:19:45.9764329Z ---> Package trousers.x86_64 0:0.3.14-2.amzn2.0.2 will be installed
2022-05-18T04:19:45.9829196Z --> Running transaction check
2022-05-18T04:19:45.9829788Z ---> Package zlib-devel.x86_64 0:1.2.7-19.amzn2.0.1 will be installed
2022-05-18T04:19:46.2356189Z --> Finished Dependency Resolution
2022-05-18T04:19:46.3446668Z 
2022-05-18T04:19:46.3447113Z Dependencies Resolved
2022-05-18T04:19:46.3559158Z 
2022-05-18T04:19:46.3559525Z ================================================================================
2022-05-18T04:19:46.3559874Z  Package                 Arch     Version                    Repository    Size
2022-05-18T04:19:46.3560239Z ================================================================================
2022-05-18T04:19:46.3560566Z Installing for group install "Development Tools":
2022-05-18T04:19:46.3561099Z  autoconf                noarch   2.69-11.amzn2              amzn2-core   701 k
2022-05-18T04:19:46.3561528Z  automake                noarch   1.13.4-3.1.amzn2           amzn2-core   679 k
2022-05-18T04:19:46.3561986Z  bison                   x86_64   3.0.4-6.amzn2.0.2          amzn2-core   674 k
2022-05-18T04:19:46.3562425Z  byacc                   x86_64   1.9.20130304-3.amzn2.0.2   amzn2-core    66 k
2022-05-18T04:19:46.3562833Z  cscope                  x86_64   15.8-10.amzn2.0.2          amzn2-core   204 k
2022-05-18T04:19:46.3563258Z  ctags                   x86_64   5.8-13.amzn2.0.2           amzn2-core   157 k
2022-05-18T04:19:46.3563685Z  diffstat                x86_64   1.57-4.amzn2.0.2           amzn2-core    35 k
2022-05-18T04:19:46.3564112Z  doxygen                 x86_64   1:1.8.5-4.amzn2            amzn2-core   3.5 M
2022-05-18T04:19:46.3564520Z  elfutils                x86_64   0.176-2.amzn2              amzn2-core   307 k
2022-05-18T04:19:46.3564944Z  flex                    x86_64   2.5.37-3.amzn2.0.3         amzn2-core   291 k
2022-05-18T04:19:46.3565355Z  gcc                     x86_64   7.3.1-14.amzn2             amzn2-core    22 M
2022-05-18T04:19:46.3565974Z  gcc-c++                 x86_64   7.3.1-14.amzn2             amzn2-core    13 M
2022-05-18T04:19:46.3566390Z  gcc-gfortran            x86_64   7.3.1-14.amzn2             amzn2-core    11 M
2022-05-18T04:19:46.3566824Z  indent                  x86_64   2.2.11-13.amzn2.0.2        amzn2-core   150 k
2022-05-18T04:19:46.3567255Z  intltool                noarch   0.50.2-7.amzn2             amzn2-core    59 k
2022-05-18T04:19:46.3567665Z  libtool                 x86_64   2.4.2-22.2.amzn2.0.2       amzn2-core   588 k
2022-05-18T04:19:46.3568095Z  patch                   x86_64   2.7.1-12.amzn2.0.2         amzn2-core   110 k
2022-05-18T04:19:46.3568519Z  patchutils              x86_64   0.3.3-4.amzn2.0.1          amzn2-core   104 k
2022-05-18T04:19:46.3568941Z  rcs                     x86_64   5.9.0-5.amzn2.0.2          amzn2-core   231 k
2022-05-18T04:19:46.3569352Z  rpm-build               x86_64   4.11.3-48.amzn2.0.2        amzn2-core   150 k
2022-05-18T04:19:46.3570244Z  rpm-sign                x86_64   4.11.3-48.amzn2.0.2        amzn2-core    50 k
2022-05-18T04:19:46.3570681Z  subversion              x86_64   1.7.14-16.amzn2.0.1        amzn2-core   1.0 M
2022-05-18T04:19:46.3571086Z  swig                    x86_64   3.0.12-11.amzn2.0.3        amzn2-core   1.4 M
2022-05-18T04:19:46.3571531Z  system-rpm-config       noarch   9.1.0-76.amzn2.0.13        amzn2-core    89 k
2022-05-18T04:19:46.3572010Z  systemtap               x86_64   4.4-1.amzn2.0.2            amzn2-core    12 k
2022-05-18T04:19:46.3572327Z Installing for dependencies:
2022-05-18T04:19:46.3572716Z  apr                     x86_64   1.7.0-9.amzn2              amzn2-core   122 k
2022-05-18T04:19:46.3573137Z  apr-util                x86_64   1.6.1-5.amzn2.0.2          amzn2-core    99 k
2022-05-18T04:19:46.3573575Z  apr-util-bdb            x86_64   1.6.1-5.amzn2.0.2          amzn2-core    19 k
2022-05-18T04:19:46.3573996Z  avahi-libs              x86_64   0.6.31-20.amzn2            amzn2-core    61 k
2022-05-18T04:19:46.3574463Z  cpp                     x86_64   7.3.1-14.amzn2             amzn2-core   9.2 M
2022-05-18T04:19:46.3574883Z  dwz                     x86_64   0.11-3.amzn2.0.3           amzn2-core    98 k
2022-05-18T04:19:46.3575414Z  efivar-libs             x86_64   31-4.amzn2.0.4             amzn2-core    68 k
2022-05-18T04:19:46.3575870Z  elfutils-libelf-devel   x86_64   0.176-2.amzn2              amzn2-core    40 k
2022-05-18T04:19:46.3576313Z  gdb                     x86_64   8.0.1-36.amzn2.0.1         amzn2-core   3.1 M
2022-05-18T04:19:46.3576756Z  gettext-common-devel    noarch   0.19.8.1-3.amzn2           amzn2-core   410 k
2022-05-18T04:19:46.3577216Z  gettext-devel           x86_64   0.19.8.1-3.amzn2           amzn2-core   320 k
2022-05-18T04:19:46.3577633Z  glibc-devel             x86_64   2.26-58.amzn2              amzn2-core   994 k
2022-05-18T04:19:46.3578063Z  glibc-headers           x86_64   2.26-58.amzn2              amzn2-core   514 k
2022-05-18T04:19:46.3578498Z  gnutls                  x86_64   3.3.29-9.amzn2.0.1         amzn2-core   661 k
2022-05-18T04:19:46.3578924Z  go-srpm-macros          noarch   3.0.15-23.amzn2.0.1        amzn2-core    23 k
2022-05-18T04:19:46.3579380Z  kernel-devel            x86_64   4.14.276-211.499.amzn2     amzn2-core    13 M
2022-05-18T04:19:46.3579825Z  kernel-headers          x86_64   4.14.276-211.499.amzn2     amzn2-core   1.2 M
2022-05-18T04:19:46.3580257Z  libatomic               x86_64   7.3.1-14.amzn2             amzn2-core    46 k
2022-05-18T04:19:46.3580662Z  libcilkrts              x86_64   7.3.1-14.amzn2             amzn2-core    85 k
2022-05-18T04:19:46.3581086Z  libgfortran             x86_64   7.3.1-14.amzn2             amzn2-core   536 k
2022-05-18T04:19:46.3581506Z  libitm                  x86_64   7.3.1-14.amzn2             amzn2-core    84 k
2022-05-18T04:19:46.3581906Z  libmodman               x86_64   2.0.1-8.amzn2.0.2          amzn2-core    29 k
2022-05-18T04:19:46.3582335Z  libmpc                  x86_64   1.0.1-3.amzn2.0.2          amzn2-core    52 k
2022-05-18T04:19:46.3582837Z  libmpx                  x86_64   7.3.1-14.amzn2             amzn2-core    51 k
2022-05-18T04:19:46.3583260Z  libproxy                x86_64   0.4.11-10.amzn2.0.3        amzn2-core    61 k
2022-05-18T04:19:46.3583679Z  libquadmath             x86_64   7.3.1-14.amzn2             amzn2-core   189 k
2022-05-18T04:19:46.3584112Z  libsanitizer            x86_64   7.3.1-14.amzn2             amzn2-core   641 k
2022-05-18T04:19:46.3584535Z  m4                      x86_64   1.4.16-10.amzn2.0.2        amzn2-core   256 k
2022-05-18T04:19:46.3584933Z  mokutil                 x86_64   1:0.3.0-10.amzn2.0.1       amzn2-core    39 k
2022-05-18T04:19:46.3585356Z  mpfr                    x86_64   3.1.1-4.amzn2.0.2          amzn2-core   208 k
2022-05-18T04:19:46.3585771Z  neon                    x86_64   0.30.0-3.amzn2.0.2         amzn2-core   166 k
2022-05-18T04:19:46.3586192Z  pakchois                x86_64   0.4-10.amzn2.0.2           amzn2-core    14 k
2022-05-18T04:19:46.3586623Z  perl-Data-Dumper        x86_64   2.145-3.amzn2.0.2          amzn2-core    48 k
2022-05-18T04:19:46.3587085Z  perl-Test-Harness       noarch   3.28-3.amzn2               amzn2-core   302 k
2022-05-18T04:19:46.3587556Z  perl-Thread-Queue       noarch   3.02-2.amzn2               amzn2-core    17 k
2022-05-18T04:19:46.3588009Z  perl-XML-Parser         x86_64   2.41-10.amzn2.0.2          amzn2-core   223 k
2022-05-18T04:19:46.3588469Z  perl-srpm-macros        noarch   1-8.amzn2.0.1              amzn2-core   4.7 k
2022-05-18T04:19:46.3588930Z  subversion-libs         x86_64   1.7.14-16.amzn2.0.1        amzn2-core   912 k
2022-05-18T04:19:46.3589380Z  systemtap-client        x86_64   4.4-1.amzn2.0.2            amzn2-core   3.7 M
2022-05-18T04:19:46.3589810Z  systemtap-devel         x86_64   4.4-1.amzn2.0.2            amzn2-core   2.3 M
2022-05-18T04:19:46.3590242Z  trousers                x86_64   0.3.14-2.amzn2.0.2         amzn2-core   294 k
2022-05-18T04:19:46.3590666Z  zlib-devel              x86_64   1.2.7-19.amzn2.0.1         amzn2-core    50 k
2022-05-18T04:19:46.3590878Z 
2022-05-18T04:19:46.3590975Z Transaction Summary
2022-05-18T04:19:46.3591259Z ================================================================================
2022-05-18T04:19:46.3591630Z Install  25 Packages (+42 Dependent packages)
2022-05-18T04:19:46.3591836Z 
2022-05-18T04:19:46.3591965Z Total download size: 96 M
2022-05-18T04:19:46.3592211Z Installed size: 303 M
2022-05-18T04:19:46.3592469Z Downloading packages:
2022-05-18T04:19:46.3618022Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-05-18T04:19:47.7886023Z --------------------------------------------------------------------------------
2022-05-18T04:19:47.7886485Z Total                                               67 MB/s |  96 MB  00:01     
2022-05-18T04:19:47.8937448Z Running transaction check
2022-05-18T04:19:47.9720613Z Running transaction test
2022-05-18T04:19:48.4196557Z Transaction test succeeded
2022-05-18T04:19:48.4199432Z Running transaction
2022-05-18T04:19:48.9518937Z   Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64                               1/67 
2022-05-18T04:19:49.0030833Z   Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64                             2/67 
2022-05-18T04:19:49.0418180Z   Installing : m4-1.4.16-10.amzn2.0.2.x86_64                               3/67 
2022-05-18T04:19:49.0712579Z   Installing : apr-1.7.0-9.amzn2.x86_64                                    4/67 
2022-05-18T04:19:49.0989148Z   Installing : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                       5/67 
2022-05-18T04:19:49.1506055Z   Installing : apr-util-1.6.1-5.amzn2.0.2.x86_64                           6/67 
2022-05-18T04:19:49.1937720Z   Installing : avahi-libs-0.6.31-20.amzn2.x86_64                           7/67 
2022-05-18T04:19:49.2361784Z   Installing : libquadmath-7.3.1-14.amzn2.x86_64                           8/67 
2022-05-18T04:19:49.2606543Z   Installing : patch-2.7.1-12.amzn2.0.2.x86_64                             9/67 
2022-05-18T04:19:49.3426130Z   Installing : perl-Thread-Queue-3.02-2.amzn2.noarch                      10/67 
2022-05-18T04:19:50.4046039Z   Installing : libgfortran-7.3.1-14.amzn2.x86_64                          11/67 
2022-05-18T04:19:50.4678880Z   Installing : cpp-7.3.1-14.amzn2.x86_64                                  12/67 
2022-05-18T04:19:50.5262517Z   Installing : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   13/67 
2022-05-18T04:19:50.5565315Z   Installing : elfutils-0.176-2.amzn2.x86_64                              14/67 
2022-05-18T04:19:50.5834944Z   Installing : dwz-0.11-3.amzn2.0.3.x86_64                                15/67 
2022-05-18T04:19:50.6164526Z   Installing : efivar-libs-31-4.amzn2.0.4.x86_64                          16/67 
2022-05-18T04:19:51.3263440Z   Installing : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        17/67 
2022-05-18T04:19:51.4485748Z   Installing : systemtap-client-4.4-1.amzn2.0.2.x86_64                    18/67 
2022-05-18T04:19:51.6221186Z   Installing : trousers-0.3.14-2.amzn2.0.2.x86_64                         19/67 
2022-05-18T04:19:51.6600002Z   Installing : gnutls-3.3.29-9.amzn2.0.1.x86_64                           20/67 
2022-05-18T04:19:51.6821415Z   Installing : zlib-devel-1.2.7-19.amzn2.0.1.x86_64                       21/67 
2022-05-18T04:19:51.7081459Z   Installing : elfutils-libelf-devel-0.176-2.amzn2.x86_64                 22/67 
2022-05-18T04:19:52.1025206Z   Installing : libcilkrts-7.3.1-14.amzn2.x86_64                           23/67 
2022-05-18T04:19:52.1382602Z   Installing : gdb-8.0.1-36.amzn2.0.1.x86_64                              24/67 
2022-05-18T04:19:52.4342357Z   Installing : libitm-7.3.1-14.amzn2.x86_64                               25/67 
2022-05-18T04:19:52.6156698Z   Installing : kernel-headers-4.14.276-211.499.amzn2.x86_64               26/67 
2022-05-18T04:19:52.7574013Z   Installing : glibc-headers-2.26-58.amzn2.x86_64                         27/67 
2022-05-18T04:19:52.7919750Z   Installing : glibc-devel-2.26-58.amzn2.x86_64                           28/67 
2022-05-18T04:19:52.8215413Z   Installing : libmpx-7.3.1-14.amzn2.x86_64                               29/67 
2022-05-18T04:19:52.8548896Z   Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch                      30/67 
2022-05-18T04:19:52.8816559Z   Installing : system-rpm-config-9.1.0-76.amzn2.0.13.noarch               31/67 
2022-05-18T04:19:52.9056733Z   Installing : go-srpm-macros-3.0.15-23.amzn2.0.1.noarch                  32/67 
2022-05-18T04:19:52.9990974Z   Installing : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  33/67 
2022-05-18T04:19:53.0465152Z   Installing : autoconf-2.69-11.amzn2.noarch                              34/67 
2022-05-18T04:19:53.1258045Z   Installing : gettext-common-devel-0.19.8.1-3.amzn2.noarch               35/67 
2022-05-18T04:19:53.2110601Z   Installing : gettext-devel-0.19.8.1-3.amzn2.x86_64                      36/67 
2022-05-18T04:19:53.3179737Z   Installing : perl-Test-Harness-3.28-3.amzn2.noarch                      37/67 
2022-05-18T04:19:53.3579669Z   Installing : automake-1.13.4-3.1.amzn2.noarch                           38/67 
2022-05-18T04:19:53.3968139Z   Installing : libmodman-2.0.1-8.amzn2.0.2.x86_64                         39/67 
2022-05-18T04:19:53.5152656Z   Installing : libproxy-0.4.11-10.amzn2.0.3.x86_64                        40/67 
2022-05-18T04:19:53.5481618Z   Installing : libsanitizer-7.3.1-14.amzn2.x86_64                         41/67 
2022-05-18T04:19:53.6040055Z   Installing : pakchois-0.4-10.amzn2.0.2.x86_64                           42/67 
2022-05-18T04:19:53.7363433Z   Installing : neon-0.30.0-3.amzn2.0.2.x86_64                             43/67 
2022-05-18T04:19:53.7677817Z   Installing : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 44/67 
2022-05-18T04:19:55.7977007Z   Installing : libatomic-7.3.1-14.amzn2.x86_64                            45/67 
2022-05-18T04:19:59.6807846Z   Installing : gcc-7.3.1-14.amzn2.x86_64                                  46/67 
2022-05-18T04:20:10.6221764Z   Installing : kernel-devel-4.14.276-211.499.amzn2.x86_64                 47/67 
2022-05-18T04:20:10.6641329Z   Installing : systemtap-devel-4.4-1.amzn2.0.2.x86_64                     48/67 
2022-05-18T04:20:11.8882209Z   Installing : systemtap-4.4-1.amzn2.0.2.x86_64                           49/67 
2022-05-18T04:20:11.9954572Z   Installing : gcc-gfortran-7.3.1-14.amzn2.x86_64                         50/67 
2022-05-18T04:20:13.6127314Z   Installing : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        51/67 
2022-05-18T04:20:13.8028451Z   Installing : gcc-c++-7.3.1-14.amzn2.x86_64                              52/67 
2022-05-18T04:20:13.9099642Z   Installing : subversion-1.7.14-16.amzn2.0.1.x86_64                      53/67 
2022-05-18T04:20:13.9485543Z   Installing : intltool-0.50.2-7.amzn2.noarch                             54/67 
2022-05-18T04:20:14.0011830Z   Installing : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       55/67 
2022-05-18T04:20:14.1042843Z   Installing : flex-2.5.37-3.amzn2.0.3.x86_64                             56/67 
2022-05-18T04:20:14.1716481Z   Installing : bison-3.0.4-6.amzn2.0.2.x86_64                             57/67 
2022-05-18T04:20:14.2184580Z   Installing : rcs-5.9.0-5.amzn2.0.2.x86_64                               58/67 
2022-05-18T04:20:14.2759926Z   Installing : indent-2.2.11-13.amzn2.0.2.x86_64                          59/67 
2022-05-18T04:20:14.9687825Z   Installing : patchutils-0.3.3-4.amzn2.0.1.x86_64                        60/67 
2022-05-18T04:20:15.0091114Z   Installing : 1:doxygen-1.8.5-4.amzn2.x86_64                             61/67 
2022-05-18T04:20:15.0583053Z   Installing : diffstat-1.57-4.amzn2.0.2.x86_64                           62/67 
2022-05-18T04:20:15.0953046Z   Installing : cscope-15.8-10.amzn2.0.2.x86_64                            63/67 
2022-05-18T04:20:15.4098939Z   Installing : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      64/67 
2022-05-18T04:20:15.4660135Z   Installing : swig-3.0.12-11.amzn2.0.3.x86_64                            65/67 
2022-05-18T04:20:15.4857980Z   Installing : ctags-5.8-13.amzn2.0.2.x86_64                              66/67 
2022-05-18T04:20:15.5536621Z   Installing : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                        67/67 
2022-05-18T04:20:15.5656713Z   Verifying  : systemtap-4.4-1.amzn2.0.2.x86_64                            1/67 
2022-05-18T04:20:15.5751992Z   Verifying  : perl-Thread-Queue-3.02-2.amzn2.noarch                       2/67 
2022-05-18T04:20:15.5845807Z   Verifying  : gettext-devel-0.19.8.1-3.amzn2.x86_64                       3/67 
2022-05-18T04:20:15.5935875Z   Verifying  : glibc-headers-2.26-58.amzn2.x86_64                          4/67 
2022-05-18T04:20:15.6036770Z   Verifying  : patch-2.7.1-12.amzn2.0.2.x86_64                             5/67 
2022-05-18T04:20:15.6124348Z   Verifying  : flex-2.5.37-3.amzn2.0.3.x86_64                              6/67 
2022-05-18T04:20:15.6213461Z   Verifying  : systemtap-client-4.4-1.amzn2.0.2.x86_64                     7/67 
2022-05-18T04:20:15.6314914Z   Verifying  : libmpc-1.0.1-3.amzn2.0.2.x86_64                             8/67 
2022-05-18T04:20:15.6408953Z   Verifying  : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                         9/67 
2022-05-18T04:20:15.6508324Z   Verifying  : ctags-5.8-13.amzn2.0.2.x86_64                              10/67 
2022-05-18T04:20:15.6590051Z   Verifying  : swig-3.0.12-11.amzn2.0.3.x86_64                            11/67 
2022-05-18T04:20:15.6669759Z   Verifying  : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      12/67 
2022-05-18T04:20:15.6761259Z   Verifying  : libatomic-7.3.1-14.amzn2.x86_64                            13/67 
2022-05-18T04:20:15.6844929Z   Verifying  : pakchois-0.4-10.amzn2.0.2.x86_64                           14/67 
2022-05-18T04:20:15.6928836Z   Verifying  : libgfortran-7.3.1-14.amzn2.x86_64                          15/67 
2022-05-18T04:20:15.7015028Z   Verifying  : go-srpm-macros-3.0.15-23.amzn2.0.1.noarch                  16/67 
2022-05-18T04:20:15.7097533Z   Verifying  : libproxy-0.4.11-10.amzn2.0.3.x86_64                        17/67 
2022-05-18T04:20:15.7191723Z   Verifying  : cscope-15.8-10.amzn2.0.2.x86_64                            18/67 
2022-05-18T04:20:15.7284700Z   Verifying  : diffstat-1.57-4.amzn2.0.2.x86_64                           19/67 
2022-05-18T04:20:15.7380265Z   Verifying  : 1:doxygen-1.8.5-4.amzn2.x86_64                             20/67 
2022-05-18T04:20:15.7466979Z   Verifying  : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        21/67 
2022-05-18T04:20:15.7547515Z   Verifying  : libsanitizer-7.3.1-14.amzn2.x86_64                         22/67 
2022-05-18T04:20:15.7631428Z   Verifying  : gnutls-3.3.29-9.amzn2.0.1.x86_64                           23/67 
2022-05-18T04:20:15.7709865Z   Verifying  : libmodman-2.0.1-8.amzn2.0.2.x86_64                         24/67 
2022-05-18T04:20:15.7803469Z   Verifying  : cpp-7.3.1-14.amzn2.x86_64                                  25/67 
2022-05-18T04:20:15.7895690Z   Verifying  : perl-Test-Harness-3.28-3.amzn2.noarch                      26/67 
2022-05-18T04:20:15.8001545Z   Verifying  : autoconf-2.69-11.amzn2.noarch                              27/67 
2022-05-18T04:20:15.8099093Z   Verifying  : intltool-0.50.2-7.amzn2.noarch                             28/67 
2022-05-18T04:20:15.8244107Z   Verifying  : kernel-devel-4.14.276-211.499.amzn2.x86_64                 29/67 
2022-05-18T04:20:15.8352773Z   Verifying  : apr-util-1.6.1-5.amzn2.0.2.x86_64                          30/67 
2022-05-18T04:20:15.8446456Z   Verifying  : libquadmath-7.3.1-14.amzn2.x86_64                          31/67 
2022-05-18T04:20:15.8532390Z   Verifying  : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       32/67 
2022-05-18T04:20:15.8619852Z   Verifying  : gettext-common-devel-0.19.8.1-3.amzn2.noarch               33/67 
2022-05-18T04:20:15.8716202Z   Verifying  : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  34/67 
2022-05-18T04:20:15.8798045Z   Verifying  : elfutils-libelf-devel-0.176-2.amzn2.x86_64                 35/67 
2022-05-18T04:20:15.8881684Z   Verifying  : perl-srpm-macros-1-8.amzn2.0.1.noarch                      36/67 
2022-05-18T04:20:15.8972127Z   Verifying  : libmpx-7.3.1-14.amzn2.x86_64                               37/67 
2022-05-18T04:20:15.9078338Z   Verifying  : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 38/67 
2022-05-18T04:20:15.9167622Z   Verifying  : automake-1.13.4-3.1.amzn2.noarch                           39/67 
2022-05-18T04:20:15.9259754Z   Verifying  : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                      40/67 
2022-05-18T04:20:15.9357870Z   Verifying  : glibc-devel-2.26-58.amzn2.x86_64                           41/67 
2022-05-18T04:20:15.9441548Z   Verifying  : avahi-libs-0.6.31-20.amzn2.x86_64                          42/67 
2022-05-18T04:20:15.9530092Z   Verifying  : kernel-headers-4.14.276-211.499.amzn2.x86_64               43/67 
2022-05-18T04:20:15.9619441Z   Verifying  : bison-3.0.4-6.amzn2.0.2.x86_64                             44/67 
2022-05-18T04:20:15.9715156Z   Verifying  : libitm-7.3.1-14.amzn2.x86_64                               45/67 
2022-05-18T04:20:15.9831350Z   Verifying  : gdb-8.0.1-36.amzn2.0.1.x86_64                              46/67 
2022-05-18T04:20:15.9921639Z   Verifying  : gcc-7.3.1-14.amzn2.x86_64                                  47/67 
2022-05-18T04:20:16.0015755Z   Verifying  : patchutils-0.3.3-4.amzn2.0.1.x86_64                        48/67 
2022-05-18T04:20:16.0115180Z   Verifying  : gcc-gfortran-7.3.1-14.amzn2.x86_64                         49/67 
2022-05-18T04:20:16.0210732Z   Verifying  : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        50/67 
2022-05-18T04:20:16.0294936Z   Verifying  : indent-2.2.11-13.amzn2.0.2.x86_64                          51/67 
2022-05-18T04:20:16.0409787Z   Verifying  : subversion-1.7.14-16.amzn2.0.1.x86_64                      52/67 
2022-05-18T04:20:16.0508725Z   Verifying  : libcilkrts-7.3.1-14.amzn2.x86_64                           53/67 
2022-05-18T04:20:16.0596874Z   Verifying  : apr-1.7.0-9.amzn2.x86_64                                   54/67 
2022-05-18T04:20:16.0679996Z   Verifying  : system-rpm-config-9.1.0-76.amzn2.0.13.noarch               55/67 
2022-05-18T04:20:16.0768141Z   Verifying  : gcc-c++-7.3.1-14.amzn2.x86_64                              56/67 
2022-05-18T04:20:16.0857077Z   Verifying  : zlib-devel-1.2.7-19.amzn2.0.1.x86_64                       57/67 
2022-05-18T04:20:16.0950876Z   Verifying  : mpfr-3.1.1-4.amzn2.0.2.x86_64                              58/67 
2022-05-18T04:20:16.1034972Z   Verifying  : trousers-0.3.14-2.amzn2.0.2.x86_64                         59/67 
2022-05-18T04:20:16.1128211Z   Verifying  : neon-0.30.0-3.amzn2.0.2.x86_64                             60/67 
2022-05-18T04:20:16.1211300Z   Verifying  : efivar-libs-31-4.amzn2.0.4.x86_64                          61/67 
2022-05-18T04:20:16.1290892Z   Verifying  : dwz-0.11-3.amzn2.0.3.x86_64                                62/67 
2022-05-18T04:20:16.1391668Z   Verifying  : rcs-5.9.0-5.amzn2.0.2.x86_64                               63/67 
2022-05-18T04:20:16.1475921Z   Verifying  : systemtap-devel-4.4-1.amzn2.0.2.x86_64                     64/67 
2022-05-18T04:20:16.1564253Z   Verifying  : elfutils-0.176-2.amzn2.x86_64                              65/67 
2022-05-18T04:20:16.1649279Z   Verifying  : m4-1.4.16-10.amzn2.0.2.x86_64                              66/67 
2022-05-18T04:20:16.2402099Z   Verifying  : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   67/67 
2022-05-18T04:20:16.2402381Z 
2022-05-18T04:20:16.2402486Z Installed:
2022-05-18T04:20:16.2405413Z   autoconf.noarch 0:2.69-11.amzn2                                               
2022-05-18T04:20:16.2405903Z   automake.noarch 0:1.13.4-3.1.amzn2                                            
2022-05-18T04:20:16.2406337Z   bison.x86_64 0:3.0.4-6.amzn2.0.2                                              
2022-05-18T04:20:16.2406753Z   byacc.x86_64 0:1.9.20130304-3.amzn2.0.2                                       
2022-05-18T04:20:16.2407153Z   cscope.x86_64 0:15.8-10.amzn2.0.2                                             
2022-05-18T04:20:16.2407582Z   ctags.x86_64 0:5.8-13.amzn2.0.2                                               
2022-05-18T04:20:16.2407988Z   diffstat.x86_64 0:1.57-4.amzn2.0.2                                            
2022-05-18T04:20:16.2408904Z   doxygen.x86_64 1:1.8.5-4.amzn2                                                
2022-05-18T04:20:16.2409327Z   elfutils.x86_64 0:0.176-2.amzn2                                               
2022-05-18T04:20:16.2410049Z   flex.x86_64 0:2.5.37-3.amzn2.0.3                                              
2022-05-18T04:20:16.2410464Z   gcc.x86_64 0:7.3.1-14.amzn2                                                   
2022-05-18T04:20:16.2410868Z   gcc-c++.x86_64 0:7.3.1-14.amzn2                                               
2022-05-18T04:20:16.2413393Z   gcc-gfortran.x86_64 0:7.3.1-14.amzn2                                          
2022-05-18T04:20:16.2413850Z   indent.x86_64 0:2.2.11-13.amzn2.0.2                                           
2022-05-18T04:20:16.2414293Z   intltool.noarch 0:0.50.2-7.amzn2                                              
2022-05-18T04:20:16.2414732Z   libtool.x86_64 0:2.4.2-22.2.amzn2.0.2                                         
2022-05-18T04:20:16.2415143Z   patch.x86_64 0:2.7.1-12.amzn2.0.2                                             
2022-05-18T04:20:16.2415570Z   patchutils.x86_64 0:0.3.3-4.amzn2.0.1                                         
2022-05-18T04:20:16.2415985Z   rcs.x86_64 0:5.9.0-5.amzn2.0.2                                                
2022-05-18T04:20:16.2416401Z   rpm-build.x86_64 0:4.11.3-48.amzn2.0.2                                        
2022-05-18T04:20:16.2416812Z   rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2                                         
2022-05-18T04:20:16.2417269Z   subversion.x86_64 0:1.7.14-16.amzn2.0.1                                       
2022-05-18T04:20:16.2417681Z   swig.x86_64 0:3.0.12-11.amzn2.0.3                                             
2022-05-18T04:20:16.2418120Z   system-rpm-config.noarch 0:9.1.0-76.amzn2.0.13                                
2022-05-18T04:20:16.2418772Z   systemtap.x86_64 0:4.4-1.amzn2.0.2                                            
2022-05-18T04:20:16.2418976Z 
2022-05-18T04:20:16.2419099Z Dependency Installed:
2022-05-18T04:20:16.2419490Z   apr.x86_64 0:1.7.0-9.amzn2                                                    
2022-05-18T04:20:16.2419886Z   apr-util.x86_64 0:1.6.1-5.amzn2.0.2                                           
2022-05-18T04:20:16.2420313Z   apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2                                       
2022-05-18T04:20:16.2420741Z   avahi-libs.x86_64 0:0.6.31-20.amzn2                                           
2022-05-18T04:20:16.2421154Z   cpp.x86_64 0:7.3.1-14.amzn2                                                   
2022-05-18T04:20:16.2421542Z   dwz.x86_64 0:0.11-3.amzn2.0.3                                                 
2022-05-18T04:20:16.2421949Z   efivar-libs.x86_64 0:31-4.amzn2.0.4                                           
2022-05-18T04:20:16.2422389Z   elfutils-libelf-devel.x86_64 0:0.176-2.amzn2                                  
2022-05-18T04:20:16.2422809Z   gdb.x86_64 0:8.0.1-36.amzn2.0.1                                               
2022-05-18T04:20:16.2423245Z   gettext-common-devel.noarch 0:0.19.8.1-3.amzn2                                
2022-05-18T04:20:16.2423692Z   gettext-devel.x86_64 0:0.19.8.1-3.amzn2                                       
2022-05-18T04:20:16.2424114Z   glibc-devel.x86_64 0:2.26-58.amzn2                                            
2022-05-18T04:20:16.2424519Z   glibc-headers.x86_64 0:2.26-58.amzn2                                          
2022-05-18T04:20:16.2424930Z   gnutls.x86_64 0:3.3.29-9.amzn2.0.1                                            
2022-05-18T04:20:16.2425354Z   go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.1                                   
2022-05-18T04:20:16.2425776Z   kernel-devel.x86_64 0:4.14.276-211.499.amzn2                                  
2022-05-18T04:20:16.2426201Z   kernel-headers.x86_64 0:4.14.276-211.499.amzn2                                
2022-05-18T04:20:16.2426628Z   libatomic.x86_64 0:7.3.1-14.amzn2                                             
2022-05-18T04:20:16.2427037Z   libcilkrts.x86_64 0:7.3.1-14.amzn2                                            
2022-05-18T04:20:16.2427572Z   libgfortran.x86_64 0:7.3.1-14.amzn2                                           
2022-05-18T04:20:16.2427996Z   libitm.x86_64 0:7.3.1-14.amzn2                                                
2022-05-18T04:20:16.2428402Z   libmodman.x86_64 0:2.0.1-8.amzn2.0.2                                          
2022-05-18T04:20:16.2428793Z   libmpc.x86_64 0:1.0.1-3.amzn2.0.2                                             
2022-05-18T04:20:16.2429196Z   libmpx.x86_64 0:7.3.1-14.amzn2                                                
2022-05-18T04:20:16.2429595Z   libproxy.x86_64 0:0.4.11-10.amzn2.0.3                                         
2022-05-18T04:20:16.2430006Z   libquadmath.x86_64 0:7.3.1-14.amzn2                                           
2022-05-18T04:20:16.2430414Z   libsanitizer.x86_64 0:7.3.1-14.amzn2                                          
2022-05-18T04:20:16.2430820Z   m4.x86_64 0:1.4.16-10.amzn2.0.2                                               
2022-05-18T04:20:16.2431218Z   mokutil.x86_64 1:0.3.0-10.amzn2.0.1                                           
2022-05-18T04:20:16.2431619Z   mpfr.x86_64 0:3.1.1-4.amzn2.0.2                                               
2022-05-18T04:20:16.2432013Z   neon.x86_64 0:0.30.0-3.amzn2.0.2                                              
2022-05-18T04:20:16.2432416Z   pakchois.x86_64 0:0.4-10.amzn2.0.2                                            
2022-05-18T04:20:16.2432844Z   perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2                                   
2022-05-18T04:20:16.2433278Z   perl-Test-Harness.noarch 0:3.28-3.amzn2                                       
2022-05-18T04:20:16.2433779Z   perl-Thread-Queue.noarch 0:3.02-2.amzn2                                       
2022-05-18T04:20:16.2434392Z   perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2                                    
2022-05-18T04:20:16.2435009Z   perl-srpm-macros.noarch 0:1-8.amzn2.0.1                                       
2022-05-18T04:20:16.2435546Z   subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1                                  
2022-05-18T04:20:16.2436018Z   systemtap-client.x86_64 0:4.4-1.amzn2.0.2                                     
2022-05-18T04:20:16.2436525Z   systemtap-devel.x86_64 0:4.4-1.amzn2.0.2                                      
2022-05-18T04:20:16.2437018Z   trousers.x86_64 0:0.3.14-2.amzn2.0.2                                          
2022-05-18T04:20:16.2437495Z   zlib-devel.x86_64 0:1.2.7-19.amzn2.0.1                                        
2022-05-18T04:20:16.2437736Z 
2022-05-18T04:20:16.2437882Z Complete!
2022-05-18T04:20:16.2795310Z ++ uname -r
2022-05-18T04:20:16.2801417Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.252-195.483.amzn2.x86_64'
2022-05-18T04:20:16.7849326Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:20:16.8008292Z Existing lock /var/run/yum.pid: another copy is running as pid 34778.
2022-05-18T04:20:16.8009054Z Another app is currently holding the yum lock; waiting for it to exit...
2022-05-18T04:20:16.8018260Z   The other application is: yum
2022-05-18T04:20:16.8019038Z     Memory :  92 M RSS (309 MB VSZ)
2022-05-18T04:20:16.8020235Z     Started: Wed May 18 04:20:15 2022 - 00:01 ago
2022-05-18T04:20:16.8020903Z     State  : Running, pid: 34778
2022-05-18T04:20:18.8046414Z Another app is currently holding the yum lock; waiting for it to exit...
2022-05-18T04:20:18.8054062Z   The other application is: yum
2022-05-18T04:20:18.8054973Z     Memory : 155 M RSS (373 MB VSZ)
2022-05-18T04:20:18.8056038Z     Started: Wed May 18 04:20:15 2022 - 00:03 ago
2022-05-18T04:20:18.8056718Z     State  : Running, pid: 34778
2022-05-18T04:20:22.0680863Z Resolving Dependencies
2022-05-18T04:20:22.0688174Z --> Running transaction check
2022-05-18T04:20:22.0689207Z ---> Package kernel-devel.x86_64 0:4.14.252-195.483.amzn2 will be installed
2022-05-18T04:20:22.4025253Z --> Finished Dependency Resolution
2022-05-18T04:20:22.5168462Z 
2022-05-18T04:20:22.5169299Z Dependencies Resolved
2022-05-18T04:20:22.5175316Z 
2022-05-18T04:20:22.5176070Z ================================================================================
2022-05-18T04:20:22.5177002Z  Package          Arch       Version                       Repository      Size
2022-05-18T04:20:22.5177700Z ================================================================================
2022-05-18T04:20:22.5178092Z Installing:
2022-05-18T04:20:22.5179342Z  kernel-devel     x86_64     4.14.252-195.483.amzn2        amzn2-core      13 M
2022-05-18T04:20:22.5179877Z 
2022-05-18T04:20:22.5180137Z Transaction Summary
2022-05-18T04:20:22.5180760Z ================================================================================
2022-05-18T04:20:22.5181449Z Install  1 Package
2022-05-18T04:20:22.5181673Z 
2022-05-18T04:20:22.5181905Z Total download size: 13 M
2022-05-18T04:20:22.5182205Z Installed size: 60 M
2022-05-18T04:20:22.5182537Z Downloading packages:
2022-05-18T04:20:22.5190420Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-05-18T04:20:22.8139233Z Running transaction check
2022-05-18T04:20:22.8327061Z Running transaction test
2022-05-18T04:20:23.2416255Z Transaction test succeeded
2022-05-18T04:20:23.2419117Z Running transaction
2022-05-18T04:20:38.6757529Z   Installing : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-05-18T04:20:38.7593188Z   Verifying  : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-05-18T04:20:38.7593550Z 
2022-05-18T04:20:38.7593643Z Installed:
2022-05-18T04:20:38.7594254Z   kernel-devel.x86_64 0:4.14.252-195.483.amzn2                                  
2022-05-18T04:20:38.7594526Z 
2022-05-18T04:20:38.7594679Z Complete!
2022-05-18T04:20:38.7961319Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-510.60.02.run
2022-05-18T04:20:42.1103184Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm
2022-05-18T04:20:43.3511990Z Verifying archive integrity... OK
2022-05-18T04:21:07.7776859Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 510.60.02..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2022-05-18T04:21:07.9156731Z 
2022-05-18T04:21:07.9158037Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver.
2022-05-18T04:21:07.9158678Z 
2022-05-18T04:21:22.9427662Z 
2022-05-18T04:21:22.9429654Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system.  If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver.
2022-05-18T04:21:22.9430298Z 
2022-05-18T04:21:30.5639155Z + sudo rm -fv /tmp/nvidia_driver
2022-05-18T04:21:30.6439329Z removed ‘/tmp/nvidia_driver’
2022-05-18T04:21:30.6454055Z + nvidia-smi
2022-05-18T04:21:35.1958253Z Wed May 18 04:21:35 2022       
2022-05-18T04:21:35.1958857Z +-----------------------------------------------------------------------------+
2022-05-18T04:21:35.1962104Z | NVIDIA-SMI 510.60.02    Driver Version: 510.60.02    CUDA Version: 11.6     |
2022-05-18T04:21:35.1962880Z |-------------------------------+----------------------+----------------------+
2022-05-18T04:21:35.1963436Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2022-05-18T04:21:35.1963921Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2022-05-18T04:21:35.1964551Z |                               |                      |               MIG M. |
2022-05-18T04:21:35.1964873Z |===============================+======================+======================|
2022-05-18T04:21:35.2008426Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |                    0 |
2022-05-18T04:21:35.2008804Z | N/A   30C    P0    38W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-05-18T04:21:35.2009130Z |                               |                      |                  N/A |
2022-05-18T04:21:35.2009866Z +-------------------------------+----------------------+----------------------+
2022-05-18T04:21:35.2057023Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |                    0 |
2022-05-18T04:21:35.2057401Z | N/A   25C    P0    37W / 150W |      0MiB /  7680MiB |     99%      Default |
2022-05-18T04:21:35.2057720Z |                               |                      |                  N/A |
2022-05-18T04:21:35.2058195Z +-------------------------------+----------------------+----------------------+
2022-05-18T04:21:35.2058552Z                                                                                
2022-05-18T04:21:35.2058982Z +-----------------------------------------------------------------------------+
2022-05-18T04:21:35.2059360Z | Processes:                                                                  |
2022-05-18T04:21:35.2059707Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2022-05-18T04:21:35.2060031Z |        ID   ID                                                   Usage      |
2022-05-18T04:21:35.2060341Z |=============================================================================|
2022-05-18T04:21:35.2063107Z |  No running processes found                                                 |
2022-05-18T04:21:35.2063835Z +-----------------------------------------------------------------------------+
2022-05-18T04:21:35.7471939Z + echo 'GPU_FLAG=--gpus all'
2022-05-18T04:21:36.0415012Z Command completed after 1 attempt(s).
2022-05-18T04:21:36.0415225Z 
2022-05-18T04:21:36.0487148Z Prepare all required actions
2022-05-18T04:21:36.0487495Z Getting action download info
2022-05-18T04:21:36.2648188Z Download action repository 'seemethere/download-artifact-s3@v3' (SHA:64048a097659c8ca71ceacbb3c01cee9ed6f1b05)
2022-05-18T04:21:36.4336907Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869)
2022-05-18T04:21:36.5541573Z ##[group]Run ./.github/actions/download-build-artifacts
2022-05-18T04:21:36.5541863Z with:
2022-05-18T04:21:36.5542145Z   name: linux-xenial-cuda11.3-py3.7-gcc7
2022-05-18T04:21:36.5542420Z env:
2022-05-18T04:21:36.5542620Z   IN_CI: 1
2022-05-18T04:21:36.5542842Z   IS_GHA: 1
2022-05-18T04:21:36.5543104Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:21:36.5543351Z   GPU_FLAG: --gpus all
2022-05-18T04:21:36.5543598Z ##[endgroup]
2022-05-18T04:21:36.5571702Z ##[group]Run seemethere/download-artifact-s3@v3
2022-05-18T04:21:36.5571995Z with:
2022-05-18T04:21:36.5572308Z   name: linux-xenial-cuda11.3-py3.7-gcc7
2022-05-18T04:21:36.5572599Z   s3-bucket: gha-artifacts
2022-05-18T04:21:36.5572863Z   region: us-east-1
2022-05-18T04:21:36.5573093Z env:
2022-05-18T04:21:36.5573283Z   IN_CI: 1
2022-05-18T04:21:36.5573503Z   IS_GHA: 1
2022-05-18T04:21:36.5573752Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:21:36.5573995Z   GPU_FLAG: --gpus all
2022-05-18T04:21:36.5574237Z ##[endgroup]
2022-05-18T04:21:37.0642165Z Found 1 objects with prefix pytorch/pytorch/2342799944/1/linux-xenial-cuda11.3-py3.7-gcc7/
2022-05-18T04:21:37.0642785Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-05-18T04:21:52.3352375Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-05-18T04:21:52.3352732Z 
2022-05-18T04:21:52.3353395Z Artifact download has finished successfully
2022-05-18T04:21:52.3504183Z ##[group]Run unzip -o artifacts.zip
2022-05-18T04:21:52.3504502Z unzip -o artifacts.zip
2022-05-18T04:21:52.3517814Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:21:52.3518112Z env:
2022-05-18T04:21:52.3518333Z   IN_CI: 1
2022-05-18T04:21:52.3518540Z   IS_GHA: 1
2022-05-18T04:21:52.3518789Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:21:52.3519057Z   GPU_FLAG: --gpus all
2022-05-18T04:21:52.3519288Z ##[endgroup]
2022-05-18T04:21:52.3593984Z Archive:  artifacts.zip
2022-05-18T04:21:52.3595905Z    creating: dist/
2022-05-18T04:21:54.8273842Z   inflating: dist/torch-1.12.0a0+git3b23752-cp37-cp37m-linux_x86_64.whl  
2022-05-18T04:21:54.8274276Z    creating: build/custom_test_artifacts/
2022-05-18T04:21:54.8274680Z    creating: build/custom_test_artifacts/custom-op-build/
2022-05-18T04:21:54.8275163Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-05-18T04:21:54.8281414Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-05-18T04:21:54.8281953Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/
2022-05-18T04:21:54.8282499Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeSystem.cmake  
2022-05-18T04:21:54.8283060Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdC/
2022-05-18T04:21:54.8283614Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdC/tmp/
2022-05-18T04:21:54.8285299Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdC/CMakeCCompilerId.c  
2022-05-18T04:21:54.8286572Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdC/a.out  
2022-05-18T04:21:54.8287136Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCXX/
2022-05-18T04:21:54.8287923Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCXX/tmp/
2022-05-18T04:21:54.8289933Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-05-18T04:21:54.8291400Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCXX/a.out  
2022-05-18T04:21:54.8292943Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_C.bin  
2022-05-18T04:21:54.8293873Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeCCompiler.cmake  
2022-05-18T04:21:54.8295322Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_CXX.bin  
2022-05-18T04:21:54.8296357Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeCXXCompiler.cmake  
2022-05-18T04:21:54.8296955Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/
2022-05-18T04:21:54.8297540Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/
2022-05-18T04:21:54.8347319Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-05-18T04:21:54.8348049Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-05-18T04:21:54.8348760Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-05-18T04:21:54.8349500Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-05-18T04:21:54.8350220Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-05-18T04:21:54.8350919Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-05-18T04:21:54.8351606Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-05-18T04:21:54.8352305Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-05-18T04:21:54.8353256Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-05-18T04:21:54.8390462Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-05-18T04:21:54.8427855Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-05-18T04:21:54.8428861Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-05-18T04:21:54.8429551Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-05-18T04:21:54.8430392Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-05-18T04:21:54.8431182Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-05-18T04:21:54.8432166Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-05-18T04:21:54.8433280Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.o  
2022-05-18T04:21:54.8435169Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-05-18T04:21:54.8502763Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/a.out  
2022-05-18T04:21:54.8570355Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_CUDA.bin  
2022-05-18T04:21:54.8571165Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeCUDACompiler.cmake  
2022-05-18T04:21:54.8571718Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-05-18T04:21:54.8572338Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/feature_tests.c  
2022-05-18T04:21:54.8573373Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/feature_tests.cxx  
2022-05-18T04:21:54.8575448Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/feature_tests.bin  
2022-05-18T04:21:54.8575996Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-05-18T04:21:54.8576551Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-05-18T04:21:54.8577089Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-05-18T04:21:54.8601120Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-05-18T04:21:54.8601713Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-05-18T04:21:54.8602307Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-05-18T04:21:54.8603333Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-05-18T04:21:54.8603952Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-05-18T04:21:54.8604685Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-05-18T04:21:54.8605276Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-05-18T04:21:54.8662642Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/CXX.includecache  
2022-05-18T04:21:54.8680613Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.internal  
2022-05-18T04:21:54.8789051Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-05-18T04:21:54.8789617Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-05-18T04:21:54.8816469Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-05-18T04:21:54.8817067Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-05-18T04:21:54.8817656Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-05-18T04:21:54.8818648Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-05-18T04:21:54.8819304Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-05-18T04:21:54.8820044Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-05-18T04:21:54.8820811Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-05-18T04:21:54.8877819Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/CXX.includecache  
2022-05-18T04:21:54.8896115Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.internal  
2022-05-18T04:21:54.8975030Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-05-18T04:21:54.8975668Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-05-18T04:21:54.8976281Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-05-18T04:21:54.8976847Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-05-18T04:21:54.8977776Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-05-18T04:21:54.8979240Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-05-18T04:21:54.8979790Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2022-05-18T04:21:54.8983034Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-05-18T04:21:54.8983533Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-05-18T04:21:54.8984234Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-05-18T04:21:54.9073521Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-05-18T04:21:54.9134340Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-05-18T04:21:54.9134813Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-05-18T04:21:54.9135255Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-05-18T04:21:54.9140415Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-05-18T04:21:54.9140953Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/
2022-05-18T04:21:54.9141499Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeSystem.cmake  
2022-05-18T04:21:54.9142035Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdC/
2022-05-18T04:21:54.9142583Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdC/tmp/
2022-05-18T04:21:54.9143876Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdC/CMakeCCompilerId.c  
2022-05-18T04:21:54.9145514Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdC/a.out  
2022-05-18T04:21:54.9146053Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCXX/
2022-05-18T04:21:54.9146611Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCXX/tmp/
2022-05-18T04:21:54.9148073Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-05-18T04:21:54.9149714Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCXX/a.out  
2022-05-18T04:21:54.9151046Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_C.bin  
2022-05-18T04:21:54.9151792Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeCCompiler.cmake  
2022-05-18T04:21:54.9153757Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_CXX.bin  
2022-05-18T04:21:54.9154517Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeCXXCompiler.cmake  
2022-05-18T04:21:54.9155103Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/
2022-05-18T04:21:54.9155661Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/
2022-05-18T04:21:54.9205373Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-05-18T04:21:54.9206090Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-05-18T04:21:54.9206801Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-05-18T04:21:54.9207519Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-05-18T04:21:54.9208215Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-05-18T04:21:54.9208909Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-05-18T04:21:54.9209804Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-05-18T04:21:54.9210688Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-05-18T04:21:54.9211513Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-05-18T04:21:54.9248711Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-05-18T04:21:54.9285373Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-05-18T04:21:54.9286345Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-05-18T04:21:54.9287034Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-05-18T04:21:54.9287639Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-05-18T04:21:54.9288536Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-05-18T04:21:54.9289701Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-05-18T04:21:54.9290948Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.o  
2022-05-18T04:21:54.9292468Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-05-18T04:21:54.9360129Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/a.out  
2022-05-18T04:21:54.9427866Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_CUDA.bin  
2022-05-18T04:21:54.9428492Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeCUDACompiler.cmake  
2022-05-18T04:21:54.9429042Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-05-18T04:21:54.9429547Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/feature_tests.c  
2022-05-18T04:21:54.9430670Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/feature_tests.cxx  
2022-05-18T04:21:54.9432677Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/feature_tests.bin  
2022-05-18T04:21:54.9433223Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-05-18T04:21:54.9433858Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-05-18T04:21:54.9434421Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-05-18T04:21:54.9461392Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-05-18T04:21:54.9461981Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-05-18T04:21:54.9463057Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-05-18T04:21:54.9464200Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-05-18T04:21:54.9464943Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-05-18T04:21:54.9465711Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-05-18T04:21:54.9466312Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-05-18T04:21:54.9523323Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/CXX.includecache  
2022-05-18T04:21:54.9541464Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.internal  
2022-05-18T04:21:54.9603637Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-05-18T04:21:54.9604413Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-05-18T04:21:54.9605087Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-05-18T04:21:54.9605643Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-05-18T04:21:54.9606401Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-05-18T04:21:54.9607336Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-05-18T04:21:54.9607967Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2022-05-18T04:21:54.9610922Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-05-18T04:21:54.9611509Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-05-18T04:21:54.9612497Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-05-18T04:21:54.9660713Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-05-18T04:21:54.9661192Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-05-18T04:21:54.9661680Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-05-18T04:21:54.9666944Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-05-18T04:21:54.9667506Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/
2022-05-18T04:21:54.9668072Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeSystem.cmake  
2022-05-18T04:21:54.9668639Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdC/
2022-05-18T04:21:54.9669219Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdC/tmp/
2022-05-18T04:21:54.9670411Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdC/CMakeCCompilerId.c  
2022-05-18T04:21:54.9671736Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdC/a.out  
2022-05-18T04:21:54.9672327Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCXX/
2022-05-18T04:21:54.9672896Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCXX/tmp/
2022-05-18T04:21:54.9675054Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-05-18T04:21:54.9676114Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCXX/a.out  
2022-05-18T04:21:54.9678026Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_C.bin  
2022-05-18T04:21:54.9678656Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeCCompiler.cmake  
2022-05-18T04:21:54.9680137Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_CXX.bin  
2022-05-18T04:21:54.9681195Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeCXXCompiler.cmake  
2022-05-18T04:21:54.9681809Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/
2022-05-18T04:21:54.9682383Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/
2022-05-18T04:21:54.9732143Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-05-18T04:21:54.9732883Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-05-18T04:21:54.9733626Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-05-18T04:21:54.9734516Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-05-18T04:21:54.9735313Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-05-18T04:21:54.9736051Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-05-18T04:21:54.9736766Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-05-18T04:21:54.9737479Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-05-18T04:21:54.9738180Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-05-18T04:21:54.9775439Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-05-18T04:21:54.9812069Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-05-18T04:21:54.9813035Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-05-18T04:21:54.9813926Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-05-18T04:21:54.9814718Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-05-18T04:21:54.9815423Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-05-18T04:21:54.9816564Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-05-18T04:21:54.9817531Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.o  
2022-05-18T04:21:54.9818977Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-05-18T04:21:54.9886784Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/a.out  
2022-05-18T04:21:54.9954735Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_CUDA.bin  
2022-05-18T04:21:54.9955410Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeCUDACompiler.cmake  
2022-05-18T04:21:54.9955991Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-05-18T04:21:54.9956550Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/feature_tests.c  
2022-05-18T04:21:54.9957280Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/feature_tests.cxx  
2022-05-18T04:21:54.9959271Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/feature_tests.bin  
2022-05-18T04:21:54.9959842Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-05-18T04:21:54.9960416Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-05-18T04:21:54.9960974Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-05-18T04:21:54.9989723Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-05-18T04:21:54.9990363Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-05-18T04:21:54.9991010Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-05-18T04:21:54.9991914Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-05-18T04:21:54.9992707Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-05-18T04:21:54.9993352Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-05-18T04:21:54.9994071Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-05-18T04:21:55.0051383Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/CXX.includecache  
2022-05-18T04:21:55.0069327Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.internal  
2022-05-18T04:21:55.0125547Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-05-18T04:21:55.0126163Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-05-18T04:21:55.0131277Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-05-18T04:21:55.0131881Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-05-18T04:21:55.0132512Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-05-18T04:21:55.0133447Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-05-18T04:21:55.0134222Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-05-18T04:21:55.0134964Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-05-18T04:21:55.0135594Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-05-18T04:21:55.0143213Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/CXX.includecache  
2022-05-18T04:21:55.0146882Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.internal  
2022-05-18T04:21:55.0290549Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-05-18T04:21:55.0291193Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-05-18T04:21:55.0291825Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-05-18T04:21:55.0292409Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-05-18T04:21:55.0293659Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-05-18T04:21:55.0295011Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-05-18T04:21:55.0295834Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2022-05-18T04:21:55.0298576Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-05-18T04:21:55.0299174Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-05-18T04:21:55.0300100Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-05-18T04:21:55.0417498Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-05-18T04:21:55.0461849Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-05-18T04:21:55.0462214Z    creating: build/lib/
2022-05-18T04:21:55.0462854Z   inflating: build/lib/libclog.a     
2022-05-18T04:21:55.0527332Z   inflating: build/lib/libgtest.a    
2022-05-18T04:21:55.0537687Z   inflating: build/lib/libpthreadpool.a  
2022-05-18T04:21:55.0625298Z   inflating: build/lib/libbenchmark.a  
2022-05-18T04:21:55.0731077Z   inflating: build/lib/libprotobuf-lite.a  
2022-05-18T04:21:55.0762665Z   inflating: build/lib/libtensorpipe_uv.a  
2022-05-18T04:21:55.0818527Z   inflating: build/lib/libasmjit.a   
2022-05-18T04:21:55.0950984Z   inflating: build/lib/libgloo.a     
2022-05-18T04:21:55.1482638Z   inflating: build/lib/libprotobuf.a  
2022-05-18T04:21:55.1502423Z   inflating: build/lib/libfmt.a      
2022-05-18T04:21:55.1503043Z   inflating: build/lib/libfoxi_loader.a  
2022-05-18T04:21:55.1569470Z   inflating: build/lib/libc10.so     
2022-05-18T04:21:55.1570668Z   inflating: build/lib/libtorch_global_deps.so  
2022-05-18T04:21:55.1572834Z   inflating: build/lib/libcaffe2_nvrtc.so  
2022-05-18T04:21:55.1582314Z   inflating: build/lib/libcpuinfo.a  
2022-05-18T04:21:55.1591293Z   inflating: build/lib/libcpuinfo_internals.a  
2022-05-18T04:21:55.1607035Z   inflating: build/lib/libqnnpack.a  
2022-05-18T04:21:55.2174983Z   inflating: build/lib/libprotoc.a   
2022-05-18T04:21:55.2198674Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-05-18T04:21:55.2201225Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-05-18T04:21:55.2219959Z   inflating: build/lib/libgmock.a    
2022-05-18T04:21:55.2220470Z   inflating: build/lib/libgtest_main.a  
2022-05-18T04:21:55.2221487Z   inflating: build/lib/libbenchmark_main.a  
2022-05-18T04:21:56.0295871Z   inflating: build/lib/libdnnl.a     
2022-05-18T04:21:56.0318783Z   inflating: build/lib/libnnpack.a   
2022-05-18T04:21:56.0970362Z   inflating: build/lib/libtensorpipe.a  
2022-05-18T04:21:56.1014343Z   inflating: build/lib/libc10_cuda.so  
2022-05-18T04:21:56.2527611Z   inflating: build/lib/libfbgemm.a   
2022-05-18T04:21:56.2528084Z   inflating: build/lib/libgmock_main.a  
2022-05-18T04:21:56.2953911Z   inflating: build/lib/libkineto.a   
2022-05-18T04:21:56.4077296Z   inflating: build/lib/libdnnl_graph.a  
2022-05-18T04:21:56.4122627Z   inflating: build/lib/libcaffe2_protos.a  
2022-05-18T04:21:56.4170292Z   inflating: build/lib/libonnx_proto.a  
2022-05-18T04:21:56.4458887Z   inflating: build/lib/libtensorpipe_cuda.a  
2022-05-18T04:21:56.5119534Z   inflating: build/lib/libonnx.a     
2022-05-18T04:21:56.5532804Z   inflating: build/lib/libgloo_cuda.a  
2022-05-18T04:21:56.5547457Z   inflating: build/lib/libtest_deploy_lib.so  
2022-05-18T04:21:57.0740914Z   inflating: build/lib/libtorch_python_static.a  
2022-05-18T04:21:57.7419379Z   inflating: build/lib/libtorch_deployinterpreter.so  
2022-05-18T04:21:59.8267890Z   inflating: build/lib/libtorch_cpu.so  
2022-05-18T04:22:00.2669208Z   inflating: build/lib/libtorch_cuda_cpp.so  
2022-05-18T04:22:01.8634121Z   inflating: build/lib/libtorch_cuda_cu.so  
2022-05-18T04:22:01.8635222Z   inflating: build/lib/libtorch_cuda.so  
2022-05-18T04:22:01.8636807Z   inflating: build/lib/libtorch.so   
2022-05-18T04:22:01.8640439Z   inflating: build/lib/libc10d_cuda_test.so  
2022-05-18T04:22:02.8486793Z   inflating: build/lib/libtorch_cuda_linalg.so  
2022-05-18T04:22:02.8510182Z   inflating: build/lib/libjitbackend_test.so  
2022-05-18T04:22:02.8540905Z   inflating: build/lib/libbackend_with_compiler.so  
2022-05-18T04:22:02.8593800Z   inflating: build/lib/libtorchbind_test.so  
2022-05-18T04:22:02.8598609Z   inflating: build/lib/libshm.so     
2022-05-18T04:22:03.5329098Z   inflating: build/lib/libtorch_deploy_internal.a  
2022-05-18T04:22:03.6911774Z   inflating: build/lib/libtorch_python.so  
2022-05-18T04:22:03.6949732Z   inflating: build/lib/libnnapi_backend.so  
2022-05-18T04:22:03.6950035Z    creating: build/bin/
2022-05-18T04:22:03.6962695Z   inflating: build/bin/remove_dt_needed  
2022-05-18T04:22:03.7019316Z   inflating: build/bin/c10_registry_test  
2022-05-18T04:22:03.7096035Z   inflating: build/bin/c10_optional_test  
2022-05-18T04:22:03.7266599Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-05-18T04:22:03.7317734Z   inflating: build/bin/c10_flags_test  
2022-05-18T04:22:03.7371306Z   inflating: build/bin/c10_exception_test  
2022-05-18T04:22:03.7430466Z   inflating: build/bin/c10_logging_test  
2022-05-18T04:22:03.7542721Z   inflating: build/bin/c10_either_test  
2022-05-18T04:22:03.7599589Z   inflating: build/bin/c10_complex_test  
2022-05-18T04:22:03.7651426Z   inflating: build/bin/c10_irange_test  
2022-05-18T04:22:03.7708321Z   inflating: build/bin/c10_bfloat16_test  
2022-05-18T04:22:03.7768454Z   inflating: build/bin/c10_string_view_test  
2022-05-18T04:22:03.7821649Z   inflating: build/bin/c10_accumulate_test  
2022-05-18T04:22:03.7876800Z   inflating: build/bin/c10_complex_math_test  
2022-05-18T04:22:03.7931186Z   inflating: build/bin/c10_Bitset_test  
2022-05-18T04:22:03.7988421Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-05-18T04:22:03.8135950Z   inflating: build/bin/c10_SmallVectorTest  
2022-05-18T04:22:03.8192794Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-05-18T04:22:03.8250832Z   inflating: build/bin/c10_typeid_test  
2022-05-18T04:22:03.8309367Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-05-18T04:22:03.8361927Z   inflating: build/bin/c10_tempfile_test  
2022-05-18T04:22:03.8412440Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-05-18T04:22:03.8461250Z   inflating: build/bin/c10_StreamGuard_test  
2022-05-18T04:22:03.8524569Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-05-18T04:22:03.8584212Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-05-18T04:22:03.8642514Z   inflating: build/bin/c10_ThreadLocal_test  
2022-05-18T04:22:03.8695650Z   inflating: build/bin/c10_DeviceGuard_test  
2022-05-18T04:22:03.8747673Z   inflating: build/bin/c10_C++17_test  
2022-05-18T04:22:03.8799681Z   inflating: build/bin/c10_Device_test  
2022-05-18T04:22:03.8848755Z   inflating: build/bin/c10_TypeTraits_test  
2022-05-18T04:22:03.8899114Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-05-18T04:22:03.8949864Z   inflating: build/bin/c10_Half_test  
2022-05-18T04:22:03.9007509Z   inflating: build/bin/c10_LeftRight_test  
2022-05-18T04:22:03.9056630Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-05-18T04:22:03.9120527Z   inflating: build/bin/c10_Metaprogramming_test  
2022-05-18T04:22:03.9169690Z   inflating: build/bin/c10_Array_test  
2022-05-18T04:22:03.9221067Z   inflating: build/bin/c10_Synchronized_test  
2022-05-18T04:22:03.9274700Z   inflating: build/bin/c10_TypeIndex_test  
2022-05-18T04:22:03.9326367Z   inflating: build/bin/c10_TypeList_test  
2022-05-18T04:22:03.9383913Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-05-18T04:22:03.9899495Z   inflating: build/bin/protoc-3.13.0.0  
2022-05-18T04:22:04.0413873Z   inflating: build/bin/protoc        
2022-05-18T04:22:04.0463373Z   inflating: build/bin/c10_cuda_CUDATest  
2022-05-18T04:22:04.0770670Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-05-18T04:22:04.1113369Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-05-18T04:22:04.1168324Z   inflating: build/bin/HashStoreTest  
2022-05-18T04:22:04.1223279Z   inflating: build/bin/FileStoreTest  
2022-05-18T04:22:04.1285248Z   inflating: build/bin/TCPStoreTest  
2022-05-18T04:22:04.1300148Z   inflating: build/bin/ProcessGroupMPITest  
2022-05-18T04:22:04.1369643Z   inflating: build/bin/cuda_cub_test  
2022-05-18T04:22:04.1372995Z   inflating: build/bin/example_allreduce  
2022-05-18T04:22:04.1423127Z   inflating: build/bin/cuda_cudnn_test  
2022-05-18T04:22:04.1485954Z   inflating: build/bin/cuda_stream_test  
2022-05-18T04:22:04.1539468Z   inflating: build/bin/cuda_apply_test  
2022-05-18T04:22:04.1593329Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2022-05-18T04:22:04.1663685Z   inflating: build/bin/cuda_complex_math_test  
2022-05-18T04:22:04.1724016Z   inflating: build/bin/cuda_atomic_ops_test  
2022-05-18T04:22:04.1778233Z   inflating: build/bin/inline_container_test  
2022-05-18T04:22:04.1827381Z   inflating: build/bin/op_allowlist_test  
2022-05-18T04:22:04.1882185Z   inflating: build/bin/cuda_caching_host_allocator_test  
2022-05-18T04:22:04.1936938Z   inflating: build/bin/cuda_vectorized_test  
2022-05-18T04:22:04.1986887Z   inflating: build/bin/cuda_optional_test  
2022-05-18T04:22:04.2048025Z   inflating: build/bin/apply_utils_test  
2022-05-18T04:22:04.2118269Z   inflating: build/bin/cuda_distributions_test  
2022-05-18T04:22:04.2171667Z   inflating: build/bin/cuda_integer_divider_test  
2022-05-18T04:22:04.2230446Z   inflating: build/bin/quantized_test  
2022-05-18T04:22:04.2291069Z   inflating: build/bin/cuda_generator_test  
2022-05-18T04:22:04.2341706Z   inflating: build/bin/variant_test  
2022-05-18T04:22:04.2402266Z   inflating: build/bin/cpu_generator_test  
2022-05-18T04:22:04.2453496Z   inflating: build/bin/dispatch_key_set_test  
2022-05-18T04:22:04.2515697Z   inflating: build/bin/type_test     
2022-05-18T04:22:04.2645784Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-05-18T04:22:04.2744139Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-05-18T04:22:04.2801583Z   inflating: build/bin/test_parallel  
2022-05-18T04:22:04.2855241Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-05-18T04:22:04.2907395Z   inflating: build/bin/reportMemoryUsage_test  
2022-05-18T04:22:04.2957943Z   inflating: build/bin/reduce_ops_test  
2022-05-18T04:22:04.3017524Z   inflating: build/bin/scalar_test   
2022-05-18T04:22:04.3116351Z   inflating: build/bin/kernel_function_test  
2022-05-18T04:22:04.3412978Z   inflating: build/bin/op_registration_test  
2022-05-18T04:22:04.3487289Z   inflating: build/bin/Dict_test     
2022-05-18T04:22:04.3541224Z   inflating: build/bin/Dimname_test  
2022-05-18T04:22:04.3602677Z   inflating: build/bin/basic         
2022-05-18T04:22:04.3655746Z   inflating: build/bin/memory_overlapping_test  
2022-05-18T04:22:04.3706703Z   inflating: build/bin/cuda_half_test  
2022-05-18T04:22:04.3759920Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2022-05-18T04:22:04.3827150Z   inflating: build/bin/pow_test      
2022-05-18T04:22:04.3828692Z   inflating: build/bin/verify_api_visibility  
2022-05-18T04:22:04.3889366Z   inflating: build/bin/cuda_complex_test  
2022-05-18T04:22:04.3948401Z   inflating: build/bin/NamedTensor_test  
2022-05-18T04:22:04.4000472Z   inflating: build/bin/weakref_test  
2022-05-18T04:22:04.4058641Z   inflating: build/bin/extension_backend_test  
2022-05-18T04:22:04.4116015Z   inflating: build/bin/half_test     
2022-05-18T04:22:04.4168335Z   inflating: build/bin/wrapdim_test  
2022-05-18T04:22:04.4224211Z   inflating: build/bin/broadcast_test  
2022-05-18T04:22:04.4274961Z   inflating: build/bin/dlconvertor_test  
2022-05-18T04:22:04.4332916Z   inflating: build/bin/scalar_tensor_test  
2022-05-18T04:22:04.4390204Z   inflating: build/bin/native_test   
2022-05-18T04:22:04.4443800Z   inflating: build/bin/undefined_tensor_test  
2022-05-18T04:22:04.4556480Z   inflating: build/bin/List_test     
2022-05-18T04:22:04.4635707Z   inflating: build/bin/tensor_iterator_test  
2022-05-18T04:22:04.4687359Z   inflating: build/bin/CppSignature_test  
2022-05-18T04:22:04.4690048Z   inflating: build/bin/thread_init_test  
2022-05-18T04:22:04.4744235Z   inflating: build/bin/math_kernel_test  
2022-05-18T04:22:04.4794469Z   inflating: build/bin/lazy_tensor_test  
2022-05-18T04:22:04.4848218Z   inflating: build/bin/memory_format_test  
2022-05-18T04:22:04.4899605Z   inflating: build/bin/operators_test  
2022-05-18T04:22:04.4951043Z   inflating: build/bin/cuda_dlconvertor_test  
2022-05-18T04:22:04.5058431Z   inflating: build/bin/kernel_lambda_test  
2022-05-18T04:22:04.5128177Z   inflating: build/bin/vmap_test     
2022-05-18T04:22:04.5189430Z   inflating: build/bin/IListRef_test  
2022-05-18T04:22:04.5286901Z   inflating: build/bin/ivalue_test   
2022-05-18T04:22:04.5336175Z   inflating: build/bin/mobile_memory_cleanup  
2022-05-18T04:22:04.5397566Z   inflating: build/bin/kernel_stackbased_test  
2022-05-18T04:22:04.5450637Z   inflating: build/bin/stride_properties_test  
2022-05-18T04:22:04.5511019Z   inflating: build/bin/atest         
2022-05-18T04:22:04.5576240Z   inflating: build/bin/KernelFunction_test  
2022-05-18T04:22:04.5634151Z   inflating: build/bin/backend_fallback_test  
2022-05-18T04:22:04.5724903Z   inflating: build/bin/cpu_rng_test  
2022-05-18T04:22:04.5848098Z   inflating: build/bin/kernel_function_legacy_test  
2022-05-18T04:22:04.5915380Z   inflating: build/bin/ProcessGroupGlooTest  
2022-05-18T04:22:04.5976392Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2022-05-18T04:22:04.6039194Z   inflating: build/bin/ProcessGroupNCCLTest  
2022-05-18T04:22:04.6056215Z   inflating: build/bin/tutorial_tensorexpr  
2022-05-18T04:22:04.6116852Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2022-05-18T04:22:04.6171695Z   inflating: build/bin/test_dist_autograd  
2022-05-18T04:22:04.6242569Z   inflating: build/bin/test_mobile_nnc  
2022-05-18T04:22:04.6245230Z   inflating: build/bin/parallel_benchmark  
2022-05-18T04:22:04.6317637Z   inflating: build/bin/test_cpp_rpc  
2022-05-18T04:22:04.6323150Z   inflating: build/bin/torch_shm_manager  
2022-05-18T04:22:04.6334424Z   inflating: build/bin/aot_model_compiler_test  
2022-05-18T04:22:04.6696034Z   inflating: build/bin/test_lazy     
2022-05-18T04:22:04.7567635Z   inflating: build/bin/test_tensorexpr  
2022-05-18T04:22:05.4267631Z   inflating: build/bin/interactive_embedded_interpreter  
2022-05-18T04:22:05.4396083Z   inflating: build/bin/nvfuser_bench  
2022-05-18T04:22:06.1163901Z   inflating: build/bin/test_deploy   
2022-05-18T04:22:06.7913690Z   inflating: build/bin/test_deploy_gpu  
2022-05-18T04:22:07.4620359Z   inflating: build/bin/deploy_benchmark  
2022-05-18T04:22:07.5540729Z   inflating: build/bin/test_jit      
2022-05-18T04:22:08.3486192Z   inflating: build/bin/test_api      
2022-05-18T04:22:08.3487228Z   inflating: .pytorch-test-times.json  
2022-05-18T04:22:08.3516206Z ##[group]Run df -H
2022-05-18T04:22:08.3516464Z df -H
2022-05-18T04:22:08.3530318Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:22:08.3530625Z env:
2022-05-18T04:22:08.3530843Z   IN_CI: 1
2022-05-18T04:22:08.3531046Z   IS_GHA: 1
2022-05-18T04:22:08.3531293Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:22:08.3531560Z   GPU_FLAG: --gpus all
2022-05-18T04:22:08.3531790Z ##[endgroup]
2022-05-18T04:22:08.3571347Z Filesystem      Size  Used Avail Use% Mounted on
2022-05-18T04:22:08.3571685Z devtmpfs        129G     0  129G   0% /dev
2022-05-18T04:22:08.3573848Z tmpfs           129G     0  129G   0% /dev/shm
2022-05-18T04:22:08.3574481Z tmpfs           129G  529k  129G   1% /run
2022-05-18T04:22:08.3574798Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2022-05-18T04:22:08.3578283Z /dev/xvda1      162G   27G  135G  17% /
2022-05-18T04:22:08.3908404Z ##[group]Run .github/scripts/parse_ref.py
2022-05-18T04:22:08.3908765Z .github/scripts/parse_ref.py
2022-05-18T04:22:08.3921081Z shell: /usr/bin/bash -e {0}
2022-05-18T04:22:08.3921315Z env:
2022-05-18T04:22:08.3921531Z   IN_CI: 1
2022-05-18T04:22:08.3921751Z   IS_GHA: 1
2022-05-18T04:22:08.3921980Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:22:08.3922250Z   GPU_FLAG: --gpus all
2022-05-18T04:22:08.3922496Z ##[endgroup]
2022-05-18T04:22:13.8090640Z ##[group]Run set -x
2022-05-18T04:22:13.8091011Z set -x
2022-05-18T04:22:13.8091239Z 
2022-05-18T04:22:13.8091509Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-05-18T04:22:13.8091855Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-05-18T04:22:13.8092204Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-05-18T04:22:13.8092528Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-05-18T04:22:13.8092783Z else
2022-05-18T04:22:13.8093061Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-05-18T04:22:13.8093333Z fi
2022-05-18T04:22:13.8093552Z 
2022-05-18T04:22:13.8093852Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-05-18T04:22:13.8094202Z export COMMIT_MESSAGES
2022-05-18T04:22:13.8094455Z 
2022-05-18T04:22:13.8094748Z # detached container should get cleaned up by teardown_ec2_linux
2022-05-18T04:22:13.8095181Z # TODO: Stop building test binaries as part of the build phase
2022-05-18T04:22:13.8095551Z # Used for GPU_FLAG since that doesn't play nice
2022-05-18T04:22:13.8095862Z # shellcheck disable=SC2086,SC2090
2022-05-18T04:22:13.8096299Z container_name=$(docker run \
2022-05-18T04:22:13.8096570Z   ${GPU_FLAG:-} \
2022-05-18T04:22:13.8096822Z   -e BUILD_ENVIRONMENT \
2022-05-18T04:22:13.8097091Z   -e PR_NUMBER \
2022-05-18T04:22:13.8097381Z   -e CUSTOM_TEST_ARTIFACT_BUILD_DIR \
2022-05-18T04:22:13.8097653Z   -e GITHUB_ACTIONS \
2022-05-18T04:22:13.8097902Z   -e IN_CI \
2022-05-18T04:22:13.8098138Z   -e IS_GHA \
2022-05-18T04:22:13.8098362Z   -e BRANCH \
2022-05-18T04:22:13.8098598Z   -e SHA1 \
2022-05-18T04:22:13.8098853Z   -e AWS_DEFAULT_REGION \
2022-05-18T04:22:13.8099120Z   -e IN_WHEEL_TEST \
2022-05-18T04:22:13.8099367Z   -e SHARD_NUMBER \
2022-05-18T04:22:13.8099626Z   -e JOB_BASE_NAME \
2022-05-18T04:22:13.8099881Z   -e TEST_CONFIG \
2022-05-18T04:22:13.8100126Z   -e NUM_TEST_SHARDS \
2022-05-18T04:22:13.8100390Z   -e PR_BODY \
2022-05-18T04:22:13.8100646Z   -e COMMIT_MESSAGES \
2022-05-18T04:22:13.8100917Z   -e PYTORCH_RETRY_TEST_CASES \
2022-05-18T04:22:13.8101191Z   -e PR_LABELS \
2022-05-18T04:22:13.8101474Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-05-18T04:22:13.8101744Z   -e SCCACHE_BUCKET \
2022-05-18T04:22:13.8101998Z   -e XLA_CUDA \
2022-05-18T04:22:13.8102279Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-05-18T04:22:13.8102600Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-05-18T04:22:13.8102917Z   --ulimit stack=10485760:83886080 \
2022-05-18T04:22:13.8103222Z   --security-opt seccomp=unconfined \
2022-05-18T04:22:13.8103527Z   --cap-add=SYS_PTRACE \
2022-05-18T04:22:13.8103777Z   --ipc=host \
2022-05-18T04:22:13.8104039Z   --shm-size="${SHM_SIZE}" \
2022-05-18T04:22:13.8104293Z   --tty \
2022-05-18T04:22:13.8104518Z   --detach \
2022-05-18T04:22:13.8104780Z   --name="${container_name}" \
2022-05-18T04:22:13.8105047Z   --user jenkins \
2022-05-18T04:22:13.8105347Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-05-18T04:22:13.8105679Z   -w /var/lib/jenkins/workspace \
2022-05-18T04:22:13.8105956Z   "${DOCKER_IMAGE}"
2022-05-18T04:22:13.8106178Z )
2022-05-18T04:22:13.8106513Z docker exec -t "${container_name}" sh -c "pip install dist/*.whl && ${TEST_COMMAND}"
2022-05-18T04:22:13.8120189Z shell: /usr/bin/bash -e {0}
2022-05-18T04:22:13.8120437Z env:
2022-05-18T04:22:13.8120651Z   IN_CI: 1
2022-05-18T04:22:13.8120852Z   IS_GHA: 1
2022-05-18T04:22:13.8121095Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:22:13.8121433Z   GPU_FLAG: --gpus all
2022-05-18T04:22:13.8121748Z   BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7
2022-05-18T04:22:13.8122054Z   PR_NUMBER: 
2022-05-18T04:22:13.8122283Z   BRANCH: master
2022-05-18T04:22:13.8122571Z   CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts
2022-05-18T04:22:13.8122910Z   SHA1: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T04:22:13.8123205Z   PYTORCH_RETRY_TEST_CASES: 1
2022-05-18T04:22:13.8123517Z   JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test
2022-05-18T04:22:13.8123833Z   TEST_CONFIG: distributed
2022-05-18T04:22:13.8124081Z   SHARD_NUMBER: 1
2022-05-18T04:22:13.8124315Z   NUM_TEST_SHARDS: 2
2022-05-18T04:22:13.8124533Z   PR_BODY: 
2022-05-18T04:22:13.8124829Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-05-18T04:22:13.8125127Z   SHM_SIZE: 2g
2022-05-18T04:22:13.8125598Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:22:13.8126079Z   XLA_CUDA: 
2022-05-18T04:22:13.8126425Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-05-18T04:22:13.8126754Z ##[endgroup]
2022-05-18T04:22:13.8157843Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-05-18T04:22:13.8158457Z + [[ linux-xenial-cuda11.3-py3.7-gcc7 == *onnx* ]]
2022-05-18T04:22:13.8158786Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-05-18T04:22:13.8161865Z ++ git cherry -v origin/master
2022-05-18T04:22:13.8195984Z + COMMIT_MESSAGES=
2022-05-18T04:22:13.8196260Z + export COMMIT_MESSAGES
2022-05-18T04:22:13.8205126Z +++ nproc --ignore=2
2022-05-18T04:22:13.8241423Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e CUSTOM_TEST_ARTIFACT_BUILD_DIR -e GITHUB_ACTIONS -e IN_CI -e IS_GHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e JOB_BASE_NAME -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_2342799944 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:22:32.1768739Z + container_name=e9eae2ba3bb0fe475a85ca8ae7d95bec67fc2d17586595b7ebbece063df7a74f
2022-05-18T04:22:32.1770189Z + docker exec -t e9eae2ba3bb0fe475a85ca8ae7d95bec67fc2d17586595b7ebbece063df7a74f sh -c 'pip install dist/*.whl && .jenkins/pytorch/test.sh'
2022-05-18T04:22:32.6767050Z Processing ./dist/torch-1.12.0a0+git3b23752-cp37-cp37m-linux_x86_64.whl
2022-05-18T04:22:32.7774458Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.12.0a0+git3b23752) (4.1.1)
2022-05-18T04:22:33.3293397Z Installing collected packages: torch
2022-05-18T04:22:44.9068993Z Successfully installed torch-1.12.0a0+git3b23752
2022-05-18T04:22:44.9572685Z + COMPACT_JOB_NAME=linux-xenial-cuda11.3-py3.7-gcc7
2022-05-18T04:22:44.9575652Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-05-18T04:22:44.9786961Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.7/site-packages/torch
2022-05-18T04:22:44.9787447Z + TORCH_BIN_DIR=/opt/conda/lib/python3.7/site-packages/torch/bin
2022-05-18T04:22:44.9787906Z + TORCH_LIB_DIR=/opt/conda/lib/python3.7/site-packages/torch/lib
2022-05-18T04:22:44.9788350Z + TORCH_TEST_DIR=/opt/conda/lib/python3.7/site-packages/torch/test
2022-05-18T04:22:44.9788670Z + BUILD_DIR=build
2022-05-18T04:22:44.9789196Z + BUILD_RENAMED_DIR=build_renamed
2022-05-18T04:22:44.9789461Z + BUILD_BIN_DIR=build/bin
2022-05-18T04:22:44.9793557Z + [[ -n distributed ]]
2022-05-18T04:22:44.9794057Z + BUILD_ENVIRONMENT=linux-xenial-cuda11.3-py3.7-gcc7-distributed
2022-05-18T04:22:44.9794778Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *bazel* ]]
2022-05-18T04:22:44.9795175Z ++ realpath build/custom_test_artifacts
2022-05-18T04:22:44.9800009Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-05-18T04:22:44.9804291Z ++ dirname .jenkins/pytorch/test.sh
2022-05-18T04:22:44.9810941Z + source .jenkins/pytorch/common.sh
2022-05-18T04:22:44.9815096Z +++ dirname .jenkins/pytorch/common.sh
2022-05-18T04:22:44.9824770Z ++ source .jenkins/pytorch/common_utils.sh
2022-05-18T04:22:44.9828668Z +++ TORCHVISION_COMMIT=8a2dc6f22ac4389ccba8859aa1e1cb14f1ee53db
2022-05-18T04:22:44.9829105Z ++ set -ex
2022-05-18T04:22:44.9836218Z ++++ dirname .jenkins/pytorch/common.sh
2022-05-18T04:22:44.9846657Z +++ cd .jenkins/pytorch
2022-05-18T04:22:44.9847327Z +++ pwd -P
2022-05-18T04:22:44.9850117Z ++ SCRIPT_DIR=/var/lib/jenkins/workspace/.jenkins/pytorch
2022-05-18T04:22:44.9850681Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *linux* ]]
2022-05-18T04:22:44.9854321Z +++ find /etc/apt/ -type f -name '*.list'
2022-05-18T04:22:44.9872361Z ++ sudo sed -i 's/.*nvidia.*/# &/' /etc/apt/sources.list /etc/apt/sources.list.d/cuda.list /etc/apt/sources.list.d/nodesource.list /etc/apt/sources.list.d/ubuntu-toolchain-r-ubuntu-test-xenial.list /etc/apt/sources.list.d/yarn.list
2022-05-18T04:22:44.9933799Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *rocm* ]]
2022-05-18T04:22:44.9934149Z ++ echo ENTERED_USER_LAND
2022-05-18T04:22:44.9934408Z ENTERED_USER_LAND
2022-05-18T04:22:44.9934652Z ++ export IN_CI=1
2022-05-18T04:22:44.9934864Z ++ IN_CI=1
2022-05-18T04:22:44.9935556Z ++ declare -f -t trap_add
2022-05-18T04:22:44.9935823Z ++ trap_add cleanup EXIT
2022-05-18T04:22:44.9936073Z ++ trap_add_cmd=cleanup
2022-05-18T04:22:44.9936307Z ++ shift
2022-05-18T04:22:44.9936597Z ++ for trap_add_name in '"$@"'
2022-05-18T04:22:44.9944063Z ++++ trap -p EXIT
2022-05-18T04:22:44.9947367Z +++ eval 'extract_trap_cmd '
2022-05-18T04:22:44.9947678Z ++++ extract_trap_cmd
2022-05-18T04:22:44.9948064Z ++++ printf '%s\n' ''
2022-05-18T04:22:44.9948337Z +++ printf '%s\n' cleanup
2022-05-18T04:22:44.9950542Z ++ trap -- '
2022-05-18T04:22:44.9950818Z cleanup' EXIT
2022-05-18T04:22:44.9953154Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *win-* ]]
2022-05-18T04:22:44.9953505Z ++ which sccache
2022-05-18T04:22:44.9963673Z ++ sccache --stop-server
2022-05-18T04:22:44.9989890Z ++ true
2022-05-18T04:22:44.9990503Z ++ rm -f /var/lib/jenkins/sccache_error.log
2022-05-18T04:22:44.9998564Z ++ [[ -n '' ]]
2022-05-18T04:22:44.9999027Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *rocm* ]]
2022-05-18T04:22:44.9999425Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
2022-05-18T04:22:44.9999717Z ++ SCCACHE_IDLE_TIMEOUT=1200
2022-05-18T04:22:45.0000021Z ++ RUST_LOG=sccache::server=error
2022-05-18T04:22:45.0000339Z ++ sccache --start-server
2022-05-18T04:22:45.0018660Z sccache: Starting the server...
2022-05-18T04:22:45.0273744Z ++ sccache --zero-stats
2022-05-18T04:22:45.0294814Z Compile requests                      0
2022-05-18T04:22:45.0295141Z Compile requests executed             0
2022-05-18T04:22:45.0295430Z Cache hits                            0
2022-05-18T04:22:45.0295686Z Cache misses                          0
2022-05-18T04:22:45.0295964Z Cache timeouts                        0
2022-05-18T04:22:45.0296241Z Cache read errors                     0
2022-05-18T04:22:45.0296497Z Forced recaches                       0
2022-05-18T04:22:45.0296772Z Cache write errors                    0
2022-05-18T04:22:45.0297054Z Compilation failures                  0
2022-05-18T04:22:45.0297320Z Cache errors                          0
2022-05-18T04:22:45.0297689Z Non-cacheable compilations            0
2022-05-18T04:22:45.0298425Z Non-cacheable calls                   0
2022-05-18T04:22:45.0298769Z Non-compilation calls                 0
2022-05-18T04:22:45.0299052Z Unsupported compiler calls            0
2022-05-18T04:22:45.0299358Z Average cache write               0.000 s
2022-05-18T04:22:45.0299822Z Average cache read miss           0.000 s
2022-05-18T04:22:45.0300119Z Average cache read hit            0.000 s
2022-05-18T04:22:45.0300421Z Failed distributed compilations       0
2022-05-18T04:22:45.0301155Z Cache location                  S3, bucket: Bucket(name=ossci-compiler-cache-circleci-v2, base_url=http://ossci-compiler-cache-circleci-v2.s3.amazonaws.com/)
2022-05-18T04:22:45.0301815Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-test == *-build ]]
2022-05-18T04:22:45.0302132Z ++ which ccache
2022-05-18T04:22:45.0309572Z ++ '[' -z linux-xenial-cuda11.3-py3.7-gcc7 ']'
2022-05-18T04:22:45.0310114Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *linux-trusty-py3.6-gcc7* ]]
2022-05-18T04:22:45.0310496Z ++ BUILD_TEST_LIBTORCH=0
2022-05-18T04:22:45.0310765Z ++ [[ distributed == *xla* ]]
2022-05-18T04:22:45.0311193Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *centos* ]]
2022-05-18T04:22:45.0311696Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *linux-bionic* ]]
2022-05-18T04:22:45.0312236Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *linux-focal* ]]
2022-05-18T04:22:45.0312632Z + echo 'Testing pytorch'
2022-05-18T04:22:45.0312872Z Testing pytorch
2022-05-18T04:22:45.0315987Z + export LANG=C.UTF-8
2022-05-18T04:22:45.0316485Z + LANG=C.UTF-8
2022-05-18T04:22:45.0316986Z + PR_NUMBER=
2022-05-18T04:22:45.0317579Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-05-18T04:22:45.0318146Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-05-18T04:22:45.0318679Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *rocm* ]]
2022-05-18T04:22:45.0319176Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *-slow-* ]]
2022-05-18T04:22:45.0319529Z + [[ distributed == \s\l\o\w ]]
2022-05-18T04:22:45.0319981Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *slow-gradcheck* ]]
2022-05-18T04:22:45.0320477Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *cuda* ]]
2022-05-18T04:22:45.0320852Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-05-18T04:22:45.0321175Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-05-18T04:22:45.0321961Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *cuda11* ]]
2022-05-18T04:22:45.0322520Z + export BUILD_SPLIT_CUDA=ON
2022-05-18T04:22:45.0322939Z + BUILD_SPLIT_CUDA=ON
2022-05-18T04:22:45.0323825Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *crossref* ]]
2022-05-18T04:22:45.0324216Z + [[ -n '' ]]
2022-05-18T04:22:45.0324507Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=0
2022-05-18T04:22:45.0324830Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=0
2022-05-18T04:22:45.0325259Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *rocm* ]]
2022-05-18T04:22:45.0325761Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *ppc64le* ]]
2022-05-18T04:22:45.0326276Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *-bazel-* ]]
2022-05-18T04:22:45.0326651Z + pip_install --user ninja
2022-05-18T04:22:45.0327010Z + pip install --progress-bar off --user ninja
2022-05-18T04:22:45.5473987Z Collecting ninja
2022-05-18T04:22:45.5702817Z   Downloading ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2022-05-18T04:22:45.5789848Z [?25l
2022-05-18T04:22:46.0477020Z [?25hInstalling collected packages: ninja
2022-05-18T04:22:46.0589468Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-05-18T04:22:46.0590144Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-05-18T04:22:46.0664526Z Successfully installed ninja-1.10.2.3
2022-05-18T04:22:46.1187360Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-05-18T04:22:46.1188025Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-05-18T04:22:46.1189333Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *asan* ]]
2022-05-18T04:22:46.1190216Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *-NO_AVX-* ]]
2022-05-18T04:22:46.1190620Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X ]]
2022-05-18T04:22:46.1191091Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *-NO_AVX2-* ]]
2022-05-18T04:22:46.1191483Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-05-18T04:22:46.1191929Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *-NO_AVX512-* ]]
2022-05-18T04:22:46.1192324Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\5\1\2 ]]
2022-05-18T04:22:46.1194836Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *tbb* ]]
2022-05-18T04:22:46.1209450Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *libtorch* ]]
2022-05-18T04:22:46.1210326Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *-bazel-* ]]
2022-05-18T04:22:46.1212881Z + cd test
2022-05-18T04:22:46.1213419Z + python -c 'import torch; print(torch.__config__.show())'
2022-05-18T04:22:50.6878186Z PyTorch built with:
2022-05-18T04:22:50.6878641Z   - GCC 7.5
2022-05-18T04:22:50.6878956Z   - C++ Version: 201402
2022-05-18T04:22:50.6879501Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-05-18T04:22:50.6880059Z   - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-05-18T04:22:50.6880748Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-05-18T04:22:50.6881125Z   - LAPACK is enabled (usually provided by MKL)
2022-05-18T04:22:50.6881461Z   - NNPACK is enabled
2022-05-18T04:22:50.6881758Z   - CPU capability usage: AVX2
2022-05-18T04:22:50.6882065Z   - CUDA Runtime 11.3
2022-05-18T04:22:50.6882459Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2022-05-18T04:22:50.6882849Z   - CuDNN 8.3.2  (built against CUDA 11.5)
2022-05-18T04:22:50.6883150Z   - Magma 2.5.2
2022-05-18T04:22:50.6886110Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-05-18T04:22:50.6888318Z 
2022-05-18T04:22:51.2874743Z + cd test
2022-05-18T04:22:51.2875291Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-05-18T04:22:52.0630856Z ATen/Parallel:
2022-05-18T04:22:52.0631179Z 	at::get_num_threads() : 16
2022-05-18T04:22:52.0631502Z 	at::get_num_interop_threads() : 16
2022-05-18T04:22:52.0631802Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-05-18T04:22:52.0632085Z 	omp_get_max_threads() : 16
2022-05-18T04:22:52.0632696Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-05-18T04:22:52.0633088Z 	mkl_get_max_threads() : 16
2022-05-18T04:22:52.0633535Z Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-05-18T04:22:52.0633891Z std::thread::hardware_concurrency() : 32
2022-05-18T04:22:52.0634185Z Environment variables:
2022-05-18T04:22:52.0634456Z 	OMP_NUM_THREADS : [not set]
2022-05-18T04:22:52.0634707Z 	MKL_NUM_THREADS : [not set]
2022-05-18T04:22:52.0635276Z ATen parallel backend: OpenMP
2022-05-18T04:22:52.0635481Z 
2022-05-18T04:22:52.1696961Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *deploy* ]]
2022-05-18T04:22:52.1697709Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *backward* ]]
2022-05-18T04:22:52.1698074Z + [[ distributed == *xla* ]]
2022-05-18T04:22:52.1698525Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *jit_legacy-test ]]
2022-05-18T04:22:52.1699047Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-test == *jit_legacy-test ]]
2022-05-18T04:22:52.1699394Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-05-18T04:22:52.1699839Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *libtorch* ]]
2022-05-18T04:22:52.1700361Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *distributed* ]]
2022-05-18T04:22:52.1700692Z + test_distributed
2022-05-18T04:22:52.1701029Z + echo 'Testing distributed python tests'
2022-05-18T04:22:52.1701344Z Testing distributed python tests
2022-05-18T04:22:52.1701781Z + python test/run_test.py --distributed-tests --shard 1 2 --verbose
2022-05-18T04:22:58.4493244Z Ignoring disabled issues:  []
2022-05-18T04:22:58.4624856Z test/run_test.py:894: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-05-18T04:22:58.4625676Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) == "11.6":
2022-05-18T04:22:58.4684335Z Found stats for current commit: 3b2375291aab7b48442f2e6fb1ef66cebc761e24 and job: linux-xenial-cuda11.3-py3.7-gcc7. Proceeding with those values.
2022-05-18T04:22:58.4686998Z Selected tests:
2022-05-18T04:22:58.4687272Z  distributed/test_distributed_spawn
2022-05-18T04:22:58.4687606Z  distributed/optim/test_zero_redundancy_optimizer
2022-05-18T04:22:58.4687950Z  distributed/fsdp/test_fsdp_optim_state
2022-05-18T04:22:58.4688354Z  distributed/test_store
2022-05-18T04:22:58.4688823Z  distributed/test_pg_wrapper
2022-05-18T04:22:58.4689462Z  distributed/fsdp/test_fsdp_clip_grad_norm
2022-05-18T04:22:58.4690632Z  distributed/fsdp/test_fsdp_grad_acc
2022-05-18T04:22:58.4690945Z  distributed/fsdp/test_fsdp_freezing_weights
2022-05-18T04:22:58.4691384Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-05-18T04:22:58.4691703Z  distributed/fsdp/test_fsdp_exec_order
2022-05-18T04:22:58.4692011Z  distributed/fsdp/test_fsdp_overlap
2022-05-18T04:22:58.4692319Z  distributed/elastic/multiprocessing/api_test
2022-05-18T04:22:58.4692672Z  distributed/_shard/sharded_tensor/ops/test_matrix_ops
2022-05-18T04:22:58.4692997Z  distributed/fsdp/test_fsdp_memory
2022-05-18T04:22:58.4693293Z  distributed/fsdp/test_fsdp_ignored_modules
2022-05-18T04:22:58.4693619Z  distributed/elastic/timer/local_timer_example
2022-05-18T04:22:58.4693927Z  distributed/fsdp/test_fsdp_input
2022-05-18T04:22:58.4694237Z  distributed/_shard/sharded_tensor/ops/test_tensor_ops
2022-05-18T04:22:58.4694593Z  distributed/_shard/sharding_spec/test_sharding_spec
2022-05-18T04:22:58.4694945Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-05-18T04:22:58.4695269Z  distributed/_shard/sharded_tensor/ops/test_init
2022-05-18T04:22:58.4695603Z  distributed/elastic/utils/distributed_test
2022-05-18T04:22:58.4695932Z  distributed/fsdp/test_fsdp_multiple_forward
2022-05-18T04:22:58.4696230Z  distributed/fsdp/test_fsdp_uneven
2022-05-18T04:22:58.4696528Z  distributed/fsdp/test_fsdp_traversal
2022-05-18T04:22:58.4696859Z  distributed/_shard/sharded_tensor/ops/test_embedding
2022-05-18T04:22:58.4697205Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-05-18T04:22:58.4697541Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-05-18T04:22:58.4697888Z  distributed/fsdp/test_flatten_params_wrapper
2022-05-18T04:22:58.4698204Z  distributed/elastic/utils/logging_test
2022-05-18T04:22:58.4698492Z  distributed/nn/jit/test_instantiator
2022-05-18T04:22:58.4698769Z  distributed/test_nccl
2022-05-18T04:22:58.4699079Z  distributed/_shard/sharding_plan/test_sharding_plan
2022-05-18T04:22:58.4699377Z  distributed/_shard/test_sharder
2022-05-18T04:22:58.4699802Z  distributed/elastic/timer/api_test
2022-05-18T04:22:58.4700130Z  distributed/pipeline/sync/skip/test_api
2022-05-18T04:22:58.4700459Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-05-18T04:22:58.4700807Z  distributed/pipeline/sync/skip/test_portal
2022-05-18T04:22:58.4701140Z  distributed/pipeline/sync/skip/test_tracker
2022-05-18T04:22:58.4701438Z  distributed/pipeline/sync/test_balance
2022-05-18T04:22:58.4701756Z  distributed/pipeline/sync/test_checkpoint
2022-05-18T04:22:58.4702097Z  distributed/pipeline/sync/test_deferred_batch_norm
2022-05-18T04:22:58.4702423Z  distributed/pipeline/sync/test_inplace
2022-05-18T04:22:58.4702711Z  distributed/pipeline/sync/test_phony
2022-05-18T04:22:58.4703026Z  distributed/pipeline/sync/test_pipeline
2022-05-18T04:22:58.4703356Z  distributed/pipeline/sync/test_transparency
2022-05-18T04:22:58.4703655Z  distributed/rpc/test_faulty_agent
2022-05-18T04:22:58.4801027Z Prioritized test from test file changes.
2022-05-18T04:22:58.4801341Z reordering tests for PR:
2022-05-18T04:22:58.4801672Z prioritized: []
2022-05-18T04:22:58.4806325Z the rest: ['distributed/test_distributed_spawn', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/fsdp/test_fsdp_optim_state', 'distributed/test_store', 'distributed/test_pg_wrapper', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/fsdp/test_fsdp_freezing_weights', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/fsdp/test_fsdp_exec_order', 'distributed/fsdp/test_fsdp_overlap', 'distributed/elastic/multiprocessing/api_test', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops', 'distributed/fsdp/test_fsdp_memory', 'distributed/fsdp/test_fsdp_ignored_modules', 'distributed/elastic/timer/local_timer_example', 'distributed/fsdp/test_fsdp_input', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/_shard/sharded_tensor/ops/test_linear', 'distributed/_shard/sharded_tensor/ops/test_init', 'distributed/elastic/utils/distributed_test', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/fsdp/test_fsdp_uneven', 'distributed/fsdp/test_fsdp_traversal', 'distributed/_shard/sharded_tensor/ops/test_embedding', 'distributed/_shard/sharded_tensor/ops/test_chunk', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag', 'distributed/fsdp/test_flatten_params_wrapper', 'distributed/elastic/utils/logging_test', 'distributed/nn/jit/test_instantiator', 'distributed/test_nccl', 'distributed/_shard/sharding_plan/test_sharding_plan', 'distributed/_shard/test_sharder', 'distributed/elastic/timer/api_test', 'distributed/pipeline/sync/skip/test_api', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_transparency', 'distributed/rpc/test_faulty_agent']
2022-05-18T04:22:58.4809342Z 
2022-05-18T04:22:58.5448937Z Running distributed/test_distributed_spawn ... [2022-05-18 04:22:58.544523]
2022-05-18T04:22:58.5489509Z /usr/bin/mpiexec
2022-05-18T04:22:58.5497514Z Running distributed tests for the test backend with env init_method
2022-05-18T04:22:58.5502213Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:22:58.549918]
2022-05-18T04:22:59.7245712Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:22:59.8422778Z Running distributed tests for the test backend with file init_method
2022-05-18T04:22:59.8425707Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:22:59.842213]
2022-05-18T04:23:00.9611617Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:23:01.0780694Z Running distributed tests for the mpi backend with env init_method
2022-05-18T04:23:01.1880207Z Executing ['mpiexec', '-n', '3', '--noprefix', '--allow-run-as-root', '/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:23:01.187522]
2022-05-18T04:23:01.1978829Z Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/KOGNJP63R56OF4HTDZNKGJ4BNZ:/var/lib/docker/overlay2/l/YTPXIK37R6YDLANTAZYTJGR26R:/var/lib/docker/overlay2/l/4ZYVIIIYE463MLMVCI7TYOK3TF:/var/lib/docker/overlay2/l/7EFX6SG3J2AN2VQI3X3IFL7MKU:/var/lib/docker/overlay2/l/2L3H2CU3W7P7JGKJB3P3XMKBX4:/var/lib/docker/overlay2/l/5LCQ5UAMPQBK4EVFUIVWSYOEIA:/var/lib/docker/overlay2/l/QZTE4QRBGZL2KTVU37RUDMZ52I:/var/lib/docker/overlay2/l/N5A7EJPAFUILMWPL3V4LA2YUWH:/var/lib/docker/overlay2/l/M76T4D3VKAADA'
2022-05-18T04:23:01.1980758Z Unexpected end of /proc/mounts line `J5BPHH35ZZTMD:/var/lib/docker/overlay2/l/5ZHW63AUZEX2W6ZYK6UOYRPFGL:/var/lib/docker/overlay2/l/LLDKRW4LIKEF5BYN3JXKKXMEV7:/var/lib/docker/overlay2/l/Z4RH2C6LAWFX27JOMMNW5LDYCX:/var/lib/docker/overlay2/l/IKCZLFDYSFWYV47TZLGS5VCYPR:/var/lib/docker/overlay2/l/G235NFX6PH7PPGVFIM4BK64Y34:/var/lib/docker/overlay2/l/FNPBNMG4LL6MIUBDQ4B4QG4WN6:/var/lib/docker/overlay2/l/O77XTVQ6OQ3XTFLDGLFS25O3JW:/var/lib/docker/overlay2/l/56QFLVHKE6FWFYGMWI3SDUR324:/var/lib/docker/overlay2/l/TIIIY4QAG6KXR5NALJXOMKIEOF:/var/lib/do'
2022-05-18T04:23:01.1982778Z Unexpected end of /proc/mounts line `cker/overlay2/l/FIBTQ3YPAXTKOZUSR52W6EESDY:/var/lib/docker/overlay2/l/WIWIYM6T56THTWHFEA6754PHIG:/var/lib/docker/overlay2/l/NGGBA42OVTNKSHPVWHKGJXEUOY:/var/lib/docker/overlay2/l/DVGYNR5KYSTCGJLA3D6BDVIQHR:/var/lib/docker/overlay2/l/XFZA7CXFIZIMAF6ZI5Q7XM3JWK:/var/lib/docker/overlay2/l/ST3PKIHV6QHPNR2YRNVWT2V7XF:/var/lib/docker/overlay2/l/MOIGKSB36VVFDU2MR7QVTWCNAF:/var/lib/docker/overlay2/l/UDXVYMAMJQXTHJKWL74UGWUYQT:/var/lib/docker/overlay2/l/LC3CJTWQKUG254KV6SC4HYQC5G:/var/lib/docker/overlay2/l/AOH3MFWBF'
2022-05-18T04:23:01.1984565Z Unexpected end of /proc/mounts line `FSWCXAYYYFHTLZXJX:/var/lib/docker/overlay2/l/JJOUHZDV3WUD67CLUVOMJINLHD:/var/lib/docker/overlay2/l/62QBI7MI2ZNMDOX2LT3CB52LSH:/var/lib/docker/overlay2/l/6SSKWNVVD443YBEHTUI2VFFM2C:/var/lib/docker/overlay2/l/REZAAWKVT33XTKFOEOFIXPTYCQ:/var/lib/docker/overlay2/l/7QB7XP7VWEVSTNO5X6EOI2CDBX:/var/lib/docker/overlay2/l/XWMUJS7NF7JJNPXBKEZZZPTLAW:/var/lib/docker/overlay2/l/2E7XDS5VNZYTRZGRHMYGGGWJVV:/var/lib/docker/overlay2/l/3LBZKRCZUHVECLQDDXUH2P2AVE:/var/lib/docker/overlay2/l/M7PHRM4SJCHDDZENUKSBHBDZNR:/var/li'
2022-05-18T04:23:01.1986334Z Unexpected end of /proc/mounts line `b/docker/overlay2/l/R6GEGH3ZCU5QRS2X2FPSPGIOE6:/var/lib/docker/overlay2/l/CLJUBI6JQ3MN6R5NY47TCOAPFI:/var/lib/docker/overlay2/l/ABHVTATVVZMR7IMXXKQVATO4FB:/var/lib/docker/overlay2/l/WDIH2DYWBO7ICNTQ6FDBAZKJAX:/var/lib/docker/overlay2/l/CJE2MFOI46E3WT4Q3A2EBG3BIH:/var/lib/docker/overlay2/l/VIIF54KHXQLOEESIJWZWDI57U7:/var/lib/docker/overlay2/l/6UGTSVNSKNLXUR5DSWB7EQBY6H:/var/lib/docker/overlay2/l/ZTZYPP45AJNYD3E3D3W57JTER3:/var/lib/docker/overlay2/l/L2GNII2SGZQQTJGSGVI6G7QRAD:/var/lib/docker/overlay2/l/YASFI'
2022-05-18T04:23:01.1988093Z Unexpected end of /proc/mounts line `LPPWOITUTZYNYQFPQNTLU:/var/lib/docker/overlay2/l/JKF4DCJZS6LMOPTCWJ7JZ5X2NJ:/var/lib/docker/overlay2/l/ZD45LVBF3HHVQEQXFXENDRXKLH:/var/lib/docker/overlay2/l/AAOLME7VHHKQ7EELDNJRHUW5KN:/var/lib/docker/overlay2/l/7FWSB6SFPAQ2ZTB6O2EQWHPYPT:/var/lib/docker/overlay2/l/6VAFGR22OTLBGH2ETREFLZWH65:/var/lib/docker/overlay2/l/GTN6VK43UHKD5HYRH5MJSPMPGB,upperdir=/var/lib/docker/overlay2/de50816a31460444d6bba969e05d70c5c935e808e00a783dd8aff59d96bfa4d3/diff,workdir=/var/lib/docker/overlay2/de50816a31460444d6bba969e05d'
2022-05-18T04:23:02.3833287Z Test results will be stored in test-reports/dist-mpi/distributed.test_distributed_spawn
2022-05-18T04:23:02.3878655Z 
2022-05-18T04:23:02.3878954Z Running tests...
2022-05-18T04:23:02.3880164Z ----------------------------------------------------------------------
2022-05-18T04:23:02.3880761Z 
2022-05-18T04:23:02.3881308Z ----------------------------------------------------------------------
2022-05-18T04:23:02.3881683Z Ran 0 tests in 0.000s
2022-05-18T04:23:02.3881848Z 
2022-05-18T04:23:02.3881941Z OK
2022-05-18T04:23:02.3882074Z 
2022-05-18T04:23:02.3882201Z Generating XML reports...
2022-05-18T04:23:02.4322243Z Test results will be stored in test-reports/dist-mpi/distributed.test_distributed_spawn
2022-05-18T04:23:02.4369861Z 
2022-05-18T04:23:02.4370748Z Running tests...
2022-05-18T04:23:02.4371662Z ----------------------------------------------------------------------
2022-05-18T04:23:02.4372212Z 
2022-05-18T04:23:02.4372739Z ----------------------------------------------------------------------
2022-05-18T04:23:02.4373103Z Ran 0 tests in 0.000s
2022-05-18T04:23:02.4373265Z 
2022-05-18T04:23:02.4373366Z OK
2022-05-18T04:23:02.4373497Z 
2022-05-18T04:23:02.4373608Z Generating XML reports...
2022-05-18T04:23:02.4469582Z Test results will be stored in test-reports/dist-mpi/distributed.test_distributed_spawn
2022-05-18T04:23:02.4519040Z 
2022-05-18T04:23:02.4519595Z Running tests...
2022-05-18T04:23:02.4520581Z ----------------------------------------------------------------------
2022-05-18T04:23:02.4521301Z 
2022-05-18T04:23:02.4521555Z ----------------------------------------------------------------------
2022-05-18T04:23:02.4521870Z Ran 0 tests in 0.000s
2022-05-18T04:23:02.4522031Z 
2022-05-18T04:23:02.4522123Z OK
2022-05-18T04:23:02.4522253Z 
2022-05-18T04:23:02.4522360Z Generating XML reports...
2022-05-18T04:23:02.5906764Z Running distributed tests for the mpi backend with file init_method
2022-05-18T04:23:02.6982278Z Executing ['mpiexec', '-n', '3', '--noprefix', '--allow-run-as-root', '/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:23:02.697677]
2022-05-18T04:23:02.7080867Z Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/KOGNJP63R56OF4HTDZNKGJ4BNZ:/var/lib/docker/overlay2/l/YTPXIK37R6YDLANTAZYTJGR26R:/var/lib/docker/overlay2/l/4ZYVIIIYE463MLMVCI7TYOK3TF:/var/lib/docker/overlay2/l/7EFX6SG3J2AN2VQI3X3IFL7MKU:/var/lib/docker/overlay2/l/2L3H2CU3W7P7JGKJB3P3XMKBX4:/var/lib/docker/overlay2/l/5LCQ5UAMPQBK4EVFUIVWSYOEIA:/var/lib/docker/overlay2/l/QZTE4QRBGZL2KTVU37RUDMZ52I:/var/lib/docker/overlay2/l/N5A7EJPAFUILMWPL3V4LA2YUWH:/var/lib/docker/overlay2/l/M76T4D3VKAADA'
2022-05-18T04:23:02.7082698Z Unexpected end of /proc/mounts line `J5BPHH35ZZTMD:/var/lib/docker/overlay2/l/5ZHW63AUZEX2W6ZYK6UOYRPFGL:/var/lib/docker/overlay2/l/LLDKRW4LIKEF5BYN3JXKKXMEV7:/var/lib/docker/overlay2/l/Z4RH2C6LAWFX27JOMMNW5LDYCX:/var/lib/docker/overlay2/l/IKCZLFDYSFWYV47TZLGS5VCYPR:/var/lib/docker/overlay2/l/G235NFX6PH7PPGVFIM4BK64Y34:/var/lib/docker/overlay2/l/FNPBNMG4LL6MIUBDQ4B4QG4WN6:/var/lib/docker/overlay2/l/O77XTVQ6OQ3XTFLDGLFS25O3JW:/var/lib/docker/overlay2/l/56QFLVHKE6FWFYGMWI3SDUR324:/var/lib/docker/overlay2/l/TIIIY4QAG6KXR5NALJXOMKIEOF:/var/lib/do'
2022-05-18T04:23:02.7084469Z Unexpected end of /proc/mounts line `cker/overlay2/l/FIBTQ3YPAXTKOZUSR52W6EESDY:/var/lib/docker/overlay2/l/WIWIYM6T56THTWHFEA6754PHIG:/var/lib/docker/overlay2/l/NGGBA42OVTNKSHPVWHKGJXEUOY:/var/lib/docker/overlay2/l/DVGYNR5KYSTCGJLA3D6BDVIQHR:/var/lib/docker/overlay2/l/XFZA7CXFIZIMAF6ZI5Q7XM3JWK:/var/lib/docker/overlay2/l/ST3PKIHV6QHPNR2YRNVWT2V7XF:/var/lib/docker/overlay2/l/MOIGKSB36VVFDU2MR7QVTWCNAF:/var/lib/docker/overlay2/l/UDXVYMAMJQXTHJKWL74UGWUYQT:/var/lib/docker/overlay2/l/LC3CJTWQKUG254KV6SC4HYQC5G:/var/lib/docker/overlay2/l/AOH3MFWBF'
2022-05-18T04:23:02.7086416Z Unexpected end of /proc/mounts line `FSWCXAYYYFHTLZXJX:/var/lib/docker/overlay2/l/JJOUHZDV3WUD67CLUVOMJINLHD:/var/lib/docker/overlay2/l/62QBI7MI2ZNMDOX2LT3CB52LSH:/var/lib/docker/overlay2/l/6SSKWNVVD443YBEHTUI2VFFM2C:/var/lib/docker/overlay2/l/REZAAWKVT33XTKFOEOFIXPTYCQ:/var/lib/docker/overlay2/l/7QB7XP7VWEVSTNO5X6EOI2CDBX:/var/lib/docker/overlay2/l/XWMUJS7NF7JJNPXBKEZZZPTLAW:/var/lib/docker/overlay2/l/2E7XDS5VNZYTRZGRHMYGGGWJVV:/var/lib/docker/overlay2/l/3LBZKRCZUHVECLQDDXUH2P2AVE:/var/lib/docker/overlay2/l/M7PHRM4SJCHDDZENUKSBHBDZNR:/var/li'
2022-05-18T04:23:02.7088199Z Unexpected end of /proc/mounts line `b/docker/overlay2/l/R6GEGH3ZCU5QRS2X2FPSPGIOE6:/var/lib/docker/overlay2/l/CLJUBI6JQ3MN6R5NY47TCOAPFI:/var/lib/docker/overlay2/l/ABHVTATVVZMR7IMXXKQVATO4FB:/var/lib/docker/overlay2/l/WDIH2DYWBO7ICNTQ6FDBAZKJAX:/var/lib/docker/overlay2/l/CJE2MFOI46E3WT4Q3A2EBG3BIH:/var/lib/docker/overlay2/l/VIIF54KHXQLOEESIJWZWDI57U7:/var/lib/docker/overlay2/l/6UGTSVNSKNLXUR5DSWB7EQBY6H:/var/lib/docker/overlay2/l/ZTZYPP45AJNYD3E3D3W57JTER3:/var/lib/docker/overlay2/l/L2GNII2SGZQQTJGSGVI6G7QRAD:/var/lib/docker/overlay2/l/YASFI'
2022-05-18T04:23:02.7090240Z Unexpected end of /proc/mounts line `LPPWOITUTZYNYQFPQNTLU:/var/lib/docker/overlay2/l/JKF4DCJZS6LMOPTCWJ7JZ5X2NJ:/var/lib/docker/overlay2/l/ZD45LVBF3HHVQEQXFXENDRXKLH:/var/lib/docker/overlay2/l/AAOLME7VHHKQ7EELDNJRHUW5KN:/var/lib/docker/overlay2/l/7FWSB6SFPAQ2ZTB6O2EQWHPYPT:/var/lib/docker/overlay2/l/6VAFGR22OTLBGH2ETREFLZWH65:/var/lib/docker/overlay2/l/GTN6VK43UHKD5HYRH5MJSPMPGB,upperdir=/var/lib/docker/overlay2/de50816a31460444d6bba969e05d70c5c935e808e00a783dd8aff59d96bfa4d3/diff,workdir=/var/lib/docker/overlay2/de50816a31460444d6bba969e05d'
2022-05-18T04:23:03.8987790Z Test results will be stored in test-reports/dist-mpi/distributed.test_distributed_spawn
2022-05-18T04:23:03.9004361Z Test results will be stored in test-reports/dist-mpi/distributed.test_distributed_spawn
2022-05-18T04:23:03.9031245Z 
2022-05-18T04:23:03.9031623Z Running tests...
2022-05-18T04:23:03.9032453Z ----------------------------------------------------------------------
2022-05-18T04:23:03.9032963Z 
2022-05-18T04:23:03.9033535Z ----------------------------------------------------------------------
2022-05-18T04:23:03.9033969Z Ran 0 tests in 0.000s
2022-05-18T04:23:03.9034130Z 
2022-05-18T04:23:03.9034223Z OK
2022-05-18T04:23:03.9034357Z 
2022-05-18T04:23:03.9034505Z Generating XML reports...
2022-05-18T04:23:03.9053526Z 
2022-05-18T04:23:03.9053917Z Running tests...
2022-05-18T04:23:03.9054758Z ----------------------------------------------------------------------
2022-05-18T04:23:03.9055284Z 
2022-05-18T04:23:03.9055853Z ----------------------------------------------------------------------
2022-05-18T04:23:03.9056257Z Ran 0 tests in 0.000s
2022-05-18T04:23:03.9056399Z 
2022-05-18T04:23:03.9056489Z OK
2022-05-18T04:23:03.9056620Z 
2022-05-18T04:23:03.9056744Z Generating XML reports...
2022-05-18T04:23:03.9057214Z Test results will be stored in test-reports/dist-mpi/distributed.test_distributed_spawn
2022-05-18T04:23:03.9102376Z 
2022-05-18T04:23:03.9102864Z Running tests...
2022-05-18T04:23:03.9103726Z ----------------------------------------------------------------------
2022-05-18T04:23:03.9104256Z 
2022-05-18T04:23:03.9104833Z ----------------------------------------------------------------------
2022-05-18T04:23:03.9105195Z Ran 0 tests in 0.000s
2022-05-18T04:23:03.9105348Z 
2022-05-18T04:23:03.9105442Z OK
2022-05-18T04:23:03.9105573Z 
2022-05-18T04:23:03.9105697Z Generating XML reports...
2022-05-18T04:23:04.0517028Z Running distributed tests for the nccl backend with env init_method
2022-05-18T04:23:04.0519259Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:23:04.051578]
2022-05-18T04:23:05.1956221Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:23:05.1995980Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-05-18T04:23:05.2056893Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2057895Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2058744Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2059550Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2060426Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2061326Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2062283Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2063260Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2064267Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2065393Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2066511Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2067605Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2068641Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2069685Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2070640Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2071609Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2072519Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2073364Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2074201Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2075223Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2076185Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2077135Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2077943Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2078762Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2079622Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2080460Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2081275Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2082123Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2082928Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2083658Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2084452Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2085247Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2086005Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2086952Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2087741Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2088557Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2089364Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2090788Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2091675Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2092528Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2093419Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2094244Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2095087Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2095912Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2096758Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2097584Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2098447Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2099300Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2100117Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2100934Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2101791Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2102638Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2103443Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2104228Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2105041Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2105845Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2106610Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2107400Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2108189Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2108930Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2109639Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2110560Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2111380Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2112172Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2112954Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2113693Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2114416Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2115178Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2115958Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2116733Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2117536Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2118297Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2119029Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2119892Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2120643Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2121610Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2122377Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2123199Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2123953Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2124753Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2125576Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2126450Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2127338Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2128212Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2129117Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2130525Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2131406Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2132238Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2133094Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2133990Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2134864Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2135770Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2136705Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2137613Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2138466Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2139331Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2140075Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2140799Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2141535Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2142259Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2142993Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2143783Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2144540Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2145437Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2146226Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2147040Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2147821Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2148614Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2149425Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2150263Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2151063Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2151879Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2152709Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2153501Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2154320Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2155141Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2155954Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2156862Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2157591Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2158377Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2159125Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2159906Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2160698Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2161617Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2162630Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2163554Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2164435Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2165273Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2166143Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2167052Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2167966Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2168823Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2170166Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2171053Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2171845Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2172581Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2173333Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2174157Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2174957Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2175825Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2176729Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2177534Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2178432Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2179444Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2180722Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2181960Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2183197Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2184418Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2185618Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2186815Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2188054Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2189235Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2190504Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2191472Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2192344Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2193089Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2193842Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2194639Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2195389Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2196207Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2197062Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2197985Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2198912Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2199727Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2200498Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2201306Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2202162Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2203006Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2203837Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2204681Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2205545Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2206394Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2207223Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2208014Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2208824Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2210174Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2210968Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2211763Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2212690Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2213693Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2214424Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2215216Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2216052Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2216891Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2217626Z test_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2218338Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2219049Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2219777Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2220532Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2221262Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2221997Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2222784Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2223494Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2224170Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2225094Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2225882Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2226656Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2227360Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2228030Z test_isend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2228789Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2229573Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2230386Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2231298Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2232183Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2233005Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2233847Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2234736Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2235577Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2236423Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2237280Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2238072Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2238897Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2239696Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2240503Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2241278Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2242220Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2243181Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2244104Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2244980Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2245894Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2246806Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2247647Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2248592Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2249496Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2250635Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2251559Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2252545Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2253614Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2254535Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2255394Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2256227Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2256979Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2257744Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2258490Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2259323Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2260270Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2261012Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2261701Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2262414Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2263173Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2263894Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2264669Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2265432Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2266191Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2266897Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2267630Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2268366Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2269092Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2269856Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2270624Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2271361Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2272094Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2272827Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2273547Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2274368Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2275327Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2276219Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2277000Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2277780Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2278626Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2279458Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2280219Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2281045Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2281929Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2282748Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2283548Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2284507Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2285316Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2286061Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2286893Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2287784Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:05.2288629Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:06.3601679Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:06.3617670Z 
2022-05-18T04:23:06.3618000Z Running tests...
2022-05-18T04:23:06.3618715Z ----------------------------------------------------------------------
2022-05-18T04:23:08.0497931Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:08.0829679Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 615
2022-05-18T04:23:08.0927389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 616
2022-05-18T04:23:09.2354405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:09.2427047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:09.2427858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:09.2455199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:09.2461601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:09.3443006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:10.5406797Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:23:10.5407838Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:23:10.6462727Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:23:10.6463771Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:23:11.7139826Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:23:11.7140864Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:23:11.7141550Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:23:11.7142366Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:23:11.7283747Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:23:11.7284623Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:23:11.7289255Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:23:11.7290981Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:23:11.7432382Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:23:11.7433256Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:23:11.7437640Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:23:11.7438602Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:23:12.3026556Z ok (5.941s)
2022-05-18T04:23:12.3026790Z 
2022-05-18T04:23:12.3027172Z ----------------------------------------------------------------------
2022-05-18T04:23:12.3027492Z Ran 1 test in 5.941s
2022-05-18T04:23:12.3027658Z 
2022-05-18T04:23:12.3027755Z OK
2022-05-18T04:23:12.3027913Z 
2022-05-18T04:23:12.3028052Z Generating XML reports...
2022-05-18T04:23:12.3083751Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042306.xml
2022-05-18T04:23:13.7117836Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:13.7134733Z 
2022-05-18T04:23:13.7135101Z Running tests...
2022-05-18T04:23:13.7135621Z ----------------------------------------------------------------------
2022-05-18T04:23:13.7181912Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.005s)
2022-05-18T04:23:13.7182244Z 
2022-05-18T04:23:13.7182587Z ----------------------------------------------------------------------
2022-05-18T04:23:13.7183140Z Ran 1 test in 0.005s
2022-05-18T04:23:13.7183306Z 
2022-05-18T04:23:13.7183399Z OK (skipped=1)
2022-05-18T04:23:13.7183559Z 
2022-05-18T04:23:13.7183686Z Generating XML reports...
2022-05-18T04:23:13.7227201Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042313.xml
2022-05-18T04:23:15.0133047Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:15.0149410Z 
2022-05-18T04:23:15.0149726Z Running tests...
2022-05-18T04:23:15.0150187Z ----------------------------------------------------------------------
2022-05-18T04:23:16.6579905Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:16.6924329Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 773
2022-05-18T04:23:16.7024886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 774
2022-05-18T04:23:17.8723278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:17.8965287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:17.8966096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:17.9027190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:17.9033841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:17.9978728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:18.2073970Z ok (3.192s)
2022-05-18T04:23:18.2074200Z 
2022-05-18T04:23:18.2074565Z ----------------------------------------------------------------------
2022-05-18T04:23:18.2074910Z Ran 1 test in 3.192s
2022-05-18T04:23:18.2075081Z 
2022-05-18T04:23:18.2075178Z OK
2022-05-18T04:23:18.2075315Z 
2022-05-18T04:23:18.2075455Z Generating XML reports...
2022-05-18T04:23:18.2132936Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042315.xml
2022-05-18T04:23:19.6227977Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:19.6242858Z 
2022-05-18T04:23:19.6243144Z Running tests...
2022-05-18T04:23:19.6243576Z ----------------------------------------------------------------------
2022-05-18T04:23:21.2546673Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:21.2670342Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2022-05-18T04:23:21.2670959Z 
2022-05-18T04:23:21.2671258Z ----------------------------------------------------------------------
2022-05-18T04:23:21.2671591Z Ran 1 test in 1.643s
2022-05-18T04:23:21.2671770Z 
2022-05-18T04:23:21.2671881Z OK (skipped=1)
2022-05-18T04:23:21.2672036Z 
2022-05-18T04:23:21.2672143Z Generating XML reports...
2022-05-18T04:23:21.2710203Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042319.xml
2022-05-18T04:23:22.6549512Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:22.6564323Z 
2022-05-18T04:23:22.6564629Z Running tests...
2022-05-18T04:23:22.6565051Z ----------------------------------------------------------------------
2022-05-18T04:23:22.6583463Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-05-18T04:23:22.6584373Z 
2022-05-18T04:23:22.6584683Z ----------------------------------------------------------------------
2022-05-18T04:23:22.6585031Z Ran 1 test in 0.002s
2022-05-18T04:23:22.6585193Z 
2022-05-18T04:23:22.6585285Z OK (skipped=1)
2022-05-18T04:23:22.6585459Z 
2022-05-18T04:23:22.6585588Z Generating XML reports...
2022-05-18T04:23:22.6627526Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042322.xml
2022-05-18T04:23:23.9231887Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:23.9246370Z 
2022-05-18T04:23:23.9246815Z Running tests...
2022-05-18T04:23:23.9247257Z ----------------------------------------------------------------------
2022-05-18T04:23:23.9266174Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-05-18T04:23:23.9266545Z 
2022-05-18T04:23:23.9266827Z ----------------------------------------------------------------------
2022-05-18T04:23:23.9267160Z Ran 1 test in 0.002s
2022-05-18T04:23:23.9267304Z 
2022-05-18T04:23:23.9267414Z OK (skipped=1)
2022-05-18T04:23:23.9267569Z 
2022-05-18T04:23:23.9267709Z Generating XML reports...
2022-05-18T04:23:23.9310062Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042323.xml
2022-05-18T04:23:25.2034552Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:25.2049329Z 
2022-05-18T04:23:25.2049903Z Running tests...
2022-05-18T04:23:25.2050829Z ----------------------------------------------------------------------
2022-05-18T04:23:26.8392879Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:26.8741396Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 992
2022-05-18T04:23:26.8842036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 993
2022-05-18T04:23:28.0048806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:28.0264217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:28.0265084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:28.0352300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:28.0358947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:28.1278690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:29.3444601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxyqxa3zi
2022-05-18T04:23:29.3445236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxyqxa3zi/_remote_module_non_scriptable.py
2022-05-18T04:23:29.4346456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5oxycn4g
2022-05-18T04:23:29.4347519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5oxycn4g/_remote_module_non_scriptable.py
2022-05-18T04:23:31.0266872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:31.0268385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:31.0488561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:31.0491250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:31.0787277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:31.0789752Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:31.1005353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:31.1007581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:31.2278169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:31.2280605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:31.2496337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:31.2498693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:31.7952996Z ok (6.590s)
2022-05-18T04:23:31.7953384Z 
2022-05-18T04:23:31.7954055Z ----------------------------------------------------------------------
2022-05-18T04:23:31.7954692Z Ran 1 test in 6.590s
2022-05-18T04:23:31.7954996Z 
2022-05-18T04:23:31.7955137Z OK
2022-05-18T04:23:31.7955395Z 
2022-05-18T04:23:31.7955622Z Generating XML reports...
2022-05-18T04:23:31.8013964Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042325.xml
2022-05-18T04:23:33.2151661Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:33.2166642Z 
2022-05-18T04:23:33.2166795Z Running tests...
2022-05-18T04:23:33.2167518Z ----------------------------------------------------------------------
2022-05-18T04:23:34.8408421Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:34.8753226Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1119
2022-05-18T04:23:34.8862818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1120
2022-05-18T04:23:36.0663052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:36.0973670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:36.0974725Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:36.1070058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:36.1076871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:36.1988833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:37.3825937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgf6r5k5w
2022-05-18T04:23:37.3826556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgf6r5k5w/_remote_module_non_scriptable.py
2022-05-18T04:23:37.5000708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwhkm4ybn
2022-05-18T04:23:37.5002107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwhkm4ybn/_remote_module_non_scriptable.py
2022-05-18T04:23:38.5585596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:38.5586475Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:38.5728898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:38.5731652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:38.8956271Z ok (5.679s)
2022-05-18T04:23:38.8956790Z 
2022-05-18T04:23:38.8957576Z ----------------------------------------------------------------------
2022-05-18T04:23:38.8958235Z Ran 1 test in 5.679s
2022-05-18T04:23:38.8958554Z 
2022-05-18T04:23:38.8958700Z OK
2022-05-18T04:23:38.8958957Z 
2022-05-18T04:23:38.8959200Z Generating XML reports...
2022-05-18T04:23:38.9017387Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042333.xml
2022-05-18T04:23:40.3350488Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:40.3365545Z 
2022-05-18T04:23:40.3365985Z Running tests...
2022-05-18T04:23:40.3366497Z ----------------------------------------------------------------------
2022-05-18T04:23:41.9839169Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:42.0192017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1246
2022-05-18T04:23:42.0294176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1247
2022-05-18T04:23:43.1483989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:43.1659264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:43.1660046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:43.1686215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:43.1693322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:43.2674462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:44.4579413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl__etqn4
2022-05-18T04:23:44.4580024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl__etqn4/_remote_module_non_scriptable.py
2022-05-18T04:23:44.5568824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwhe6k_5e
2022-05-18T04:23:44.5571116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwhe6k_5e/_remote_module_non_scriptable.py
2022-05-18T04:23:45.6177531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:45.6178344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:45.6340952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:45.6342826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:45.9407097Z ok (5.604s)
2022-05-18T04:23:45.9407312Z 
2022-05-18T04:23:45.9407692Z ----------------------------------------------------------------------
2022-05-18T04:23:45.9408010Z Ran 1 test in 5.604s
2022-05-18T04:23:45.9408175Z 
2022-05-18T04:23:45.9408276Z OK
2022-05-18T04:23:45.9408410Z 
2022-05-18T04:23:45.9408542Z Generating XML reports...
2022-05-18T04:23:45.9467135Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042340.xml
2022-05-18T04:23:47.4046506Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:47.4062451Z 
2022-05-18T04:23:47.4062698Z Running tests...
2022-05-18T04:23:47.4063182Z ----------------------------------------------------------------------
2022-05-18T04:23:49.0477763Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:49.0836604Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1373
2022-05-18T04:23:49.0938337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1374
2022-05-18T04:23:50.2566495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:50.2765544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:50.2766357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:50.2768869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:50.2776194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:50.3780605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:51.5479179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp823v1tp5
2022-05-18T04:23:51.5479763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp823v1tp5/_remote_module_non_scriptable.py
2022-05-18T04:23:51.6680596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jtjwkkv
2022-05-18T04:23:51.6681703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jtjwkkv/_remote_module_non_scriptable.py
2022-05-18T04:23:52.7447178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:52.7447744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:53.3037830Z ok (5.897s)
2022-05-18T04:23:53.3038216Z 
2022-05-18T04:23:53.3038847Z ----------------------------------------------------------------------
2022-05-18T04:23:53.3039489Z Ran 1 test in 5.897s
2022-05-18T04:23:53.3039807Z 
2022-05-18T04:23:53.3039973Z OK
2022-05-18T04:23:53.3040254Z 
2022-05-18T04:23:53.3040941Z Generating XML reports...
2022-05-18T04:23:53.3096996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042347.xml
2022-05-18T04:23:54.7145803Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:54.7161294Z 
2022-05-18T04:23:54.7161649Z Running tests...
2022-05-18T04:23:54.7162089Z ----------------------------------------------------------------------
2022-05-18T04:23:56.3293537Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:56.3639750Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1500
2022-05-18T04:23:56.3740743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1501
2022-05-18T04:23:57.5317874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:57.5590385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:57.5591163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:57.5621565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:57.5628428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:57.6606117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:58.8530423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbc_lgvt
2022-05-18T04:23:58.8531048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbc_lgvt/_remote_module_non_scriptable.py
2022-05-18T04:23:58.9472360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxqj1t89
2022-05-18T04:23:58.9472956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxqj1t89/_remote_module_non_scriptable.py
2022-05-18T04:24:00.4361601Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:00.4362169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:00.4584210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:00.4584707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:00.8846620Z ok (6.168s)
2022-05-18T04:24:00.8846842Z 
2022-05-18T04:24:00.8847245Z ----------------------------------------------------------------------
2022-05-18T04:24:00.8847566Z Ran 1 test in 6.168s
2022-05-18T04:24:00.8847735Z 
2022-05-18T04:24:00.8847831Z OK
2022-05-18T04:24:00.8847964Z 
2022-05-18T04:24:00.8848111Z Generating XML reports...
2022-05-18T04:24:00.8904942Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042354.xml
2022-05-18T04:24:02.3479887Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:02.3495566Z 
2022-05-18T04:24:02.3495806Z Running tests...
2022-05-18T04:24:02.3496227Z ----------------------------------------------------------------------
2022-05-18T04:24:04.0423996Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:04.0781348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1627
2022-05-18T04:24:04.0885328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1628
2022-05-18T04:24:05.2443386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:05.2910013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:05.2910831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:05.2949447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:05.2956654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:05.3925445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:06.6157753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyc0fj4hq
2022-05-18T04:24:06.6158711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyc0fj4hq/_remote_module_non_scriptable.py
2022-05-18T04:24:06.7001392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzci52sho
2022-05-18T04:24:06.7002412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzci52sho/_remote_module_non_scriptable.py
2022-05-18T04:24:08.0532573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:08.0533113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:08.0710732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:08.0711245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:08.4989679Z ok (6.149s)
2022-05-18T04:24:08.4990058Z 
2022-05-18T04:24:08.4990675Z ----------------------------------------------------------------------
2022-05-18T04:24:08.4991304Z Ran 1 test in 6.149s
2022-05-18T04:24:08.4991639Z 
2022-05-18T04:24:08.4991807Z OK
2022-05-18T04:24:08.4992050Z 
2022-05-18T04:24:08.4992282Z Generating XML reports...
2022-05-18T04:24:08.5049154Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042402.xml
2022-05-18T04:24:09.9362368Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:09.9377732Z 
2022-05-18T04:24:09.9378206Z Running tests...
2022-05-18T04:24:09.9378698Z ----------------------------------------------------------------------
2022-05-18T04:24:11.5883051Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:11.6229933Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1754
2022-05-18T04:24:11.6331098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1755
2022-05-18T04:24:12.8071922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:12.8258555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:12.8259362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:12.8274571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:12.8281873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:12.9273859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:14.1457967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyexl0x5q
2022-05-18T04:24:14.1459307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyexl0x5q/_remote_module_non_scriptable.py
2022-05-18T04:24:14.2157332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrpnqrez
2022-05-18T04:24:14.2158833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrpnqrez/_remote_module_non_scriptable.py
2022-05-18T04:24:15.2530795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:15.2531365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:15.2674849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:15.2675363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:15.6428390Z ok (5.705s)
2022-05-18T04:24:15.6428725Z 
2022-05-18T04:24:15.6429375Z ----------------------------------------------------------------------
2022-05-18T04:24:15.6429702Z Ran 1 test in 5.705s
2022-05-18T04:24:15.6429866Z 
2022-05-18T04:24:15.6429958Z OK
2022-05-18T04:24:15.6430294Z 
2022-05-18T04:24:15.6430449Z Generating XML reports...
2022-05-18T04:24:15.6486601Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042409.xml
2022-05-18T04:24:17.0588093Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:17.0602809Z 
2022-05-18T04:24:17.0603075Z Running tests...
2022-05-18T04:24:17.0603526Z ----------------------------------------------------------------------
2022-05-18T04:24:18.6607777Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:18.6723746Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.612s)
2022-05-18T04:24:18.6724370Z 
2022-05-18T04:24:18.6724627Z ----------------------------------------------------------------------
2022-05-18T04:24:18.6724953Z Ran 1 test in 1.612s
2022-05-18T04:24:18.6725115Z 
2022-05-18T04:24:18.6725534Z OK (skipped=1)
2022-05-18T04:24:18.6725691Z 
2022-05-18T04:24:18.6725798Z Generating XML reports...
2022-05-18T04:24:18.6762244Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042417.xml
2022-05-18T04:24:20.0574120Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:20.0589019Z 
2022-05-18T04:24:20.0589482Z Running tests...
2022-05-18T04:24:20.0589929Z ----------------------------------------------------------------------
2022-05-18T04:24:21.7036507Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:21.7390882Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1917
2022-05-18T04:24:21.7492590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1918
2022-05-18T04:24:22.8284683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:22.8590823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:22.8591608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:22.8691257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:22.8698473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:22.9602832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:23.2543119Z ok (3.195s)
2022-05-18T04:24:23.2543349Z 
2022-05-18T04:24:23.2544045Z ----------------------------------------------------------------------
2022-05-18T04:24:23.2544383Z Ran 1 test in 3.195s
2022-05-18T04:24:23.2544551Z 
2022-05-18T04:24:23.2544645Z OK
2022-05-18T04:24:23.2545551Z 
2022-05-18T04:24:23.2545947Z Generating XML reports...
2022-05-18T04:24:23.2600464Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042420.xml
2022-05-18T04:24:24.7024883Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:24.7040748Z 
2022-05-18T04:24:24.7041345Z Running tests...
2022-05-18T04:24:24.7041863Z ----------------------------------------------------------------------
2022-05-18T04:24:26.3435880Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:26.3557158Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.651s)
2022-05-18T04:24:26.3557834Z 
2022-05-18T04:24:26.3558116Z ----------------------------------------------------------------------
2022-05-18T04:24:26.3558447Z Ran 1 test in 1.652s
2022-05-18T04:24:26.3558609Z 
2022-05-18T04:24:26.3558699Z OK (skipped=1)
2022-05-18T04:24:26.3558860Z 
2022-05-18T04:24:26.3558984Z Generating XML reports...
2022-05-18T04:24:26.3606035Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042424.xml
2022-05-18T04:24:27.7498521Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:27.7518339Z 
2022-05-18T04:24:27.7518463Z Running tests...
2022-05-18T04:24:27.7519170Z ----------------------------------------------------------------------
2022-05-18T04:24:29.4038729Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:29.4387014Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2066
2022-05-18T04:24:29.4487559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2067
2022-05-18T04:24:30.6321467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:30.6766889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:30.6767729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:30.6827643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:30.6834167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:30.7782110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:33.4583379Z ok (5.706s)
2022-05-18T04:24:33.4584372Z 
2022-05-18T04:24:33.4584773Z ----------------------------------------------------------------------
2022-05-18T04:24:33.4585130Z Ran 1 test in 5.706s
2022-05-18T04:24:33.4585297Z 
2022-05-18T04:24:33.4585390Z OK
2022-05-18T04:24:33.4585525Z 
2022-05-18T04:24:33.4585659Z Generating XML reports...
2022-05-18T04:24:33.4643176Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042427.xml
2022-05-18T04:24:34.8560863Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:34.8577189Z 
2022-05-18T04:24:34.8577561Z Running tests...
2022-05-18T04:24:34.8578004Z ----------------------------------------------------------------------
2022-05-18T04:24:34.8599253Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-05-18T04:24:34.8600231Z 
2022-05-18T04:24:34.8600764Z ----------------------------------------------------------------------
2022-05-18T04:24:34.8601301Z Ran 1 test in 0.002s
2022-05-18T04:24:34.8601477Z 
2022-05-18T04:24:34.8601591Z OK (skipped=1)
2022-05-18T04:24:34.8601750Z 
2022-05-18T04:24:34.8601872Z Generating XML reports...
2022-05-18T04:24:34.8644001Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042434.xml
2022-05-18T04:24:36.1334083Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:36.1350222Z 
2022-05-18T04:24:36.1350465Z Running tests...
2022-05-18T04:24:36.1350903Z ----------------------------------------------------------------------
2022-05-18T04:24:36.1370307Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:24:37.7761161Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:37.8112319Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2224
2022-05-18T04:24:37.8213300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2225
2022-05-18T04:24:39.0123022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:39.0129725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:39.0130531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:39.0225745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:39.0233188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:39.1145727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:40.3313727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87mj9biq
2022-05-18T04:24:40.3314375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87mj9biq/_remote_module_non_scriptable.py
2022-05-18T04:24:40.3892333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxln6311
2022-05-18T04:24:40.3893513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxln6311/_remote_module_non_scriptable.py
2022-05-18T04:24:42.0309681Z ok (5.896s)
2022-05-18T04:24:42.0309907Z 
2022-05-18T04:24:42.0310263Z ----------------------------------------------------------------------
2022-05-18T04:24:42.0310601Z Ran 1 test in 5.896s
2022-05-18T04:24:42.0310764Z 
2022-05-18T04:24:42.0310857Z OK
2022-05-18T04:24:42.0310997Z 
2022-05-18T04:24:42.0311128Z Generating XML reports...
2022-05-18T04:24:42.0378299Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042436.xml
2022-05-18T04:24:43.4615838Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:43.4630120Z 
2022-05-18T04:24:43.4630450Z Running tests...
2022-05-18T04:24:43.4631151Z ----------------------------------------------------------------------
2022-05-18T04:24:43.4654461Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:24:45.0837170Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:45.1187241Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2351
2022-05-18T04:24:45.1290991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2352
2022-05-18T04:24:46.3017375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:46.3179584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:46.3180412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:46.3222277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:46.3229402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:46.4193520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:47.6390828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjhsufcb
2022-05-18T04:24:47.6391436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjhsufcb/_remote_module_non_scriptable.py
2022-05-18T04:24:47.6985140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqlx8dxc
2022-05-18T04:24:47.6986293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqlx8dxc/_remote_module_non_scriptable.py
2022-05-18T04:24:48.9770135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:48.9775999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:49.3388824Z ok (5.876s)
2022-05-18T04:24:49.3389043Z 
2022-05-18T04:24:49.3389595Z ----------------------------------------------------------------------
2022-05-18T04:24:49.3390076Z Ran 1 test in 5.876s
2022-05-18T04:24:49.3390243Z 
2022-05-18T04:24:49.3390328Z OK
2022-05-18T04:24:49.3390461Z 
2022-05-18T04:24:49.3390593Z Generating XML reports...
2022-05-18T04:24:49.3446834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042443.xml
2022-05-18T04:24:50.7822561Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:50.7837709Z 
2022-05-18T04:24:50.7837938Z Running tests...
2022-05-18T04:24:50.7838378Z ----------------------------------------------------------------------
2022-05-18T04:24:50.7864314Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:24:52.4279695Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:52.4631951Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2478
2022-05-18T04:24:52.4734108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2479
2022-05-18T04:24:53.6722322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:53.6940893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:53.6941739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:53.7026275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:53.7032889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:53.7955709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:54.9826718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2b7qc_u
2022-05-18T04:24:54.9827332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2b7qc_u/_remote_module_non_scriptable.py
2022-05-18T04:24:55.0932609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8axc9bsf
2022-05-18T04:24:55.0933460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8axc9bsf/_remote_module_non_scriptable.py
2022-05-18T04:24:56.4284080Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:56.4284657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:56.7837865Z ok (6.000s)
2022-05-18T04:24:56.7838151Z 
2022-05-18T04:24:56.7838525Z ----------------------------------------------------------------------
2022-05-18T04:24:56.7838879Z Ran 1 test in 6.000s
2022-05-18T04:24:56.7839040Z 
2022-05-18T04:24:56.7839132Z OK
2022-05-18T04:24:56.7839266Z 
2022-05-18T04:24:56.7840305Z Generating XML reports...
2022-05-18T04:24:56.7896851Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042450.xml
2022-05-18T04:24:58.2314181Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:58.2329030Z 
2022-05-18T04:24:58.2329370Z Running tests...
2022-05-18T04:24:58.2330068Z ----------------------------------------------------------------------
2022-05-18T04:24:58.2349277Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:24:59.8820061Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:59.9180414Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2605
2022-05-18T04:24:59.9282817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2606
2022-05-18T04:25:01.0753414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:01.0966642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:01.0967455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:01.1057383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:01.1064565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:01.1981279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:02.4340227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbyj2n2v
2022-05-18T04:25:02.4341395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbyj2n2v/_remote_module_non_scriptable.py
2022-05-18T04:25:02.4987660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzx91k4hg
2022-05-18T04:25:02.4988950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzx91k4hg/_remote_module_non_scriptable.py
2022-05-18T04:25:04.1381244Z ok (5.905s)
2022-05-18T04:25:04.1381455Z 
2022-05-18T04:25:04.1381822Z ----------------------------------------------------------------------
2022-05-18T04:25:04.1382138Z Ran 1 test in 5.905s
2022-05-18T04:25:04.1382301Z 
2022-05-18T04:25:04.1382401Z OK
2022-05-18T04:25:04.1382537Z 
2022-05-18T04:25:04.1382670Z Generating XML reports...
2022-05-18T04:25:04.1439603Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042458.xml
2022-05-18T04:25:05.5632942Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:05.5648057Z 
2022-05-18T04:25:05.5648346Z Running tests...
2022-05-18T04:25:05.5648776Z ----------------------------------------------------------------------
2022-05-18T04:25:05.5668471Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:25:05.5668786Z 
2022-05-18T04:25:05.5669067Z ----------------------------------------------------------------------
2022-05-18T04:25:05.5669393Z Ran 1 test in 0.002s
2022-05-18T04:25:05.5669555Z 
2022-05-18T04:25:05.5669671Z OK (skipped=1)
2022-05-18T04:25:05.5669825Z 
2022-05-18T04:25:05.5669931Z Generating XML reports...
2022-05-18T04:25:05.5711945Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042505.xml
2022-05-18T04:25:06.8089139Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:06.8103837Z 
2022-05-18T04:25:06.8104280Z Running tests...
2022-05-18T04:25:06.8104798Z ----------------------------------------------------------------------
2022-05-18T04:25:06.8124390Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:25:06.8124734Z 
2022-05-18T04:25:06.8125015Z ----------------------------------------------------------------------
2022-05-18T04:25:06.8125326Z Ran 1 test in 0.002s
2022-05-18T04:25:06.8125488Z 
2022-05-18T04:25:06.8125597Z OK (skipped=1)
2022-05-18T04:25:06.8125750Z 
2022-05-18T04:25:06.8125875Z Generating XML reports...
2022-05-18T04:25:06.8166482Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042506.xml
2022-05-18T04:25:08.0422928Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:08.0439235Z 
2022-05-18T04:25:08.0439744Z Running tests...
2022-05-18T04:25:08.0440243Z ----------------------------------------------------------------------
2022-05-18T04:25:08.0460795Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:25:08.0461309Z 
2022-05-18T04:25:08.0461802Z ----------------------------------------------------------------------
2022-05-18T04:25:08.0462137Z Ran 1 test in 0.002s
2022-05-18T04:25:08.0462300Z 
2022-05-18T04:25:08.0462407Z OK (skipped=1)
2022-05-18T04:25:08.0462559Z 
2022-05-18T04:25:08.0462664Z Generating XML reports...
2022-05-18T04:25:08.0506710Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042508.xml
2022-05-18T04:25:09.3381077Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:09.3397462Z 
2022-05-18T04:25:09.3397835Z Running tests...
2022-05-18T04:25:09.3398339Z ----------------------------------------------------------------------
2022-05-18T04:25:09.3419232Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:25:09.3420169Z 
2022-05-18T04:25:09.3420780Z ----------------------------------------------------------------------
2022-05-18T04:25:09.3421268Z Ran 1 test in 0.002s
2022-05-18T04:25:09.3421435Z 
2022-05-18T04:25:09.3421541Z OK (skipped=1)
2022-05-18T04:25:09.3421697Z 
2022-05-18T04:25:09.3421819Z Generating XML reports...
2022-05-18T04:25:09.3465822Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042509.xml
2022-05-18T04:25:10.6159166Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:10.6174821Z 
2022-05-18T04:25:10.6175164Z Running tests...
2022-05-18T04:25:10.6175621Z ----------------------------------------------------------------------
2022-05-18T04:25:10.6195506Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:25:10.6196042Z 
2022-05-18T04:25:10.6196666Z ----------------------------------------------------------------------
2022-05-18T04:25:10.6197056Z Ran 1 test in 0.002s
2022-05-18T04:25:10.6197239Z 
2022-05-18T04:25:10.6197352Z OK (skipped=1)
2022-05-18T04:25:10.6197507Z 
2022-05-18T04:25:10.6197632Z Generating XML reports...
2022-05-18T04:25:10.6240425Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042510.xml
2022-05-18T04:25:11.8895115Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:11.8910704Z 
2022-05-18T04:25:11.8910972Z Running tests...
2022-05-18T04:25:11.8911423Z ----------------------------------------------------------------------
2022-05-18T04:25:11.8944658Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s)
2022-05-18T04:25:11.8945174Z 
2022-05-18T04:25:11.8945934Z ----------------------------------------------------------------------
2022-05-18T04:25:11.8946290Z Ran 1 test in 0.003s
2022-05-18T04:25:11.8946458Z 
2022-05-18T04:25:11.8946570Z OK (skipped=1)
2022-05-18T04:25:11.8946733Z 
2022-05-18T04:25:11.8946857Z Generating XML reports...
2022-05-18T04:25:11.8989203Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042511.xml
2022-05-18T04:25:13.1683430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:13.1698655Z 
2022-05-18T04:25:13.1698898Z Running tests...
2022-05-18T04:25:13.1699331Z ----------------------------------------------------------------------
2022-05-18T04:25:13.1719896Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:25:13.1720239Z 
2022-05-18T04:25:13.1720509Z ----------------------------------------------------------------------
2022-05-18T04:25:13.1720846Z Ran 1 test in 0.002s
2022-05-18T04:25:13.1721007Z 
2022-05-18T04:25:13.1721114Z OK (skipped=1)
2022-05-18T04:25:13.1721267Z 
2022-05-18T04:25:13.1721389Z Generating XML reports...
2022-05-18T04:25:13.1763728Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042513.xml
2022-05-18T04:25:14.4446441Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:14.4463089Z 
2022-05-18T04:25:14.4463360Z Running tests...
2022-05-18T04:25:14.4463790Z ----------------------------------------------------------------------
2022-05-18T04:25:16.1082478Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:16.1442741Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2977
2022-05-18T04:25:16.1547265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2978
2022-05-18T04:25:17.3646447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:17.3905705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:17.3906481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:17.3950341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:17.3956641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:17.4921620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:21.4664521Z ok (7.020s)
2022-05-18T04:25:21.4664823Z 
2022-05-18T04:25:21.4665374Z ----------------------------------------------------------------------
2022-05-18T04:25:21.4665725Z Ran 1 test in 7.020s
2022-05-18T04:25:21.4665900Z 
2022-05-18T04:25:21.4665993Z OK
2022-05-18T04:25:21.4666148Z 
2022-05-18T04:25:21.4666399Z Generating XML reports...
2022-05-18T04:25:21.4722292Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042514.xml
2022-05-18T04:25:22.9058456Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:22.9074939Z 
2022-05-18T04:25:22.9075561Z Running tests...
2022-05-18T04:25:22.9076072Z ----------------------------------------------------------------------
2022-05-18T04:25:24.5586355Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:24.5946721Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3105
2022-05-18T04:25:24.6050813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3106
2022-05-18T04:25:25.7346261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:25.7825432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:25.7826244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:25.7852237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:25.7858951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:25.8840419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:29.8165281Z ok (6.909s)
2022-05-18T04:25:29.8165513Z 
2022-05-18T04:25:29.8166408Z ----------------------------------------------------------------------
2022-05-18T04:25:29.8166757Z Ran 1 test in 6.909s
2022-05-18T04:25:29.8166928Z 
2022-05-18T04:25:29.8167033Z OK
2022-05-18T04:25:29.8167178Z 
2022-05-18T04:25:29.8167311Z Generating XML reports...
2022-05-18T04:25:29.8223594Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042522.xml
2022-05-18T04:25:31.2374294Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:31.2388806Z 
2022-05-18T04:25:31.2389222Z Running tests...
2022-05-18T04:25:31.2389707Z ----------------------------------------------------------------------
2022-05-18T04:25:31.2409068Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:25:31.2409398Z 
2022-05-18T04:25:31.2410209Z ----------------------------------------------------------------------
2022-05-18T04:25:31.2410573Z Ran 1 test in 0.002s
2022-05-18T04:25:31.2410736Z 
2022-05-18T04:25:31.2410853Z OK (skipped=1)
2022-05-18T04:25:31.2411009Z 
2022-05-18T04:25:31.2411117Z Generating XML reports...
2022-05-18T04:25:31.2450994Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042531.xml
2022-05-18T04:25:32.4723781Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:32.4739412Z 
2022-05-18T04:25:32.4739962Z Running tests...
2022-05-18T04:25:32.4740477Z ----------------------------------------------------------------------
2022-05-18T04:25:32.4759683Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:25:32.4760335Z 
2022-05-18T04:25:32.4760944Z ----------------------------------------------------------------------
2022-05-18T04:25:32.4761445Z Ran 1 test in 0.002s
2022-05-18T04:25:32.4761609Z 
2022-05-18T04:25:32.4761735Z OK (skipped=1)
2022-05-18T04:25:32.4761890Z 
2022-05-18T04:25:32.4762020Z Generating XML reports...
2022-05-18T04:25:32.4803387Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042532.xml
2022-05-18T04:25:33.7443127Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:33.7458503Z 
2022-05-18T04:25:33.7458766Z Running tests...
2022-05-18T04:25:33.7459461Z ----------------------------------------------------------------------
2022-05-18T04:25:35.3806535Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:35.4167018Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3303
2022-05-18T04:25:35.4270349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3304
2022-05-18T04:25:36.5149409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:36.5690275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:36.5691361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:36.5757466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:36.5765046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:36.6706457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:38.7363579Z ok (4.990s)
2022-05-18T04:25:38.7363808Z 
2022-05-18T04:25:38.7364206Z ----------------------------------------------------------------------
2022-05-18T04:25:38.7364523Z Ran 1 test in 4.990s
2022-05-18T04:25:38.7364687Z 
2022-05-18T04:25:38.7364782Z OK
2022-05-18T04:25:38.7364924Z 
2022-05-18T04:25:38.7365319Z Generating XML reports...
2022-05-18T04:25:38.7421546Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042533.xml
2022-05-18T04:25:40.1530925Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:40.1545263Z 
2022-05-18T04:25:40.1545514Z Running tests...
2022-05-18T04:25:40.1545952Z ----------------------------------------------------------------------
2022-05-18T04:25:41.7639197Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:41.7991754Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3429
2022-05-18T04:25:41.8094990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3430
2022-05-18T04:25:42.9742626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:42.9837128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:42.9838016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:42.9844062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:42.9850535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:43.0851710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:45.5183369Z ok (5.363s)
2022-05-18T04:25:45.5183592Z 
2022-05-18T04:25:45.5183981Z ----------------------------------------------------------------------
2022-05-18T04:25:45.5184301Z Ran 1 test in 5.364s
2022-05-18T04:25:45.5184467Z 
2022-05-18T04:25:45.5184564Z OK
2022-05-18T04:25:45.5184699Z 
2022-05-18T04:25:45.5184837Z Generating XML reports...
2022-05-18T04:25:45.5241736Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042540.xml
2022-05-18T04:25:46.9560416Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:46.9576469Z 
2022-05-18T04:25:46.9576970Z Running tests...
2022-05-18T04:25:46.9577479Z ----------------------------------------------------------------------
2022-05-18T04:25:48.6050559Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:48.6415034Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3555
2022-05-18T04:25:48.6518358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3556
2022-05-18T04:25:49.8226475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:49.8422097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:49.8422918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:49.8429597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:49.8437040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:49.9438834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:52.5608252Z ok (5.603s)
2022-05-18T04:25:52.5612731Z 
2022-05-18T04:25:52.5613443Z ----------------------------------------------------------------------
2022-05-18T04:25:52.5613890Z Ran 1 test in 5.603s
2022-05-18T04:25:52.5614193Z 
2022-05-18T04:25:52.5614324Z OK
2022-05-18T04:25:52.5614475Z 
2022-05-18T04:25:52.5614593Z Generating XML reports...
2022-05-18T04:25:52.5672975Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042546.xml
2022-05-18T04:25:54.0031149Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:54.0046759Z 
2022-05-18T04:25:54.0047183Z Running tests...
2022-05-18T04:25:54.0047669Z ----------------------------------------------------------------------
2022-05-18T04:25:55.6363397Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:55.6728507Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3678
2022-05-18T04:25:55.6832973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3679
2022-05-18T04:25:56.8403269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:56.8739944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:56.8740928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:56.8808796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:56.8815889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:56.9752526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:56.9963867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:25:56.9964595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:25:56.9965295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:25:56.9965997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:26:00.4017543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:26:00.4018103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:26:00.4018903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:26:00.4019602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:26:00.4464808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:26:00.4465369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:26:00.4466151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:26:00.4466857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:26:00.8948057Z ok (6.890s)
2022-05-18T04:26:00.8948305Z 
2022-05-18T04:26:00.8948688Z ----------------------------------------------------------------------
2022-05-18T04:26:00.8949027Z Ran 1 test in 6.890s
2022-05-18T04:26:00.8949195Z 
2022-05-18T04:26:00.8949289Z OK
2022-05-18T04:26:00.8949423Z 
2022-05-18T04:26:00.8949537Z Generating XML reports...
2022-05-18T04:26:00.9006430Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042553.xml
2022-05-18T04:26:02.3413596Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:02.3429344Z 
2022-05-18T04:26:02.3429569Z Running tests...
2022-05-18T04:26:02.3430448Z ----------------------------------------------------------------------
2022-05-18T04:26:02.3451314Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:26:02.3451950Z 
2022-05-18T04:26:02.3452242Z ----------------------------------------------------------------------
2022-05-18T04:26:02.3452584Z Ran 1 test in 0.002s
2022-05-18T04:26:02.3452755Z 
2022-05-18T04:26:02.3452865Z OK (skipped=1)
2022-05-18T04:26:02.3453017Z 
2022-05-18T04:26:02.3453125Z Generating XML reports...
2022-05-18T04:26:02.3495234Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042602.xml
2022-05-18T04:26:03.6229269Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:03.6244550Z 
2022-05-18T04:26:03.6244887Z Running tests...
2022-05-18T04:26:03.6245332Z ----------------------------------------------------------------------
2022-05-18T04:26:03.6266716Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:26:03.6267332Z 
2022-05-18T04:26:03.6267629Z ----------------------------------------------------------------------
2022-05-18T04:26:03.6267945Z Ran 1 test in 0.002s
2022-05-18T04:26:03.6268505Z 
2022-05-18T04:26:03.6268693Z OK (skipped=1)
2022-05-18T04:26:03.6268924Z 
2022-05-18T04:26:03.6269052Z Generating XML reports...
2022-05-18T04:26:03.6311011Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042603.xml
2022-05-18T04:26:04.9065584Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:04.9080483Z 
2022-05-18T04:26:04.9080756Z Running tests...
2022-05-18T04:26:04.9081199Z ----------------------------------------------------------------------
2022-05-18T04:26:04.9103015Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:26:04.9103538Z 
2022-05-18T04:26:04.9103927Z ----------------------------------------------------------------------
2022-05-18T04:26:04.9104280Z Ran 1 test in 0.002s
2022-05-18T04:26:04.9104424Z 
2022-05-18T04:26:04.9104552Z OK (skipped=1)
2022-05-18T04:26:04.9104711Z 
2022-05-18T04:26:04.9104835Z Generating XML reports...
2022-05-18T04:26:04.9146381Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042604.xml
2022-05-18T04:26:06.1829767Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:06.1845327Z 
2022-05-18T04:26:06.1845905Z Running tests...
2022-05-18T04:26:06.1846543Z ----------------------------------------------------------------------
2022-05-18T04:26:06.1867224Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:26:06.1867632Z 
2022-05-18T04:26:06.1867929Z ----------------------------------------------------------------------
2022-05-18T04:26:06.1868297Z Ran 1 test in 0.002s
2022-05-18T04:26:06.1868604Z 
2022-05-18T04:26:06.1868787Z OK (skipped=1)
2022-05-18T04:26:06.1868948Z 
2022-05-18T04:26:06.1869095Z Generating XML reports...
2022-05-18T04:26:06.1911449Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042606.xml
2022-05-18T04:26:07.4601592Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:07.4617083Z 
2022-05-18T04:26:07.4617547Z Running tests...
2022-05-18T04:26:07.4618282Z ----------------------------------------------------------------------
2022-05-18T04:26:07.4638776Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:26:07.4639153Z 
2022-05-18T04:26:07.4639651Z ----------------------------------------------------------------------
2022-05-18T04:26:07.4640264Z Ran 1 test in 0.002s
2022-05-18T04:26:07.4640443Z 
2022-05-18T04:26:07.4640553Z OK (skipped=1)
2022-05-18T04:26:07.4640710Z 
2022-05-18T04:26:07.4640838Z Generating XML reports...
2022-05-18T04:26:07.4682183Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042607.xml
2022-05-18T04:26:08.7410304Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:08.7425379Z 
2022-05-18T04:26:08.7425530Z Running tests...
2022-05-18T04:26:08.7426286Z ----------------------------------------------------------------------
2022-05-18T04:26:08.7446852Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:26:08.7447191Z 
2022-05-18T04:26:08.7447461Z ----------------------------------------------------------------------
2022-05-18T04:26:08.7447787Z Ran 1 test in 0.002s
2022-05-18T04:26:08.7447949Z 
2022-05-18T04:26:08.7448085Z OK (skipped=1)
2022-05-18T04:26:08.7448242Z 
2022-05-18T04:26:08.7448349Z Generating XML reports...
2022-05-18T04:26:08.7491012Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042608.xml
2022-05-18T04:26:10.0215364Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:10.0230156Z 
2022-05-18T04:26:10.0230524Z Running tests...
2022-05-18T04:26:10.0231145Z ----------------------------------------------------------------------
2022-05-18T04:26:10.0251921Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:26:10.0252595Z 
2022-05-18T04:26:10.0253027Z ----------------------------------------------------------------------
2022-05-18T04:26:10.0253738Z Ran 1 test in 0.002s
2022-05-18T04:26:10.0254252Z 
2022-05-18T04:26:10.0254494Z OK (skipped=1)
2022-05-18T04:26:10.0254699Z 
2022-05-18T04:26:10.0254877Z Generating XML reports...
2022-05-18T04:26:10.0296398Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042610.xml
2022-05-18T04:26:11.2819719Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:11.2833990Z 
2022-05-18T04:26:11.2834422Z Running tests...
2022-05-18T04:26:11.2835514Z ----------------------------------------------------------------------
2022-05-18T04:26:11.2855851Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:26:11.2856666Z 
2022-05-18T04:26:11.2857232Z ----------------------------------------------------------------------
2022-05-18T04:26:11.2857718Z Ran 1 test in 0.002s
2022-05-18T04:26:11.2857916Z 
2022-05-18T04:26:11.2858069Z OK (skipped=1)
2022-05-18T04:26:11.2858262Z 
2022-05-18T04:26:11.2858369Z Generating XML reports...
2022-05-18T04:26:11.2899184Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042611.xml
2022-05-18T04:26:12.5561745Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:12.5576798Z 
2022-05-18T04:26:12.5577174Z Running tests...
2022-05-18T04:26:12.5578129Z ----------------------------------------------------------------------
2022-05-18T04:26:12.5598385Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:26:12.5599147Z 
2022-05-18T04:26:12.5599496Z ----------------------------------------------------------------------
2022-05-18T04:26:12.5599854Z Ran 1 test in 0.002s
2022-05-18T04:26:12.5600054Z 
2022-05-18T04:26:12.5617986Z OK (skipped=1)
2022-05-18T04:26:12.5618175Z 
2022-05-18T04:26:12.5618310Z Generating XML reports...
2022-05-18T04:26:12.5642156Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042612.xml
2022-05-18T04:26:13.8381985Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:13.8397420Z 
2022-05-18T04:26:13.8397686Z Running tests...
2022-05-18T04:26:13.8398404Z ----------------------------------------------------------------------
2022-05-18T04:26:13.8419418Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:13.8419963Z 
2022-05-18T04:26:13.8420232Z ----------------------------------------------------------------------
2022-05-18T04:26:13.8420567Z Ran 1 test in 0.002s
2022-05-18T04:26:13.8420797Z 
2022-05-18T04:26:13.8420997Z OK (skipped=1)
2022-05-18T04:26:13.8421256Z 
2022-05-18T04:26:13.8421382Z Generating XML reports...
2022-05-18T04:26:13.8464180Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042613.xml
2022-05-18T04:26:15.1260944Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:15.1275757Z 
2022-05-18T04:26:15.1275899Z Running tests...
2022-05-18T04:26:15.1276644Z ----------------------------------------------------------------------
2022-05-18T04:26:15.1297574Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:26:15.1297899Z 
2022-05-18T04:26:15.1298146Z ----------------------------------------------------------------------
2022-05-18T04:26:15.1298471Z Ran 1 test in 0.002s
2022-05-18T04:26:15.1298635Z 
2022-05-18T04:26:15.1298746Z OK (skipped=1)
2022-05-18T04:26:15.1298902Z 
2022-05-18T04:26:15.1299007Z Generating XML reports...
2022-05-18T04:26:15.1341500Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042615.xml
2022-05-18T04:26:16.4039121Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:16.4054368Z 
2022-05-18T04:26:16.4054506Z Running tests...
2022-05-18T04:26:16.4055248Z ----------------------------------------------------------------------
2022-05-18T04:26:16.4075886Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:26:16.4076234Z 
2022-05-18T04:26:16.4076502Z ----------------------------------------------------------------------
2022-05-18T04:26:16.4076830Z Ran 1 test in 0.002s
2022-05-18T04:26:16.4076994Z 
2022-05-18T04:26:16.4077104Z OK (skipped=1)
2022-05-18T04:26:16.4077239Z 
2022-05-18T04:26:16.4077365Z Generating XML reports...
2022-05-18T04:26:16.4119537Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042616.xml
2022-05-18T04:26:17.6935218Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:17.6949921Z 
2022-05-18T04:26:17.6950606Z Running tests...
2022-05-18T04:26:17.6951588Z ----------------------------------------------------------------------
2022-05-18T04:26:17.6971612Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:26:17.6972100Z 
2022-05-18T04:26:17.6972668Z ----------------------------------------------------------------------
2022-05-18T04:26:17.6973348Z Ran 1 test in 0.002s
2022-05-18T04:26:17.6973579Z 
2022-05-18T04:26:17.6973688Z OK (skipped=1)
2022-05-18T04:26:17.6973846Z 
2022-05-18T04:26:17.6973973Z Generating XML reports...
2022-05-18T04:26:17.7015146Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042617.xml
2022-05-18T04:26:18.9686273Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:18.9701707Z 
2022-05-18T04:26:18.9702060Z Running tests...
2022-05-18T04:26:18.9702691Z ----------------------------------------------------------------------
2022-05-18T04:26:18.9726121Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:18.9726476Z 
2022-05-18T04:26:18.9726770Z ----------------------------------------------------------------------
2022-05-18T04:26:18.9727100Z Ran 1 test in 0.003s
2022-05-18T04:26:18.9727263Z 
2022-05-18T04:26:18.9727374Z OK (skipped=1)
2022-05-18T04:26:18.9727536Z 
2022-05-18T04:26:18.9727645Z Generating XML reports...
2022-05-18T04:26:18.9770073Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042618.xml
2022-05-18T04:26:20.2566304Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:20.2581843Z 
2022-05-18T04:26:20.2582150Z Running tests...
2022-05-18T04:26:20.2582826Z ----------------------------------------------------------------------
2022-05-18T04:26:20.2602868Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:20.2603495Z 
2022-05-18T04:26:20.2604451Z ----------------------------------------------------------------------
2022-05-18T04:26:20.2604764Z Ran 1 test in 0.002s
2022-05-18T04:26:20.2604927Z 
2022-05-18T04:26:20.2605039Z OK (skipped=1)
2022-05-18T04:26:20.2605194Z 
2022-05-18T04:26:20.2605318Z Generating XML reports...
2022-05-18T04:26:20.2647286Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042620.xml
2022-05-18T04:26:21.5121110Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:21.5137372Z 
2022-05-18T04:26:21.5137856Z Running tests...
2022-05-18T04:26:21.5138333Z ----------------------------------------------------------------------
2022-05-18T04:26:21.5158593Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:21.5158949Z 
2022-05-18T04:26:21.5159226Z ----------------------------------------------------------------------
2022-05-18T04:26:21.5159796Z Ran 1 test in 0.002s
2022-05-18T04:26:21.5159968Z 
2022-05-18T04:26:21.5160084Z OK (skipped=1)
2022-05-18T04:26:21.5160238Z 
2022-05-18T04:26:21.5160366Z Generating XML reports...
2022-05-18T04:26:21.5202279Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042621.xml
2022-05-18T04:26:22.7821378Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:22.7836191Z 
2022-05-18T04:26:22.7836336Z Running tests...
2022-05-18T04:26:22.7837079Z ----------------------------------------------------------------------
2022-05-18T04:26:22.7858878Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:22.7859210Z 
2022-05-18T04:26:22.7859491Z ----------------------------------------------------------------------
2022-05-18T04:26:22.7859802Z Ran 1 test in 0.002s
2022-05-18T04:26:22.7859975Z 
2022-05-18T04:26:22.7860087Z OK (skipped=1)
2022-05-18T04:26:22.7860242Z 
2022-05-18T04:26:22.7860366Z Generating XML reports...
2022-05-18T04:26:22.7902617Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042622.xml
2022-05-18T04:26:24.0379398Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:24.0394449Z 
2022-05-18T04:26:24.0394839Z Running tests...
2022-05-18T04:26:24.0395340Z ----------------------------------------------------------------------
2022-05-18T04:26:24.0416231Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:24.0416554Z 
2022-05-18T04:26:24.0417449Z ----------------------------------------------------------------------
2022-05-18T04:26:24.0417823Z Ran 1 test in 0.002s
2022-05-18T04:26:24.0417990Z 
2022-05-18T04:26:24.0418114Z OK (skipped=1)
2022-05-18T04:26:24.0418286Z 
2022-05-18T04:26:24.0418395Z Generating XML reports...
2022-05-18T04:26:24.0459105Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042624.xml
2022-05-18T04:26:25.3225475Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:25.3240820Z 
2022-05-18T04:26:25.3240958Z Running tests...
2022-05-18T04:26:25.3241672Z ----------------------------------------------------------------------
2022-05-18T04:26:25.3262644Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:25.3262965Z 
2022-05-18T04:26:25.3263248Z ----------------------------------------------------------------------
2022-05-18T04:26:25.3263588Z Ran 1 test in 0.002s
2022-05-18T04:26:25.3263736Z 
2022-05-18T04:26:25.3263846Z OK (skipped=1)
2022-05-18T04:26:25.3264003Z 
2022-05-18T04:26:25.3264136Z Generating XML reports...
2022-05-18T04:26:25.3306696Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042625.xml
2022-05-18T04:26:26.5689289Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:26.5706060Z 
2022-05-18T04:26:26.5706510Z Running tests...
2022-05-18T04:26:26.5707363Z ----------------------------------------------------------------------
2022-05-18T04:26:26.5729195Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:26.5730123Z 
2022-05-18T04:26:26.5730711Z ----------------------------------------------------------------------
2022-05-18T04:26:26.5731359Z Ran 1 test in 0.002s
2022-05-18T04:26:26.5731676Z 
2022-05-18T04:26:26.5731883Z OK (skipped=1)
2022-05-18T04:26:26.5732178Z 
2022-05-18T04:26:26.5732412Z Generating XML reports...
2022-05-18T04:26:26.5776132Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042626.xml
2022-05-18T04:26:27.8239942Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:27.8254621Z 
2022-05-18T04:26:27.8254925Z Running tests...
2022-05-18T04:26:27.8255553Z ----------------------------------------------------------------------
2022-05-18T04:26:27.8276932Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:27.8277340Z 
2022-05-18T04:26:27.8277624Z ----------------------------------------------------------------------
2022-05-18T04:26:27.8277933Z Ran 1 test in 0.002s
2022-05-18T04:26:27.8278211Z 
2022-05-18T04:26:27.8278408Z OK (skipped=1)
2022-05-18T04:26:27.8278620Z 
2022-05-18T04:26:27.8278759Z Generating XML reports...
2022-05-18T04:26:27.8318822Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042627.xml
2022-05-18T04:26:29.0995994Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:29.1011091Z 
2022-05-18T04:26:29.1011394Z Running tests...
2022-05-18T04:26:29.1011836Z ----------------------------------------------------------------------
2022-05-18T04:26:29.1034132Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:29.1034618Z 
2022-05-18T04:26:29.1034910Z ----------------------------------------------------------------------
2022-05-18T04:26:29.1035239Z Ran 1 test in 0.002s
2022-05-18T04:26:29.1035403Z 
2022-05-18T04:26:29.1035515Z OK (skipped=1)
2022-05-18T04:26:29.1035670Z 
2022-05-18T04:26:29.1035776Z Generating XML reports...
2022-05-18T04:26:29.1078647Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042629.xml
2022-05-18T04:26:30.3818164Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:30.3833734Z 
2022-05-18T04:26:30.3833986Z Running tests...
2022-05-18T04:26:30.3834422Z ----------------------------------------------------------------------
2022-05-18T04:26:30.3855485Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:30.3855797Z 
2022-05-18T04:26:30.3856088Z ----------------------------------------------------------------------
2022-05-18T04:26:30.3856398Z Ran 1 test in 0.002s
2022-05-18T04:26:30.3856560Z 
2022-05-18T04:26:30.3856670Z OK (skipped=1)
2022-05-18T04:26:30.3856833Z 
2022-05-18T04:26:30.3856959Z Generating XML reports...
2022-05-18T04:26:30.3899766Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042630.xml
2022-05-18T04:26:31.6610868Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:31.6626125Z 
2022-05-18T04:26:31.6626362Z Running tests...
2022-05-18T04:26:31.6627132Z ----------------------------------------------------------------------
2022-05-18T04:26:31.6647251Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:31.6647558Z 
2022-05-18T04:26:31.6647837Z ----------------------------------------------------------------------
2022-05-18T04:26:31.6648148Z Ran 1 test in 0.002s
2022-05-18T04:26:31.6648321Z 
2022-05-18T04:26:31.6648436Z OK (skipped=1)
2022-05-18T04:26:31.6648595Z 
2022-05-18T04:26:31.6648720Z Generating XML reports...
2022-05-18T04:26:31.6691259Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042631.xml
2022-05-18T04:26:32.9502237Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:32.9517628Z 
2022-05-18T04:26:32.9517878Z Running tests...
2022-05-18T04:26:32.9518323Z ----------------------------------------------------------------------
2022-05-18T04:26:32.9542444Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-05-18T04:26:32.9542775Z 
2022-05-18T04:26:32.9543055Z ----------------------------------------------------------------------
2022-05-18T04:26:32.9543383Z Ran 1 test in 0.003s
2022-05-18T04:26:32.9543545Z 
2022-05-18T04:26:32.9543654Z OK (skipped=1)
2022-05-18T04:26:32.9543810Z 
2022-05-18T04:26:32.9543916Z Generating XML reports...
2022-05-18T04:26:32.9586478Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042632.xml
2022-05-18T04:26:34.2331281Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:34.2346938Z 
2022-05-18T04:26:34.2347102Z Running tests...
2022-05-18T04:26:34.2347578Z ----------------------------------------------------------------------
2022-05-18T04:26:34.2372755Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-05-18T04:26:34.2373114Z 
2022-05-18T04:26:34.2373403Z ----------------------------------------------------------------------
2022-05-18T04:26:34.2373716Z Ran 1 test in 0.003s
2022-05-18T04:26:34.2373884Z 
2022-05-18T04:26:34.2373995Z OK (skipped=1)
2022-05-18T04:26:34.2374152Z 
2022-05-18T04:26:34.2374279Z Generating XML reports...
2022-05-18T04:26:34.2417045Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042634.xml
2022-05-18T04:26:35.5163966Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:35.5179929Z 
2022-05-18T04:26:35.5180389Z Running tests...
2022-05-18T04:26:35.5181163Z ----------------------------------------------------------------------
2022-05-18T04:26:35.5203007Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:35.5203339Z 
2022-05-18T04:26:35.5203630Z ----------------------------------------------------------------------
2022-05-18T04:26:35.5203963Z Ran 1 test in 0.002s
2022-05-18T04:26:35.5204129Z 
2022-05-18T04:26:35.5204220Z OK (skipped=1)
2022-05-18T04:26:35.5204375Z 
2022-05-18T04:26:35.5204500Z Generating XML reports...
2022-05-18T04:26:35.5247050Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042635.xml
2022-05-18T04:26:36.7589375Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:36.7604416Z 
2022-05-18T04:26:36.7604663Z Running tests...
2022-05-18T04:26:36.7605093Z ----------------------------------------------------------------------
2022-05-18T04:26:38.4288818Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:38.4645551Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4786
2022-05-18T04:26:38.4748757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4787
2022-05-18T04:26:39.6221439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:39.6332295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:39.6333116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:39.6424030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:39.6430876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:39.7348224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:42.2841393Z ok (5.523s)
2022-05-18T04:26:42.2841639Z 
2022-05-18T04:26:42.2842035Z ----------------------------------------------------------------------
2022-05-18T04:26:42.2842399Z Ran 1 test in 5.524s
2022-05-18T04:26:42.2842564Z 
2022-05-18T04:26:42.2842662Z OK
2022-05-18T04:26:42.2845353Z 
2022-05-18T04:26:42.2845713Z Generating XML reports...
2022-05-18T04:26:42.2905316Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042636.xml
2022-05-18T04:26:43.7295347Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:43.7311196Z 
2022-05-18T04:26:43.7311430Z Running tests...
2022-05-18T04:26:43.7311876Z ----------------------------------------------------------------------
2022-05-18T04:26:43.7333535Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:43.7333840Z 
2022-05-18T04:26:43.7334130Z ----------------------------------------------------------------------
2022-05-18T04:26:43.7334446Z Ran 1 test in 0.002s
2022-05-18T04:26:43.7334628Z 
2022-05-18T04:26:43.7334743Z OK (skipped=1)
2022-05-18T04:26:43.7334901Z 
2022-05-18T04:26:43.7335029Z Generating XML reports...
2022-05-18T04:26:43.7377506Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042643.xml
2022-05-18T04:26:45.0273875Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:45.0289824Z 
2022-05-18T04:26:45.0290077Z Running tests...
2022-05-18T04:26:45.0290818Z ----------------------------------------------------------------------
2022-05-18T04:26:45.0312884Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:45.0313199Z 
2022-05-18T04:26:45.0313760Z ----------------------------------------------------------------------
2022-05-18T04:26:45.0314111Z Ran 1 test in 0.002s
2022-05-18T04:26:45.0314283Z 
2022-05-18T04:26:45.0314394Z OK (skipped=1)
2022-05-18T04:26:45.0314574Z 
2022-05-18T04:26:45.0314702Z Generating XML reports...
2022-05-18T04:26:45.0357249Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042645.xml
2022-05-18T04:26:46.2837300Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:46.2853776Z 
2022-05-18T04:26:46.2854032Z Running tests...
2022-05-18T04:26:46.2854473Z ----------------------------------------------------------------------
2022-05-18T04:26:46.2878385Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:26:46.2878708Z 
2022-05-18T04:26:46.2879083Z ----------------------------------------------------------------------
2022-05-18T04:26:46.2879438Z Ran 1 test in 0.002s
2022-05-18T04:26:46.2879613Z 
2022-05-18T04:26:46.2879724Z OK (skipped=1)
2022-05-18T04:26:46.2879880Z 
2022-05-18T04:26:46.2879987Z Generating XML reports...
2022-05-18T04:26:46.2923673Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042646.xml
2022-05-18T04:26:47.5682135Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:47.5698712Z 
2022-05-18T04:26:47.5698970Z Running tests...
2022-05-18T04:26:47.5699389Z ----------------------------------------------------------------------
2022-05-18T04:26:49.2303488Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:49.2664426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5014
2022-05-18T04:26:49.2767785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5015
2022-05-18T04:26:50.4115969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:50.4299787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:50.4300582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:50.4318405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:50.4325041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:50.5315503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:52.6856593Z ok (5.115s)
2022-05-18T04:26:52.6856858Z 
2022-05-18T04:26:52.6857258Z ----------------------------------------------------------------------
2022-05-18T04:26:52.6857605Z Ran 1 test in 5.116s
2022-05-18T04:26:52.6857774Z 
2022-05-18T04:26:52.6857887Z OK
2022-05-18T04:26:52.6858073Z 
2022-05-18T04:26:52.6858193Z Generating XML reports...
2022-05-18T04:26:52.6914891Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042647.xml
2022-05-18T04:26:54.1005198Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:54.1020119Z 
2022-05-18T04:26:54.1020379Z Running tests...
2022-05-18T04:26:54.1020826Z ----------------------------------------------------------------------
2022-05-18T04:26:55.6942673Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:55.7294895Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5140
2022-05-18T04:26:55.7396680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5141
2022-05-18T04:26:56.8799254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:56.8869695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:56.8870497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:56.8900385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:56.8907074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:56.9885299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:59.1482401Z ok (5.046s)
2022-05-18T04:26:59.1482605Z 
2022-05-18T04:26:59.1483000Z ----------------------------------------------------------------------
2022-05-18T04:26:59.1483343Z Ran 1 test in 5.046s
2022-05-18T04:26:59.1483506Z 
2022-05-18T04:26:59.1483610Z OK
2022-05-18T04:26:59.1483746Z 
2022-05-18T04:26:59.1483876Z Generating XML reports...
2022-05-18T04:26:59.1540389Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042654.xml
2022-05-18T04:27:00.5765583Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:00.5780522Z 
2022-05-18T04:27:00.5780672Z Running tests...
2022-05-18T04:27:00.5781514Z ----------------------------------------------------------------------
2022-05-18T04:27:02.1953031Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:02.2309955Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5266
2022-05-18T04:27:02.2414068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5267
2022-05-18T04:27:03.3850469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:03.4285002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:03.4286523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:03.4355121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:03.4361829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:03.5301306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:05.6500330Z ok (5.072s)
2022-05-18T04:27:05.6500523Z 
2022-05-18T04:27:05.6500935Z ----------------------------------------------------------------------
2022-05-18T04:27:05.6501288Z Ran 1 test in 5.072s
2022-05-18T04:27:05.6501457Z 
2022-05-18T04:27:05.6501554Z OK
2022-05-18T04:27:05.6501692Z 
2022-05-18T04:27:05.6501838Z Generating XML reports...
2022-05-18T04:27:05.6569610Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042700.xml
2022-05-18T04:27:07.0711789Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:07.0726954Z 
2022-05-18T04:27:07.0727388Z Running tests...
2022-05-18T04:27:07.0727895Z ----------------------------------------------------------------------
2022-05-18T04:27:07.0746956Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:27:07.0747656Z 
2022-05-18T04:27:07.0747998Z ----------------------------------------------------------------------
2022-05-18T04:27:07.0748362Z Ran 1 test in 0.002s
2022-05-18T04:27:07.0748528Z 
2022-05-18T04:27:07.0748622Z OK (skipped=1)
2022-05-18T04:27:07.0748779Z 
2022-05-18T04:27:07.0748905Z Generating XML reports...
2022-05-18T04:27:07.0790639Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042707.xml
2022-05-18T04:27:08.3446270Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:08.3462304Z 
2022-05-18T04:27:08.3462607Z Running tests...
2022-05-18T04:27:08.3463073Z ----------------------------------------------------------------------
2022-05-18T04:27:08.3482745Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:27:08.3483070Z 
2022-05-18T04:27:08.3483588Z ----------------------------------------------------------------------
2022-05-18T04:27:08.3483960Z Ran 1 test in 0.002s
2022-05-18T04:27:08.3484127Z 
2022-05-18T04:27:08.3484239Z OK (skipped=1)
2022-05-18T04:27:08.3484396Z 
2022-05-18T04:27:08.3484528Z Generating XML reports...
2022-05-18T04:27:08.3526639Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042708.xml
2022-05-18T04:27:09.6178620Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:09.6193474Z 
2022-05-18T04:27:09.6193812Z Running tests...
2022-05-18T04:27:09.6194507Z ----------------------------------------------------------------------
2022-05-18T04:27:11.2859838Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:11.3220180Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5462
2022-05-18T04:27:11.3323520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5463
2022-05-18T04:27:12.5100186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:12.5362338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:12.5363130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:12.5403895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:12.5411073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:12.6377559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:15.1413442Z ok (5.522s)
2022-05-18T04:27:15.1413672Z 
2022-05-18T04:27:15.1414061Z ----------------------------------------------------------------------
2022-05-18T04:27:15.1414630Z Ran 1 test in 5.522s
2022-05-18T04:27:15.1414845Z 
2022-05-18T04:27:15.1414945Z OK
2022-05-18T04:27:15.1415084Z 
2022-05-18T04:27:15.1415222Z Generating XML reports...
2022-05-18T04:27:15.1471754Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042709.xml
2022-05-18T04:27:16.5696246Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:16.5710725Z 
2022-05-18T04:27:16.5710974Z Running tests...
2022-05-18T04:27:16.5711425Z ----------------------------------------------------------------------
2022-05-18T04:27:18.1850807Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:18.2205109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5585
2022-05-18T04:27:18.2307398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5586
2022-05-18T04:27:19.3461379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:19.3831537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:19.3833039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:19.3865855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:19.3872094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:19.4848282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:22.1401182Z ok (5.569s)
2022-05-18T04:27:22.1401398Z 
2022-05-18T04:27:22.1401823Z ----------------------------------------------------------------------
2022-05-18T04:27:22.1402164Z Ran 1 test in 5.569s
2022-05-18T04:27:22.1402309Z 
2022-05-18T04:27:22.1402405Z OK
2022-05-18T04:27:22.1402541Z 
2022-05-18T04:27:22.1402678Z Generating XML reports...
2022-05-18T04:27:22.1460500Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042716.xml
2022-05-18T04:27:23.5798456Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:23.5813755Z 
2022-05-18T04:27:23.5814118Z Running tests...
2022-05-18T04:27:23.5814559Z ----------------------------------------------------------------------
2022-05-18T04:27:23.5834336Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:27:23.5834952Z 
2022-05-18T04:27:23.5835244Z ----------------------------------------------------------------------
2022-05-18T04:27:23.5835573Z Ran 1 test in 0.002s
2022-05-18T04:27:23.5835716Z 
2022-05-18T04:27:23.5835835Z OK (skipped=1)
2022-05-18T04:27:23.5835990Z 
2022-05-18T04:27:23.5836115Z Generating XML reports...
2022-05-18T04:27:23.5877903Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042723.xml
2022-05-18T04:27:24.8292344Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:24.8308360Z 
2022-05-18T04:27:24.8308675Z Running tests...
2022-05-18T04:27:24.8309114Z ----------------------------------------------------------------------
2022-05-18T04:27:26.4936727Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:26.5299227Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5743
2022-05-18T04:27:26.5406113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5744
2022-05-18T04:27:27.6848505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:27.7427808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:27.7428621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:27.7455514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:27.7462568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:27.7465592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:27:27.8439274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:27.8442899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:27:27.8443659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:27:27.8483533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:27:30.5500920Z ok (5.719s)
2022-05-18T04:27:30.5501144Z 
2022-05-18T04:27:30.5501522Z ----------------------------------------------------------------------
2022-05-18T04:27:30.5501859Z Ran 1 test in 5.719s
2022-05-18T04:27:30.5502024Z 
2022-05-18T04:27:30.5502115Z OK
2022-05-18T04:27:30.5502248Z 
2022-05-18T04:27:30.5502634Z Generating XML reports...
2022-05-18T04:27:30.5560999Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042724.xml
2022-05-18T04:27:31.9799404Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:31.9814660Z 
2022-05-18T04:27:31.9814965Z Running tests...
2022-05-18T04:27:31.9815399Z ----------------------------------------------------------------------
2022-05-18T04:27:31.9834570Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:27:31.9835023Z 
2022-05-18T04:27:31.9835470Z ----------------------------------------------------------------------
2022-05-18T04:27:31.9835816Z Ran 1 test in 0.002s
2022-05-18T04:27:31.9835979Z 
2022-05-18T04:27:31.9836099Z OK (skipped=1)
2022-05-18T04:27:31.9836254Z 
2022-05-18T04:27:31.9836381Z Generating XML reports...
2022-05-18T04:27:31.9878766Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042731.xml
2022-05-18T04:27:33.2102419Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:33.2117184Z 
2022-05-18T04:27:33.2117635Z Running tests...
2022-05-18T04:27:33.2118139Z ----------------------------------------------------------------------
2022-05-18T04:27:34.8591653Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:34.8954920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5905
2022-05-18T04:27:34.9060513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5906
2022-05-18T04:27:36.0336570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:36.0386248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:36.0387075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:36.0439075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:36.0445941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:36.1400558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:36.3109747Z skip: Skipped due to small world size. (3.099s)
2022-05-18T04:27:36.3110196Z 
2022-05-18T04:27:36.3110947Z ----------------------------------------------------------------------
2022-05-18T04:27:36.3111602Z Ran 1 test in 3.099s
2022-05-18T04:27:36.3111802Z 
2022-05-18T04:27:36.3111911Z OK (skipped=1)
2022-05-18T04:27:36.3112066Z 
2022-05-18T04:27:36.3112191Z Generating XML reports...
2022-05-18T04:27:36.3168820Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042733.xml
2022-05-18T04:27:37.7331301Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:37.7346826Z 
2022-05-18T04:27:37.7347206Z Running tests...
2022-05-18T04:27:37.7348172Z ----------------------------------------------------------------------
2022-05-18T04:27:37.7372091Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:27:37.7372690Z 
2022-05-18T04:27:37.7373267Z ----------------------------------------------------------------------
2022-05-18T04:27:37.7373927Z Ran 1 test in 0.003s
2022-05-18T04:27:37.7374175Z 
2022-05-18T04:27:37.7374285Z OK (skipped=1)
2022-05-18T04:27:37.7374438Z 
2022-05-18T04:27:37.7374566Z Generating XML reports...
2022-05-18T04:27:37.7424002Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042737.xml
2022-05-18T04:27:38.9792964Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:38.9809983Z 
2022-05-18T04:27:38.9810449Z Running tests...
2022-05-18T04:27:38.9810945Z ----------------------------------------------------------------------
2022-05-18T04:27:38.9831818Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:27:38.9832152Z 
2022-05-18T04:27:38.9832887Z ----------------------------------------------------------------------
2022-05-18T04:27:38.9833632Z Ran 1 test in 0.002s
2022-05-18T04:27:38.9833981Z 
2022-05-18T04:27:38.9834223Z OK (skipped=1)
2022-05-18T04:27:38.9834523Z 
2022-05-18T04:27:38.9834711Z Generating XML reports...
2022-05-18T04:27:38.9877722Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042738.xml
2022-05-18T04:27:40.2125456Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:40.2140382Z 
2022-05-18T04:27:40.2140809Z Running tests...
2022-05-18T04:27:40.2141734Z ----------------------------------------------------------------------
2022-05-18T04:27:41.8894440Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:41.9253624Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6088
2022-05-18T04:27:41.9355668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6089
2022-05-18T04:27:43.1042279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:43.1283875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:43.1284671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:43.1345878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:43.1352633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:43.2299911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:47.1469444Z ok (6.933s)
2022-05-18T04:27:47.1469792Z 
2022-05-18T04:27:47.1470271Z ----------------------------------------------------------------------
2022-05-18T04:27:47.1470618Z Ran 1 test in 6.933s
2022-05-18T04:27:47.1470779Z 
2022-05-18T04:27:47.1470871Z OK
2022-05-18T04:27:47.1471012Z 
2022-05-18T04:27:47.1471143Z Generating XML reports...
2022-05-18T04:27:47.1528904Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042740.xml
2022-05-18T04:27:48.5998679Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:48.6015752Z 
2022-05-18T04:27:48.6016020Z Running tests...
2022-05-18T04:27:48.6016445Z ----------------------------------------------------------------------
2022-05-18T04:27:50.2551331Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:50.2907709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6216
2022-05-18T04:27:50.3013531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6217
2022-05-18T04:27:51.3871493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:51.4113821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:51.4114640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:51.4175257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:51.4181941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:51.5129854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:55.5152683Z ok (6.913s)
2022-05-18T04:27:55.5156586Z 
2022-05-18T04:27:55.5157313Z ----------------------------------------------------------------------
2022-05-18T04:27:55.5157691Z Ran 1 test in 6.914s
2022-05-18T04:27:55.5157860Z 
2022-05-18T04:27:55.5157965Z OK
2022-05-18T04:27:55.5158081Z 
2022-05-18T04:27:55.5158216Z Generating XML reports...
2022-05-18T04:27:55.5216345Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042748.xml
2022-05-18T04:27:56.9696152Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:56.9711368Z 
2022-05-18T04:27:56.9711773Z Running tests...
2022-05-18T04:27:56.9712273Z ----------------------------------------------------------------------
2022-05-18T04:27:56.9731865Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:27:56.9732784Z 
2022-05-18T04:27:56.9733152Z ----------------------------------------------------------------------
2022-05-18T04:27:56.9733481Z Ran 1 test in 0.002s
2022-05-18T04:27:56.9733649Z 
2022-05-18T04:27:56.9733756Z OK (skipped=1)
2022-05-18T04:27:56.9733895Z 
2022-05-18T04:27:56.9734021Z Generating XML reports...
2022-05-18T04:27:56.9775924Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042756.xml
2022-05-18T04:27:58.2474241Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:58.2489112Z 
2022-05-18T04:27:58.2489577Z Running tests...
2022-05-18T04:27:58.2490354Z ----------------------------------------------------------------------
2022-05-18T04:27:59.8733773Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:59.9087307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6379
2022-05-18T04:27:59.9195789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6380
2022-05-18T04:28:01.0608530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:01.0764254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:01.0765048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:01.0810618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:01.0817130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:01.0820349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:01.1777833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:01.1782144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:01.1783223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:28:01.1837775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:28:05.2316466Z ok (6.982s)
2022-05-18T04:28:05.2316689Z 
2022-05-18T04:28:05.2317077Z ----------------------------------------------------------------------
2022-05-18T04:28:05.2317401Z Ran 1 test in 6.983s
2022-05-18T04:28:05.2317573Z 
2022-05-18T04:28:05.2317919Z OK
2022-05-18T04:28:05.2318069Z 
2022-05-18T04:28:05.2318200Z Generating XML reports...
2022-05-18T04:28:05.2375771Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042758.xml
2022-05-18T04:28:06.6581988Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:06.6596350Z 
2022-05-18T04:28:06.6596602Z Running tests...
2022-05-18T04:28:06.6597030Z ----------------------------------------------------------------------
2022-05-18T04:28:06.6616620Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:28:06.6616954Z 
2022-05-18T04:28:06.6617242Z ----------------------------------------------------------------------
2022-05-18T04:28:06.6617576Z Ran 1 test in 0.002s
2022-05-18T04:28:06.6617739Z 
2022-05-18T04:28:06.6617848Z OK (skipped=1)
2022-05-18T04:28:06.6617988Z 
2022-05-18T04:28:06.6618130Z Generating XML reports...
2022-05-18T04:28:06.6659044Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042806.xml
2022-05-18T04:28:07.9354833Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:07.9370272Z 
2022-05-18T04:28:07.9370503Z Running tests...
2022-05-18T04:28:07.9370915Z ----------------------------------------------------------------------
2022-05-18T04:28:09.5788685Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:09.6148992Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6546
2022-05-18T04:28:09.6255992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6547
2022-05-18T04:28:10.7708923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:10.7771954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:10.7772746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:10.7810194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:10.7817105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:10.8787050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:11.0303669Z skip: Skipped due to small world size. (3.093s)
2022-05-18T04:28:11.0303904Z 
2022-05-18T04:28:11.0304286Z ----------------------------------------------------------------------
2022-05-18T04:28:11.0304618Z Ran 1 test in 3.093s
2022-05-18T04:28:11.0304780Z 
2022-05-18T04:28:11.0304871Z OK (skipped=1)
2022-05-18T04:28:11.0305025Z 
2022-05-18T04:28:11.0306481Z Generating XML reports...
2022-05-18T04:28:11.0364106Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042807.xml
2022-05-18T04:28:12.4686919Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:12.4703107Z 
2022-05-18T04:28:12.4703348Z Running tests...
2022-05-18T04:28:12.4703935Z ----------------------------------------------------------------------
2022-05-18T04:28:12.4724052Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:28:12.4724740Z 
2022-05-18T04:28:12.4725318Z ----------------------------------------------------------------------
2022-05-18T04:28:12.4725966Z Ran 1 test in 0.002s
2022-05-18T04:28:12.4726284Z 
2022-05-18T04:28:12.4726492Z OK (skipped=1)
2022-05-18T04:28:12.4726784Z 
2022-05-18T04:28:12.4727022Z Generating XML reports...
2022-05-18T04:28:12.4771507Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042812.xml
2022-05-18T04:28:13.7174264Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:13.7191236Z 
2022-05-18T04:28:13.7191708Z Running tests...
2022-05-18T04:28:13.7192191Z ----------------------------------------------------------------------
2022-05-18T04:28:13.7213333Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:28:13.7213676Z 
2022-05-18T04:28:13.7214454Z ----------------------------------------------------------------------
2022-05-18T04:28:13.7214825Z Ran 1 test in 0.002s
2022-05-18T04:28:13.7214994Z 
2022-05-18T04:28:13.7215089Z OK (skipped=1)
2022-05-18T04:28:13.7215252Z 
2022-05-18T04:28:13.7215382Z Generating XML reports...
2022-05-18T04:28:13.7258014Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042813.xml
2022-05-18T04:28:14.9980469Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:14.9995395Z 
2022-05-18T04:28:14.9996003Z Running tests...
2022-05-18T04:28:14.9996600Z ----------------------------------------------------------------------
2022-05-18T04:28:16.6707713Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:16.7072282Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6729
2022-05-18T04:28:16.7177807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6730
2022-05-18T04:28:17.8463598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:17.8606179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:17.8606985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:17.8667145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:17.8674951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:17.9622805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:20.6270048Z ok (5.627s)
2022-05-18T04:28:20.6270270Z 
2022-05-18T04:28:20.6270673Z ----------------------------------------------------------------------
2022-05-18T04:28:20.6271018Z Ran 1 test in 5.627s
2022-05-18T04:28:20.6271164Z 
2022-05-18T04:28:20.6271262Z OK
2022-05-18T04:28:20.6271397Z 
2022-05-18T04:28:20.6271533Z Generating XML reports...
2022-05-18T04:28:20.6327592Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042814.xml
2022-05-18T04:28:22.0434464Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:22.0449380Z 
2022-05-18T04:28:22.0449533Z Running tests...
2022-05-18T04:28:22.0450019Z ----------------------------------------------------------------------
2022-05-18T04:28:23.6540004Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:23.6896426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6852
2022-05-18T04:28:23.7002716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6853
2022-05-18T04:28:24.8437932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:24.8481806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:24.8482857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:24.8539735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:24.8545806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:24.9498270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:27.6096583Z ok (5.564s)
2022-05-18T04:28:27.6096822Z 
2022-05-18T04:28:27.6097205Z ----------------------------------------------------------------------
2022-05-18T04:28:27.6097564Z Ran 1 test in 5.565s
2022-05-18T04:28:27.6097735Z 
2022-05-18T04:28:27.6097832Z OK
2022-05-18T04:28:27.6097949Z 
2022-05-18T04:28:27.6098082Z Generating XML reports...
2022-05-18T04:28:27.6155326Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042822.xml
2022-05-18T04:28:29.0539491Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:29.0554976Z 
2022-05-18T04:28:29.0555325Z Running tests...
2022-05-18T04:28:29.0556042Z ----------------------------------------------------------------------
2022-05-18T04:28:29.0576278Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:28:29.0576629Z 
2022-05-18T04:28:29.0577054Z ----------------------------------------------------------------------
2022-05-18T04:28:29.0577506Z Ran 1 test in 0.002s
2022-05-18T04:28:29.0577675Z 
2022-05-18T04:28:29.0577786Z OK (skipped=1)
2022-05-18T04:28:29.0577941Z 
2022-05-18T04:28:29.0578068Z Generating XML reports...
2022-05-18T04:28:29.0620119Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042829.xml
2022-05-18T04:28:30.3362173Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:30.3377524Z 
2022-05-18T04:28:30.3377934Z Running tests...
2022-05-18T04:28:30.3378641Z ----------------------------------------------------------------------
2022-05-18T04:28:31.9794396Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:32.0160860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7010
2022-05-18T04:28:32.0268877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7011
2022-05-18T04:28:33.1712585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:33.1886578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:33.1887370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:33.1915129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:33.1921961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:33.1925338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:33.2899166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:33.2903196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:33.2903885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:28:33.2943638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:28:35.9366362Z ok (5.598s)
2022-05-18T04:28:35.9367062Z 
2022-05-18T04:28:35.9368020Z ----------------------------------------------------------------------
2022-05-18T04:28:35.9368650Z Ran 1 test in 5.599s
2022-05-18T04:28:35.9368938Z 
2022-05-18T04:28:35.9369107Z OK
2022-05-18T04:28:35.9369338Z 
2022-05-18T04:28:35.9369921Z Generating XML reports...
2022-05-18T04:28:35.9424912Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042830.xml
2022-05-18T04:28:37.3859140Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:37.3874540Z 
2022-05-18T04:28:37.3875040Z Running tests...
2022-05-18T04:28:37.3875521Z ----------------------------------------------------------------------
2022-05-18T04:28:37.3895427Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:28:37.3895786Z 
2022-05-18T04:28:37.3896083Z ----------------------------------------------------------------------
2022-05-18T04:28:37.3896430Z Ran 1 test in 0.002s
2022-05-18T04:28:37.3896594Z 
2022-05-18T04:28:37.3896701Z OK (skipped=1)
2022-05-18T04:28:37.3896854Z 
2022-05-18T04:28:37.3896983Z Generating XML reports...
2022-05-18T04:28:37.3939305Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042837.xml
2022-05-18T04:28:38.6671985Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:38.6686951Z 
2022-05-18T04:28:38.6687116Z Running tests...
2022-05-18T04:28:38.6687846Z ----------------------------------------------------------------------
2022-05-18T04:28:40.3074180Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:40.3440641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7172
2022-05-18T04:28:40.3545542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7173
2022-05-18T04:28:41.4965743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:41.5014649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:41.5015702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:41.5067096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:41.5073830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:41.6029991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:41.7593939Z skip: Skipped due to small world size. (3.090s)
2022-05-18T04:28:41.7594434Z 
2022-05-18T04:28:41.7595158Z ----------------------------------------------------------------------
2022-05-18T04:28:41.7595771Z Ran 1 test in 3.091s
2022-05-18T04:28:41.7595937Z 
2022-05-18T04:28:41.7596045Z OK (skipped=1)
2022-05-18T04:28:41.7596199Z 
2022-05-18T04:28:41.7596305Z Generating XML reports...
2022-05-18T04:28:41.7652507Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042838.xml
2022-05-18T04:28:43.1817986Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:43.1832949Z 
2022-05-18T04:28:43.1833258Z Running tests...
2022-05-18T04:28:43.1833691Z ----------------------------------------------------------------------
2022-05-18T04:28:44.8325713Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:44.8694435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7285
2022-05-18T04:28:44.8796716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7286
2022-05-18T04:28:46.0059170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:46.0287238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:46.0288042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:46.0363500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:46.0370168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:46.1303677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:48.3361788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:48.3362817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:48.3364314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:28:48.3365719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:28:48.6888631Z ok (5.505s)
2022-05-18T04:28:48.6892530Z 
2022-05-18T04:28:48.6893634Z ----------------------------------------------------------------------
2022-05-18T04:28:48.6893988Z Ran 1 test in 5.506s
2022-05-18T04:28:48.6894156Z 
2022-05-18T04:28:48.6894250Z OK
2022-05-18T04:28:48.6894386Z 
2022-05-18T04:28:48.6894503Z Generating XML reports...
2022-05-18T04:28:48.6953640Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042843.xml
2022-05-18T04:28:50.1067849Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:50.1082213Z 
2022-05-18T04:28:50.1082446Z Running tests...
2022-05-18T04:28:50.1082902Z ----------------------------------------------------------------------
2022-05-18T04:28:51.7203853Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:51.7558362Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7419
2022-05-18T04:28:51.7664598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7420
2022-05-18T04:28:52.8865955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:52.9062348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:52.9063145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:52.9068223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:52.9074759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:53.0077176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:53.1711760Z skip: Need at least 3 CUDA devices (3.063s)
2022-05-18T04:28:53.1712003Z 
2022-05-18T04:28:53.1712369Z ----------------------------------------------------------------------
2022-05-18T04:28:53.1712685Z Ran 1 test in 3.063s
2022-05-18T04:28:53.1712856Z 
2022-05-18T04:28:53.1712969Z OK (skipped=1)
2022-05-18T04:28:53.1713124Z 
2022-05-18T04:28:53.1713251Z Generating XML reports...
2022-05-18T04:28:53.1768943Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042850.xml
2022-05-18T04:28:54.5911825Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:54.5926673Z 
2022-05-18T04:28:54.5927063Z Running tests...
2022-05-18T04:28:54.5927821Z ----------------------------------------------------------------------
2022-05-18T04:28:54.5948351Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-05-18T04:28:54.5948682Z 
2022-05-18T04:28:54.5948976Z ----------------------------------------------------------------------
2022-05-18T04:28:54.5949308Z Ran 1 test in 0.002s
2022-05-18T04:28:54.5949468Z 
2022-05-18T04:28:54.5949576Z OK (skipped=1)
2022-05-18T04:28:54.5949714Z 
2022-05-18T04:28:54.5949836Z Generating XML reports...
2022-05-18T04:28:54.5991241Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042854.xml
2022-05-18T04:28:55.8540959Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:55.8555248Z 
2022-05-18T04:28:55.8555735Z Running tests...
2022-05-18T04:28:55.8556385Z ----------------------------------------------------------------------
2022-05-18T04:28:55.8576131Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-05-18T04:28:55.8576432Z 
2022-05-18T04:28:55.8576844Z ----------------------------------------------------------------------
2022-05-18T04:28:55.8577573Z Ran 1 test in 0.002s
2022-05-18T04:28:55.8577742Z 
2022-05-18T04:28:55.8577834Z OK (skipped=1)
2022-05-18T04:28:55.8577985Z 
2022-05-18T04:28:55.8578107Z Generating XML reports...
2022-05-18T04:28:55.8618455Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042855.xml
2022-05-18T04:28:57.1331705Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:57.1346928Z 
2022-05-18T04:28:57.1347316Z Running tests...
2022-05-18T04:28:57.1347790Z ----------------------------------------------------------------------
2022-05-18T04:28:58.7680245Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:58.8038015Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7602
2022-05-18T04:28:58.8140699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7603
2022-05-18T04:28:59.9672637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:59.9823637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:59.9824433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:59.9874956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:59.9881503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:00.0838510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:03.4245309Z ok (6.290s)
2022-05-18T04:29:03.4245546Z 
2022-05-18T04:29:03.4245934Z ----------------------------------------------------------------------
2022-05-18T04:29:03.4246267Z Ran 1 test in 6.290s
2022-05-18T04:29:03.4246442Z 
2022-05-18T04:29:03.4246535Z OK
2022-05-18T04:29:03.4246651Z 
2022-05-18T04:29:03.4246785Z Generating XML reports...
2022-05-18T04:29:03.4304560Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042857.xml
2022-05-18T04:29:04.8452242Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:04.8467161Z 
2022-05-18T04:29:04.8467350Z Running tests...
2022-05-18T04:29:04.8468018Z ----------------------------------------------------------------------
2022-05-18T04:29:04.8487403Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-05-18T04:29:04.8487725Z 
2022-05-18T04:29:04.8488367Z ----------------------------------------------------------------------
2022-05-18T04:29:04.8488838Z Ran 1 test in 0.002s
2022-05-18T04:29:04.8488999Z 
2022-05-18T04:29:04.8489106Z OK (skipped=1)
2022-05-18T04:29:04.8489269Z 
2022-05-18T04:29:04.8489391Z Generating XML reports...
2022-05-18T04:29:04.8529482Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042904.xml
2022-05-18T04:29:06.1173958Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:06.1189016Z 
2022-05-18T04:29:06.1189495Z Running tests...
2022-05-18T04:29:06.1190011Z ----------------------------------------------------------------------
2022-05-18T04:29:07.7493881Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:07.7859699Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7760
2022-05-18T04:29:07.7963050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7761
2022-05-18T04:29:08.9476159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:08.9544191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:08.9545178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:08.9577951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:08.9584464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:09.0558496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:09.2014569Z skip: Skipped due to small world size. (3.082s)
2022-05-18T04:29:09.2015025Z 
2022-05-18T04:29:09.2015451Z ----------------------------------------------------------------------
2022-05-18T04:29:09.2015792Z Ran 1 test in 3.082s
2022-05-18T04:29:09.2015937Z 
2022-05-18T04:29:09.2016044Z OK (skipped=1)
2022-05-18T04:29:09.2016197Z 
2022-05-18T04:29:09.2016331Z Generating XML reports...
2022-05-18T04:29:09.2072150Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042906.xml
2022-05-18T04:29:10.5853552Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:10.5868804Z 
2022-05-18T04:29:10.5869234Z Running tests...
2022-05-18T04:29:10.5869715Z ----------------------------------------------------------------------
2022-05-18T04:29:10.5890504Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-05-18T04:29:10.5890818Z 
2022-05-18T04:29:10.5891104Z ----------------------------------------------------------------------
2022-05-18T04:29:10.5891431Z Ran 1 test in 0.002s
2022-05-18T04:29:10.5891592Z 
2022-05-18T04:29:10.5891705Z OK (skipped=1)
2022-05-18T04:29:10.5891861Z 
2022-05-18T04:29:10.5891985Z Generating XML reports...
2022-05-18T04:29:10.5935549Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042910.xml
2022-05-18T04:29:11.8200298Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:11.8215917Z 
2022-05-18T04:29:11.8216185Z Running tests...
2022-05-18T04:29:11.8216624Z ----------------------------------------------------------------------
2022-05-18T04:29:13.4761760Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:13.5127089Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7908
2022-05-18T04:29:13.5231849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7909
2022-05-18T04:29:14.6789214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:14.7349113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:14.7349919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:14.7396274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:14.7402686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:14.8363925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:15.0280226Z skip: Skipped due to small world size. (3.206s)
2022-05-18T04:29:15.0280508Z 
2022-05-18T04:29:15.0280876Z ----------------------------------------------------------------------
2022-05-18T04:29:15.0281202Z Ran 1 test in 3.206s
2022-05-18T04:29:15.0281368Z 
2022-05-18T04:29:15.0281506Z OK (skipped=1)
2022-05-18T04:29:15.0281673Z 
2022-05-18T04:29:15.0281798Z Generating XML reports...
2022-05-18T04:29:15.0339273Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042911.xml
2022-05-18T04:29:16.4313191Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:16.4328051Z 
2022-05-18T04:29:16.4328306Z Running tests...
2022-05-18T04:29:16.4328746Z ----------------------------------------------------------------------
2022-05-18T04:29:16.4350151Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-05-18T04:29:16.4350498Z 
2022-05-18T04:29:16.4350823Z ----------------------------------------------------------------------
2022-05-18T04:29:16.4351154Z Ran 1 test in 0.002s
2022-05-18T04:29:16.4351324Z 
2022-05-18T04:29:16.4351436Z OK (skipped=1)
2022-05-18T04:29:16.4351574Z 
2022-05-18T04:29:16.4351716Z Generating XML reports...
2022-05-18T04:29:16.4393238Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042916.xml
2022-05-18T04:29:17.7123329Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:17.7138882Z 
2022-05-18T04:29:17.7139290Z Running tests...
2022-05-18T04:29:17.7139774Z ----------------------------------------------------------------------
2022-05-18T04:29:17.7164251Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-05-18T04:29:17.7164638Z 
2022-05-18T04:29:17.7164986Z ----------------------------------------------------------------------
2022-05-18T04:29:17.7165347Z Ran 1 test in 0.003s
2022-05-18T04:29:17.7165493Z 
2022-05-18T04:29:17.7165605Z OK (skipped=1)
2022-05-18T04:29:17.7165763Z 
2022-05-18T04:29:17.7165899Z Generating XML reports...
2022-05-18T04:29:17.7209235Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042917.xml
2022-05-18T04:29:18.9880523Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:18.9896045Z 
2022-05-18T04:29:18.9896265Z Running tests...
2022-05-18T04:29:18.9896682Z ----------------------------------------------------------------------
2022-05-18T04:29:18.9917642Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-05-18T04:29:18.9917967Z 
2022-05-18T04:29:18.9918252Z ----------------------------------------------------------------------
2022-05-18T04:29:18.9918584Z Ran 1 test in 0.002s
2022-05-18T04:29:18.9918744Z 
2022-05-18T04:29:18.9918836Z OK (skipped=1)
2022-05-18T04:29:18.9918988Z 
2022-05-18T04:29:18.9919112Z Generating XML reports...
2022-05-18T04:29:18.9961818Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042918.xml
2022-05-18T04:29:20.2606841Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:20.2622147Z 
2022-05-18T04:29:20.2622515Z Running tests...
2022-05-18T04:29:20.2623200Z ----------------------------------------------------------------------
2022-05-18T04:29:20.2649680Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.003s)
2022-05-18T04:29:20.2650282Z 
2022-05-18T04:29:20.2650818Z ----------------------------------------------------------------------
2022-05-18T04:29:20.2651175Z Ran 1 test in 0.003s
2022-05-18T04:29:20.2651341Z 
2022-05-18T04:29:20.2651451Z OK (skipped=1)
2022-05-18T04:29:20.2651586Z 
2022-05-18T04:29:20.2651725Z Generating XML reports...
2022-05-18T04:29:20.2694042Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042920.xml
2022-05-18T04:29:21.4971138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:21.4987930Z 
2022-05-18T04:29:21.4988412Z Running tests...
2022-05-18T04:29:21.4988948Z ----------------------------------------------------------------------
2022-05-18T04:29:21.5016251Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.003s)
2022-05-18T04:29:21.5016559Z 
2022-05-18T04:29:21.5017077Z ----------------------------------------------------------------------
2022-05-18T04:29:21.5017473Z Ran 1 test in 0.003s
2022-05-18T04:29:21.5017645Z 
2022-05-18T04:29:21.5017754Z OK (skipped=1)
2022-05-18T04:29:21.5017894Z 
2022-05-18T04:29:21.5018020Z Generating XML reports...
2022-05-18T04:29:21.5061161Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042921.xml
2022-05-18T04:29:22.7813433Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:22.7828635Z 
2022-05-18T04:29:22.7828919Z Running tests...
2022-05-18T04:29:22.7829574Z ----------------------------------------------------------------------
2022-05-18T04:29:24.4164831Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:24.4523277Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8196
2022-05-18T04:29:24.4625792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8197
2022-05-18T04:29:25.6071275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:25.6291295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:25.6292325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:25.6375147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:25.6381894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:25.7302450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:25.7611313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:25.7612069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:25.7612762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:29:25.7614010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:29:25.7616438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:29:25.7719532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:29:25.7720248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:29:25.7821270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:29:26.0678336Z ok (3.285s)
2022-05-18T04:29:26.0678558Z 
2022-05-18T04:29:26.0678968Z ----------------------------------------------------------------------
2022-05-18T04:29:26.0679306Z Ran 1 test in 3.285s
2022-05-18T04:29:26.0679471Z 
2022-05-18T04:29:26.0679566Z OK
2022-05-18T04:29:26.0679700Z 
2022-05-18T04:29:26.0679814Z Generating XML reports...
2022-05-18T04:29:26.0736402Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042922.xml
2022-05-18T04:29:27.5155097Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:27.5170202Z 
2022-05-18T04:29:27.5170597Z Running tests...
2022-05-18T04:29:27.5171095Z ----------------------------------------------------------------------
2022-05-18T04:29:29.1547195Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:29.1915120Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8319
2022-05-18T04:29:29.2019134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8320
2022-05-18T04:29:30.3455118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:30.3532880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:30.3533991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:30.3556306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:30.3563116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:30.4543915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:32.1093586Z ok (4.592s)
2022-05-18T04:29:32.1093807Z 
2022-05-18T04:29:32.1094402Z ----------------------------------------------------------------------
2022-05-18T04:29:32.1094788Z Ran 1 test in 4.592s
2022-05-18T04:29:32.1094951Z 
2022-05-18T04:29:32.1095045Z OK
2022-05-18T04:29:32.1095187Z 
2022-05-18T04:29:32.1095331Z Generating XML reports...
2022-05-18T04:29:32.1152229Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042927.xml
2022-05-18T04:29:33.5308746Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:33.5322884Z 
2022-05-18T04:29:33.5323185Z Running tests...
2022-05-18T04:29:33.5323628Z ----------------------------------------------------------------------
2022-05-18T04:29:35.1448124Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:35.1805409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8441
2022-05-18T04:29:35.1912707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8442
2022-05-18T04:29:36.3648471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:36.3938384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:36.3939181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:36.3952639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:36.3958832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:36.4953223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:36.6961825Z skip: Skipped due to small world size. (3.164s)
2022-05-18T04:29:36.6962193Z 
2022-05-18T04:29:36.6962712Z ----------------------------------------------------------------------
2022-05-18T04:29:36.6963035Z Ran 1 test in 3.164s
2022-05-18T04:29:36.6963198Z 
2022-05-18T04:29:36.6963307Z OK (skipped=1)
2022-05-18T04:29:36.6963547Z 
2022-05-18T04:29:36.6963790Z Generating XML reports...
2022-05-18T04:29:36.7022045Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042933.xml
2022-05-18T04:29:38.1191570Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:38.1206977Z 
2022-05-18T04:29:38.1207407Z Running tests...
2022-05-18T04:29:38.1207845Z ----------------------------------------------------------------------
2022-05-18T04:29:39.7591116Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:39.7959200Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8554
2022-05-18T04:29:39.8063131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8555
2022-05-18T04:29:40.9810457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:41.0025169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:41.0025982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:41.0113536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:41.0120044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:41.1036483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:42.6135248Z ok (4.492s)
2022-05-18T04:29:42.6135474Z 
2022-05-18T04:29:42.6135874Z ----------------------------------------------------------------------
2022-05-18T04:29:42.6136219Z Ran 1 test in 4.493s
2022-05-18T04:29:42.6136388Z 
2022-05-18T04:29:42.6136464Z OK
2022-05-18T04:29:42.6136602Z 
2022-05-18T04:29:42.6136736Z Generating XML reports...
2022-05-18T04:29:42.6194698Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042938.xml
2022-05-18T04:29:44.0645204Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:44.0660993Z 
2022-05-18T04:29:44.0661305Z Running tests...
2022-05-18T04:29:44.0661741Z ----------------------------------------------------------------------
2022-05-18T04:29:45.7157465Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:45.7525019Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8668
2022-05-18T04:29:45.7633369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8669
2022-05-18T04:29:46.9336369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:46.9343075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:46.9343862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:46.9437661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:46.9444726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:47.0354357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:47.2682507Z ok (3.202s)
2022-05-18T04:29:47.2682716Z 
2022-05-18T04:29:47.2683099Z ----------------------------------------------------------------------
2022-05-18T04:29:47.2683433Z Ran 1 test in 3.202s
2022-05-18T04:29:47.2683600Z 
2022-05-18T04:29:47.2683695Z OK
2022-05-18T04:29:47.2683830Z 
2022-05-18T04:29:47.2683941Z Generating XML reports...
2022-05-18T04:29:47.2741558Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042944.xml
2022-05-18T04:29:48.7055879Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:48.7070934Z 
2022-05-18T04:29:48.7071057Z Running tests...
2022-05-18T04:29:48.7071759Z ----------------------------------------------------------------------
2022-05-18T04:29:50.3311469Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:50.3673685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8781
2022-05-18T04:29:50.3781317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8782
2022-05-18T04:29:51.5559321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:51.5749468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:51.5750558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:51.5761689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:51.5768495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:51.6760665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:53.3857054Z ok (4.678s)
2022-05-18T04:29:53.3857426Z 
2022-05-18T04:29:53.3857839Z ----------------------------------------------------------------------
2022-05-18T04:29:53.3858202Z Ran 1 test in 4.679s
2022-05-18T04:29:53.3858374Z 
2022-05-18T04:29:53.3858513Z OK
2022-05-18T04:29:53.3858753Z 
2022-05-18T04:29:53.3858958Z Generating XML reports...
2022-05-18T04:29:53.3916140Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042948.xml
2022-05-18T04:29:54.8150341Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:54.8164771Z 
2022-05-18T04:29:54.8165200Z Running tests...
2022-05-18T04:29:54.8165676Z ----------------------------------------------------------------------
2022-05-18T04:29:56.4255397Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:56.4618042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8903
2022-05-18T04:29:56.4724393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8904
2022-05-18T04:29:57.6331899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:57.6568400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:57.6569397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:57.6635087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:57.6642429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:57.7579924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:59.4800297Z ok (4.663s)
2022-05-18T04:29:59.4800528Z 
2022-05-18T04:29:59.4800924Z ----------------------------------------------------------------------
2022-05-18T04:29:59.4801264Z Ran 1 test in 4.664s
2022-05-18T04:29:59.4801434Z 
2022-05-18T04:29:59.4801530Z OK
2022-05-18T04:29:59.4801667Z 
2022-05-18T04:29:59.4801803Z Generating XML reports...
2022-05-18T04:29:59.4858749Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042954.xml
2022-05-18T04:30:00.9284310Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:00.9300299Z 
2022-05-18T04:30:00.9300450Z Running tests...
2022-05-18T04:30:00.9301163Z ----------------------------------------------------------------------
2022-05-18T04:30:02.6120144Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:02.6486360Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9025
2022-05-18T04:30:02.6595674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9026
2022-05-18T04:30:03.7547865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:03.8062067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:03.8062865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:03.8154916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:03.8162159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:03.9072873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:04.1647279Z ok (3.234s)
2022-05-18T04:30:04.1647489Z 
2022-05-18T04:30:04.1647854Z ----------------------------------------------------------------------
2022-05-18T04:30:04.1648175Z Ran 1 test in 3.235s
2022-05-18T04:30:04.1648345Z 
2022-05-18T04:30:04.1648447Z OK
2022-05-18T04:30:04.1648587Z 
2022-05-18T04:30:04.1648722Z Generating XML reports...
2022-05-18T04:30:04.1705728Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043000.xml
2022-05-18T04:30:05.6042356Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:05.6058560Z 
2022-05-18T04:30:05.6059034Z Running tests...
2022-05-18T04:30:05.6059538Z ----------------------------------------------------------------------
2022-05-18T04:30:05.6079794Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:30:05.6080134Z 
2022-05-18T04:30:05.6080484Z ----------------------------------------------------------------------
2022-05-18T04:30:05.6080994Z Ran 1 test in 0.002s
2022-05-18T04:30:05.6081294Z 
2022-05-18T04:30:05.6081417Z OK (skipped=1)
2022-05-18T04:30:05.6081574Z 
2022-05-18T04:30:05.6081711Z Generating XML reports...
2022-05-18T04:30:05.6124019Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043005.xml
2022-05-18T04:30:06.8885629Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:06.8902306Z 
2022-05-18T04:30:06.8902608Z Running tests...
2022-05-18T04:30:06.8903332Z ----------------------------------------------------------------------
2022-05-18T04:30:08.5481021Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:08.5849043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9173
2022-05-18T04:30:08.5955045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9174
2022-05-18T04:30:09.7287880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:09.7372524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:09.7373540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:09.7389096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:09.7395552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:09.8387890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:12.1041081Z ok (5.214s)
2022-05-18T04:30:12.1041319Z 
2022-05-18T04:30:12.1041711Z ----------------------------------------------------------------------
2022-05-18T04:30:12.1042058Z Ran 1 test in 5.214s
2022-05-18T04:30:12.1042230Z 
2022-05-18T04:30:12.1042326Z OK
2022-05-18T04:30:12.1042462Z 
2022-05-18T04:30:12.1042596Z Generating XML reports...
2022-05-18T04:30:12.1099702Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043006.xml
2022-05-18T04:30:13.5478268Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:13.5494341Z 
2022-05-18T04:30:13.5494527Z Running tests...
2022-05-18T04:30:13.5494961Z ----------------------------------------------------------------------
2022-05-18T04:30:13.5515358Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:30:13.5515824Z 
2022-05-18T04:30:13.5516111Z ----------------------------------------------------------------------
2022-05-18T04:30:13.5516429Z Ran 1 test in 0.002s
2022-05-18T04:30:13.5516595Z 
2022-05-18T04:30:13.5516706Z OK (skipped=1)
2022-05-18T04:30:13.5516882Z 
2022-05-18T04:30:13.5517010Z Generating XML reports...
2022-05-18T04:30:13.5559877Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043013.xml
2022-05-18T04:30:14.8287925Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:14.8304481Z 
2022-05-18T04:30:14.8304689Z Running tests...
2022-05-18T04:30:14.8305117Z ----------------------------------------------------------------------
2022-05-18T04:30:14.8325768Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:30:14.8326055Z 
2022-05-18T04:30:14.8326339Z ----------------------------------------------------------------------
2022-05-18T04:30:14.8326665Z Ran 1 test in 0.002s
2022-05-18T04:30:14.8326835Z 
2022-05-18T04:30:14.8326945Z OK (skipped=1)
2022-05-18T04:30:14.8327099Z 
2022-05-18T04:30:14.8327207Z Generating XML reports...
2022-05-18T04:30:14.8370597Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043014.xml
2022-05-18T04:30:16.0856435Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:16.0870874Z 
2022-05-18T04:30:16.0871317Z Running tests...
2022-05-18T04:30:16.0871792Z ----------------------------------------------------------------------
2022-05-18T04:30:16.0893049Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-05-18T04:30:16.0893376Z 
2022-05-18T04:30:16.0893662Z ----------------------------------------------------------------------
2022-05-18T04:30:16.0893993Z Ran 1 test in 0.002s
2022-05-18T04:30:16.0894136Z 
2022-05-18T04:30:16.0894244Z OK (skipped=1)
2022-05-18T04:30:16.0894400Z 
2022-05-18T04:30:16.0894525Z Generating XML reports...
2022-05-18T04:30:16.0934500Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043016.xml
2022-05-18T04:30:17.3610603Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:17.3625972Z 
2022-05-18T04:30:17.3626281Z Running tests...
2022-05-18T04:30:17.3626977Z ----------------------------------------------------------------------
2022-05-18T04:30:19.0221194Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:19.0585350Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9404
2022-05-18T04:30:19.0690873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9405
2022-05-18T04:30:20.2098103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:20.2559778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:20.2560621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:20.2604195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:20.2611134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:20.3575242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:22.9795600Z ok (5.617s)
2022-05-18T04:30:22.9795831Z 
2022-05-18T04:30:22.9796231Z ----------------------------------------------------------------------
2022-05-18T04:30:22.9796556Z Ran 1 test in 5.617s
2022-05-18T04:30:22.9796729Z 
2022-05-18T04:30:22.9796825Z OK
2022-05-18T04:30:22.9796964Z 
2022-05-18T04:30:22.9797093Z Generating XML reports...
2022-05-18T04:30:22.9854846Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043017.xml
2022-05-18T04:30:24.4068658Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:24.4083784Z 
2022-05-18T04:30:24.4084048Z Running tests...
2022-05-18T04:30:24.4084493Z ----------------------------------------------------------------------
2022-05-18T04:30:26.0417174Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:26.0778774Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9527
2022-05-18T04:30:26.0882025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9528
2022-05-18T04:30:27.2624018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:27.2887951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:27.2888774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:27.2928239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:27.2935321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:27.3899962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:27.4055073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:30:27.4055590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:30:27.4056281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:30:27.4056950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:30:27.4059403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:30:27.4160155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:30:27.4161215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:30:27.4161962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:30:28.7434851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3ovp_jv
2022-05-18T04:30:28.7436180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3ovp_jv/_remote_module_non_scriptable.py
2022-05-18T04:30:28.7586793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi3e032b5
2022-05-18T04:30:28.7589568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi3e032b5/_remote_module_non_scriptable.py
2022-05-18T04:30:29.1960451Z ok (4.787s)
2022-05-18T04:30:29.1960637Z 
2022-05-18T04:30:29.1961014Z ----------------------------------------------------------------------
2022-05-18T04:30:29.1961625Z Ran 1 test in 4.788s
2022-05-18T04:30:29.1961790Z 
2022-05-18T04:30:29.1961883Z OK
2022-05-18T04:30:29.1962016Z 
2022-05-18T04:30:29.1962143Z Generating XML reports...
2022-05-18T04:30:29.2019333Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043024.xml
2022-05-18T04:30:30.6347344Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:30.6362856Z 
2022-05-18T04:30:30.6363134Z Running tests...
2022-05-18T04:30:30.6363567Z ----------------------------------------------------------------------
2022-05-18T04:30:32.2831118Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:32.3211778Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9657
2022-05-18T04:30:32.3316102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9658
2022-05-18T04:30:33.4534257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:33.4796442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:33.4797243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:33.4837880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:33.4844446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:33.5808875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:33.5917947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:30:33.5918466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:30:33.5919402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:30:33.5920365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:30:33.5921344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:30:33.5921835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:30:33.5922484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:30:33.5923580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:30:34.9346057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1br3_1l
2022-05-18T04:30:34.9346682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1br3_1l/_remote_module_non_scriptable.py
2022-05-18T04:30:34.9390610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsu6fueb
2022-05-18T04:30:34.9393645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsu6fueb/_remote_module_non_scriptable.py
2022-05-18T04:30:35.3393469Z ok (4.703s)
2022-05-18T04:30:35.3393685Z 
2022-05-18T04:30:35.3394675Z ----------------------------------------------------------------------
2022-05-18T04:30:35.3395323Z Ran 1 test in 4.703s
2022-05-18T04:30:35.3395498Z 
2022-05-18T04:30:35.3395594Z OK
2022-05-18T04:30:35.3395742Z 
2022-05-18T04:30:35.3395882Z Generating XML reports...
2022-05-18T04:30:35.3452348Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043030.xml
2022-05-18T04:30:36.7930055Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:36.7947520Z 
2022-05-18T04:30:36.7947963Z Running tests...
2022-05-18T04:30:36.7948444Z ----------------------------------------------------------------------
2022-05-18T04:30:38.4475879Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:38.4864825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9787
2022-05-18T04:30:38.4968940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9788
2022-05-18T04:30:39.6759670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:39.7038644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:39.7039463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:39.7063385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:39.7070413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:39.8053850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:41.0393380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi4s91cbl
2022-05-18T04:30:41.0394004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi4s91cbl/_remote_module_non_scriptable.py
2022-05-18T04:30:41.1158075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewqhmbvu
2022-05-18T04:30:41.1159235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewqhmbvu/_remote_module_non_scriptable.py
2022-05-18T04:30:41.4728711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:41.4729268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:41.8052251Z ok (5.010s)
2022-05-18T04:30:41.8052631Z 
2022-05-18T04:30:41.8053281Z ----------------------------------------------------------------------
2022-05-18T04:30:41.8053612Z Ran 1 test in 5.010s
2022-05-18T04:30:41.8053781Z 
2022-05-18T04:30:41.8053872Z OK
2022-05-18T04:30:41.8054007Z 
2022-05-18T04:30:41.8054132Z Generating XML reports...
2022-05-18T04:30:41.8111260Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043036.xml
2022-05-18T04:30:43.2110127Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:43.2126512Z 
2022-05-18T04:30:43.2127106Z Running tests...
2022-05-18T04:30:43.2127877Z ----------------------------------------------------------------------
2022-05-18T04:30:44.8738955Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:44.9096978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9913
2022-05-18T04:30:44.9203301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9914
2022-05-18T04:30:46.0491190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:46.0648051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:46.0649890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:46.0693354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:46.0699826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:46.1665893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:47.3608408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuh9im2u4
2022-05-18T04:30:47.3610200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuh9im2u4/_remote_module_non_scriptable.py
2022-05-18T04:30:47.4686294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7xb_85i
2022-05-18T04:30:47.4687422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7xb_85i/_remote_module_non_scriptable.py
2022-05-18T04:30:47.8331821Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:47.8332862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:47.8338711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:47.8343850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:48.1285005Z ok (4.916s)
2022-05-18T04:30:48.1285188Z 
2022-05-18T04:30:48.1285730Z ----------------------------------------------------------------------
2022-05-18T04:30:48.1286043Z Ran 1 test in 4.916s
2022-05-18T04:30:48.1286206Z 
2022-05-18T04:30:48.1286303Z OK
2022-05-18T04:30:48.1286435Z 
2022-05-18T04:30:48.1286563Z Generating XML reports...
2022-05-18T04:30:48.1343724Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043043.xml
2022-05-18T04:30:49.5644774Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:49.5660126Z 
2022-05-18T04:30:49.5660601Z Running tests...
2022-05-18T04:30:49.5661282Z ----------------------------------------------------------------------
2022-05-18T04:30:51.2236614Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:51.2599159Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10039
2022-05-18T04:30:51.2707238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10040
2022-05-18T04:30:52.4459332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:52.4682272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:52.4683097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:52.4762849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:52.4769696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:52.5697952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:53.7946520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_i41uuwy
2022-05-18T04:30:53.7947165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_i41uuwy/_remote_module_non_scriptable.py
2022-05-18T04:30:53.8643662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wi2ucb3
2022-05-18T04:30:53.8644478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wi2ucb3/_remote_module_non_scriptable.py
2022-05-18T04:30:54.2388716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:54.2389251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:54.2397049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:54.2397561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:54.2526088Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:54.2526654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:54.2535142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:54.2535713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:54.5802814Z ok (5.014s)
2022-05-18T04:30:54.5803126Z 
2022-05-18T04:30:54.5825666Z ----------------------------------------------------------------------
2022-05-18T04:30:54.5826189Z Ran 1 test in 5.014s
2022-05-18T04:30:54.5826410Z 
2022-05-18T04:30:54.5826530Z OK
2022-05-18T04:30:54.5826679Z 
2022-05-18T04:30:54.5826912Z Generating XML reports...
2022-05-18T04:30:54.5860877Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043049.xml
2022-05-18T04:30:56.0291421Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:56.0306554Z 
2022-05-18T04:30:56.0307079Z Running tests...
2022-05-18T04:30:56.0307571Z ----------------------------------------------------------------------
2022-05-18T04:30:57.6809446Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:57.6929815Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.662s)
2022-05-18T04:30:57.6930523Z 
2022-05-18T04:30:57.6930794Z ----------------------------------------------------------------------
2022-05-18T04:30:57.6931126Z Ran 1 test in 1.662s
2022-05-18T04:30:57.6931290Z 
2022-05-18T04:30:57.6931399Z OK (skipped=1)
2022-05-18T04:30:57.6931574Z 
2022-05-18T04:30:57.6931707Z Generating XML reports...
2022-05-18T04:30:57.6970029Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043056.xml
2022-05-18T04:30:59.0890407Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:59.0905934Z 
2022-05-18T04:30:59.0906185Z Running tests...
2022-05-18T04:30:59.0906594Z ----------------------------------------------------------------------
2022-05-18T04:31:00.7352886Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:00.7717980Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10201
2022-05-18T04:31:00.7823244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10202
2022-05-18T04:31:01.9441791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:01.9657200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:01.9658038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:01.9745286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:01.9752602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:02.0671517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:03.3077481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5wzeeko
2022-05-18T04:31:03.3078098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5wzeeko/_remote_module_non_scriptable.py
2022-05-18T04:31:03.3442915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprrzyxvf
2022-05-18T04:31:03.3445010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprrzyxvf/_remote_module_non_scriptable.py
2022-05-18T04:31:04.3849863Z <class 'torch.nn.parameter.Parameter'>
2022-05-18T04:31:04.6917113Z ok (5.601s)
2022-05-18T04:31:04.6917325Z 
2022-05-18T04:31:04.6917725Z ----------------------------------------------------------------------
2022-05-18T04:31:04.6918042Z Ran 1 test in 5.601s
2022-05-18T04:31:04.6918210Z 
2022-05-18T04:31:04.6918308Z OK
2022-05-18T04:31:04.6918448Z 
2022-05-18T04:31:04.6918586Z Generating XML reports...
2022-05-18T04:31:04.6976103Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043059.xml
2022-05-18T04:31:06.1473476Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:06.1488645Z 
2022-05-18T04:31:06.1489264Z Running tests...
2022-05-18T04:31:06.1490170Z ----------------------------------------------------------------------
2022-05-18T04:31:07.8014880Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:07.8381467Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10324
2022-05-18T04:31:07.8487335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10325
2022-05-18T04:31:08.9739751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:08.9945711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:08.9946799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:09.0043799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:09.0050310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:09.0961516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:10.2859261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi327o3s7
2022-05-18T04:31:10.2859890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi327o3s7/_remote_module_non_scriptable.py
2022-05-18T04:31:10.4185721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzwxbwxqv
2022-05-18T04:31:10.4187064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzwxbwxqv/_remote_module_non_scriptable.py
2022-05-18T04:31:11.6580189Z ok (5.509s)
2022-05-18T04:31:11.6580499Z 
2022-05-18T04:31:11.6581160Z ----------------------------------------------------------------------
2022-05-18T04:31:11.6581597Z Ran 1 test in 5.509s
2022-05-18T04:31:11.6581764Z 
2022-05-18T04:31:11.6581866Z OK
2022-05-18T04:31:11.6582002Z 
2022-05-18T04:31:11.6582369Z Generating XML reports...
2022-05-18T04:31:11.6638041Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043106.xml
2022-05-18T04:31:13.0958550Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:13.0974793Z 
2022-05-18T04:31:13.0975251Z Running tests...
2022-05-18T04:31:13.0975735Z ----------------------------------------------------------------------
2022-05-18T04:31:14.7496603Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:14.7858014Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10447
2022-05-18T04:31:14.7964623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10448
2022-05-18T04:31:15.9586659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:15.9613365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:15.9614705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:15.9687638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:15.9694689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:16.0627512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:17.2661607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvd3efnwr
2022-05-18T04:31:17.2662218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvd3efnwr/_remote_module_non_scriptable.py
2022-05-18T04:31:17.3381734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8gmyc44y
2022-05-18T04:31:17.3382739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8gmyc44y/_remote_module_non_scriptable.py
2022-05-18T04:31:18.5864409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:18.5864984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:18.9062028Z ok (5.808s)
2022-05-18T04:31:18.9062226Z 
2022-05-18T04:31:18.9062616Z ----------------------------------------------------------------------
2022-05-18T04:31:18.9062963Z Ran 1 test in 5.809s
2022-05-18T04:31:18.9063136Z 
2022-05-18T04:31:18.9063210Z OK
2022-05-18T04:31:18.9063349Z 
2022-05-18T04:31:18.9063483Z Generating XML reports...
2022-05-18T04:31:18.9121250Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043113.xml
2022-05-18T04:31:20.3520072Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:20.3535839Z 
2022-05-18T04:31:20.3536149Z Running tests...
2022-05-18T04:31:20.3536586Z ----------------------------------------------------------------------
2022-05-18T04:31:21.9878391Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:22.0245902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10574
2022-05-18T04:31:22.0353363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10575
2022-05-18T04:31:23.2482874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:23.2489139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:23.2490457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:23.2586628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:23.2593185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:23.3505426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:24.5448382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3avyfe0
2022-05-18T04:31:24.5449000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3avyfe0/_remote_module_non_scriptable.py
2022-05-18T04:31:24.6473808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gy7brm2
2022-05-18T04:31:24.6474632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gy7brm2/_remote_module_non_scriptable.py
2022-05-18T04:31:25.0077495Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:31:25.3435436Z ok (4.990s)
2022-05-18T04:31:25.3435650Z 
2022-05-18T04:31:25.3436019Z ----------------------------------------------------------------------
2022-05-18T04:31:25.3436359Z Ran 1 test in 4.990s
2022-05-18T04:31:25.3436523Z 
2022-05-18T04:31:25.3436625Z OK
2022-05-18T04:31:25.3436761Z 
2022-05-18T04:31:25.3436894Z Generating XML reports...
2022-05-18T04:31:25.3492509Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043120.xml
2022-05-18T04:31:26.7848318Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:26.7864720Z 
2022-05-18T04:31:26.7864968Z Running tests...
2022-05-18T04:31:26.7865424Z ----------------------------------------------------------------------
2022-05-18T04:31:28.4309395Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:28.4670827Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10700
2022-05-18T04:31:28.4779509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10701
2022-05-18T04:31:29.6110119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:29.6259253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:29.6260300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:29.6312738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:29.6319920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:29.7275480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:30.9218135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1c9wt1d
2022-05-18T04:31:30.9219297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1c9wt1d/_remote_module_non_scriptable.py
2022-05-18T04:31:31.0252783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6esp4dhh
2022-05-18T04:31:31.0254992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6esp4dhh/_remote_module_non_scriptable.py
2022-05-18T04:31:31.3878461Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:31:31.3880117Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:31:31.7864795Z ok (5.000s)
2022-05-18T04:31:31.7865023Z 
2022-05-18T04:31:31.7865614Z ----------------------------------------------------------------------
2022-05-18T04:31:31.7866141Z Ran 1 test in 5.000s
2022-05-18T04:31:31.7866314Z 
2022-05-18T04:31:31.7866401Z OK
2022-05-18T04:31:31.7866542Z 
2022-05-18T04:31:31.7866684Z Generating XML reports...
2022-05-18T04:31:31.7925873Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043126.xml
2022-05-18T04:31:33.2135851Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:33.2150430Z 
2022-05-18T04:31:33.2150572Z Running tests...
2022-05-18T04:31:33.2151025Z ----------------------------------------------------------------------
2022-05-18T04:31:33.2177176Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2022-05-18T04:31:33.2177461Z 
2022-05-18T04:31:33.2177725Z ----------------------------------------------------------------------
2022-05-18T04:31:33.2178053Z Ran 1 test in 0.003s
2022-05-18T04:31:33.2178223Z 
2022-05-18T04:31:33.2178315Z OK (skipped=1)
2022-05-18T04:31:33.2178493Z 
2022-05-18T04:31:33.2178622Z Generating XML reports...
2022-05-18T04:31:33.2219261Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043133.xml
2022-05-18T04:31:34.4628697Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:34.4642740Z 
2022-05-18T04:31:34.4643052Z Running tests...
2022-05-18T04:31:34.4643780Z ----------------------------------------------------------------------
2022-05-18T04:31:36.0993466Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:36.1354013Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10861
2022-05-18T04:31:36.1456699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10862
2022-05-18T04:31:37.2970858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:37.3180207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:37.3181450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:37.3274612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:37.3281597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:37.4196069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:38.6808037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffa9ofh4
2022-05-18T04:31:38.6809004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffa9ofh4/_remote_module_non_scriptable.py
2022-05-18T04:31:38.7059552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfz5wvrwz
2022-05-18T04:31:38.7060985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfz5wvrwz/_remote_module_non_scriptable.py
2022-05-18T04:31:40.0357423Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:40.0357991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:40.4556484Z ok (5.991s)
2022-05-18T04:31:40.4556704Z 
2022-05-18T04:31:40.4557084Z ----------------------------------------------------------------------
2022-05-18T04:31:40.4557408Z Ran 1 test in 5.991s
2022-05-18T04:31:40.4557580Z 
2022-05-18T04:31:40.4557678Z OK
2022-05-18T04:31:40.4557814Z 
2022-05-18T04:31:40.4557949Z Generating XML reports...
2022-05-18T04:31:40.4614208Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043134.xml
2022-05-18T04:31:41.9052252Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:41.9068553Z 
2022-05-18T04:31:41.9068803Z Running tests...
2022-05-18T04:31:41.9069242Z ----------------------------------------------------------------------
2022-05-18T04:31:43.5595528Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:43.5963917Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10988
2022-05-18T04:31:43.6070367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10989
2022-05-18T04:31:44.7555144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:44.7600956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:44.7601778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:44.7656854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:44.7664256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:44.8617394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:46.0592745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20966doj
2022-05-18T04:31:46.0593657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20966doj/_remote_module_non_scriptable.py
2022-05-18T04:31:46.1510927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpppswaep1
2022-05-18T04:31:46.1511728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpppswaep1/_remote_module_non_scriptable.py
2022-05-18T04:31:47.1914602Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T04:31:47.1916429Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T04:31:47.1918563Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T04:31:47.1919882Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T04:31:47.8170708Z ok (5.910s)
2022-05-18T04:31:47.8170936Z 
2022-05-18T04:31:47.8171323Z ----------------------------------------------------------------------
2022-05-18T04:31:47.8171643Z Ran 1 test in 5.910s
2022-05-18T04:31:47.8171810Z 
2022-05-18T04:31:47.8172178Z OK
2022-05-18T04:31:47.8174230Z 
2022-05-18T04:31:47.8174569Z Generating XML reports...
2022-05-18T04:31:47.8228206Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043141.xml
2022-05-18T04:31:49.2178350Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:49.2194387Z 
2022-05-18T04:31:49.2194650Z Running tests...
2022-05-18T04:31:49.2195067Z ----------------------------------------------------------------------
2022-05-18T04:31:50.8662529Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:50.9023257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11145
2022-05-18T04:31:50.9128668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11146
2022-05-18T04:31:52.0452833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:52.0661429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:52.0662942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:52.0756140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:52.0763212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:52.1678393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:53.3716704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqr8geev9
2022-05-18T04:31:53.3717912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqr8geev9/_remote_module_non_scriptable.py
2022-05-18T04:31:53.4736056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgezz9mzl
2022-05-18T04:31:53.4737192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgezz9mzl/_remote_module_non_scriptable.py
2022-05-18T04:31:54.7137671Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:54.7138280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:54.7327059Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpgezz9mzl'>
2022-05-18T04:31:54.7328100Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:31:54.7329425Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpqr8geev9'>
2022-05-18T04:31:54.7330769Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:31:55.1228803Z ok (5.903s)
2022-05-18T04:31:55.1228998Z 
2022-05-18T04:31:55.1229709Z ----------------------------------------------------------------------
2022-05-18T04:31:55.1230105Z Ran 1 test in 5.903s
2022-05-18T04:31:55.1230276Z 
2022-05-18T04:31:55.1230375Z OK
2022-05-18T04:31:55.1230522Z 
2022-05-18T04:31:55.1230655Z Generating XML reports...
2022-05-18T04:31:55.1287394Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043149.xml
2022-05-18T04:31:56.5669441Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:56.5693201Z 
2022-05-18T04:31:56.5693686Z Running tests...
2022-05-18T04:31:56.5694201Z ----------------------------------------------------------------------
2022-05-18T04:31:58.2292242Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:58.2414187Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.672s)
2022-05-18T04:31:58.2414919Z 
2022-05-18T04:31:58.2415214Z ----------------------------------------------------------------------
2022-05-18T04:31:58.2415557Z Ran 1 test in 1.672s
2022-05-18T04:31:58.2415721Z 
2022-05-18T04:31:58.2415810Z OK (skipped=1)
2022-05-18T04:31:58.2415966Z 
2022-05-18T04:31:58.2416091Z Generating XML reports...
2022-05-18T04:31:58.2454151Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043156.xml
2022-05-18T04:31:59.6317130Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:59.6332498Z 
2022-05-18T04:31:59.6332990Z Running tests...
2022-05-18T04:31:59.6333480Z ----------------------------------------------------------------------
2022-05-18T04:32:01.2839296Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:01.3206710Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11308
2022-05-18T04:32:01.3312936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11309
2022-05-18T04:32:02.5127106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:02.5660788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:02.5661591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:02.5734899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:02.5741475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:02.5744722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:32:02.6671934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:02.6675793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:32:02.6676517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:32:02.6760827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:32:04.0075253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4e349ej
2022-05-18T04:32:04.0076313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4e349ej/_remote_module_non_scriptable.py
2022-05-18T04:32:04.0129614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtofj4yp
2022-05-18T04:32:04.0132785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtofj4yp/_remote_module_non_scriptable.py
2022-05-18T04:32:05.3448289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:05.3448862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:05.3458626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:05.3461272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:05.9420968Z ok (6.308s)
2022-05-18T04:32:05.9421184Z 
2022-05-18T04:32:05.9422097Z ----------------------------------------------------------------------
2022-05-18T04:32:05.9422769Z Ran 1 test in 6.309s
2022-05-18T04:32:05.9423092Z 
2022-05-18T04:32:05.9423253Z OK
2022-05-18T04:32:05.9423511Z 
2022-05-18T04:32:05.9423719Z Generating XML reports...
2022-05-18T04:32:05.9481693Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043159.xml
2022-05-18T04:32:07.3870474Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:07.3885717Z 
2022-05-18T04:32:07.3885972Z Running tests...
2022-05-18T04:32:07.3886419Z ----------------------------------------------------------------------
2022-05-18T04:32:09.0398600Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:09.0766051Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11439
2022-05-18T04:32:09.0872719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11440
2022-05-18T04:32:10.2359091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:10.3089684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:10.3090765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:10.3170286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:10.3177510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:10.3179597Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:32:10.4104680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:10.4105662Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:32:11.6114040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4_b3ndi
2022-05-18T04:32:11.6114656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4_b3ndi/_remote_module_non_scriptable.py
2022-05-18T04:32:11.7153530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp59rtxy6
2022-05-18T04:32:11.7154822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp59rtxy6/_remote_module_non_scriptable.py
2022-05-18T04:32:13.0610570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:13.0611778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:13.0621129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:13.0625238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:13.0835428Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:32:13.0839153Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:32:13.3002938Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:32:13.3005607Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:32:13.3065808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:13.3068430Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:13.3077043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:13.3081152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:13.3295301Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:32:13.3298931Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:32:13.4654507Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T04:32:13.4663946Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T04:32:13.4721633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:13.4725090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:13.4734058Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:13.4737927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:14.0989116Z ok (6.710s)
2022-05-18T04:32:14.0989419Z 
2022-05-18T04:32:14.0990061Z ----------------------------------------------------------------------
2022-05-18T04:32:14.0990447Z Ran 1 test in 6.710s
2022-05-18T04:32:14.0990612Z 
2022-05-18T04:32:14.0990707Z OK
2022-05-18T04:32:14.0990862Z 
2022-05-18T04:32:14.0993806Z Generating XML reports...
2022-05-18T04:32:14.1047575Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043207.xml
2022-05-18T04:32:15.5401096Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:15.5417373Z 
2022-05-18T04:32:15.5417534Z Running tests...
2022-05-18T04:32:15.5418577Z ----------------------------------------------------------------------
2022-05-18T04:32:17.1933194Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:17.2300183Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11566
2022-05-18T04:32:17.2408783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11567
2022-05-18T04:32:18.4319240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:18.4373988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:18.4374785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:18.4420290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:18.4426780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:18.4429404Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:32:18.5389769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:18.5390615Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:32:19.7122478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnwinm4m
2022-05-18T04:32:19.7123172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnwinm4m/_remote_module_non_scriptable.py
2022-05-18T04:32:19.8548627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mdihdjo
2022-05-18T04:32:19.8549617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mdihdjo/_remote_module_non_scriptable.py
2022-05-18T04:32:21.1950179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:21.1951039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:21.1959843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:21.1960350Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:21.1966867Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:32:21.1967419Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:32:21.1993738Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:32:21.1995115Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:32:21.1995761Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:32:21.1997091Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T04:32:21.1997883Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:32:21.1998979Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T04:32:21.4953804Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:32:21.4957662Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:32:21.5020918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:21.5023182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:21.5032401Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:21.5036266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:21.5038571Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:32:21.5042910Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:32:21.5067702Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:32:21.5068831Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:32:21.5069476Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:32:21.5071884Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:32:22.1517464Z ok (6.610s)
2022-05-18T04:32:22.1517850Z 
2022-05-18T04:32:22.1518506Z ----------------------------------------------------------------------
2022-05-18T04:32:22.1519123Z Ran 1 test in 6.610s
2022-05-18T04:32:22.1519431Z 
2022-05-18T04:32:22.1519855Z OK
2022-05-18T04:32:22.1520105Z 
2022-05-18T04:32:22.1520357Z Generating XML reports...
2022-05-18T04:32:22.1577516Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043215.xml
2022-05-18T04:32:23.5715940Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:23.5730198Z 
2022-05-18T04:32:23.5730461Z Running tests...
2022-05-18T04:32:23.5730895Z ----------------------------------------------------------------------
2022-05-18T04:32:23.5753458Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:32:23.5753894Z 
2022-05-18T04:32:23.5754177Z ----------------------------------------------------------------------
2022-05-18T04:32:23.5754485Z Ran 1 test in 0.002s
2022-05-18T04:32:23.5754648Z 
2022-05-18T04:32:23.5754780Z OK (skipped=1)
2022-05-18T04:32:23.5754934Z 
2022-05-18T04:32:23.5755057Z Generating XML reports...
2022-05-18T04:32:23.5795040Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043223.xml
2022-05-18T04:32:24.8430936Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:24.8446421Z 
2022-05-18T04:32:24.8446686Z Running tests...
2022-05-18T04:32:24.8447124Z ----------------------------------------------------------------------
2022-05-18T04:32:24.8470938Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:32:24.8471520Z 
2022-05-18T04:32:24.8471821Z ----------------------------------------------------------------------
2022-05-18T04:32:24.8472156Z Ran 1 test in 0.002s
2022-05-18T04:32:24.8473204Z 
2022-05-18T04:32:24.8473563Z OK (skipped=1)
2022-05-18T04:32:24.8473889Z 
2022-05-18T04:32:24.8474029Z Generating XML reports...
2022-05-18T04:32:24.8515433Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043224.xml
2022-05-18T04:32:26.1176387Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:26.1191567Z 
2022-05-18T04:32:26.1191854Z Running tests...
2022-05-18T04:32:26.1192293Z ----------------------------------------------------------------------
2022-05-18T04:32:26.1217670Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:32:26.1218522Z 
2022-05-18T04:32:26.1219440Z ----------------------------------------------------------------------
2022-05-18T04:32:26.1220153Z Ran 1 test in 0.003s
2022-05-18T04:32:26.1220451Z 
2022-05-18T04:32:26.1220570Z OK (skipped=1)
2022-05-18T04:32:26.1220726Z 
2022-05-18T04:32:26.1220851Z Generating XML reports...
2022-05-18T04:32:26.1262670Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043226.xml
2022-05-18T04:32:27.3931312Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:27.3946800Z 
2022-05-18T04:32:27.3947168Z Running tests...
2022-05-18T04:32:27.3948136Z ----------------------------------------------------------------------
2022-05-18T04:32:27.3973448Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:32:27.3974532Z 
2022-05-18T04:32:27.3974990Z ----------------------------------------------------------------------
2022-05-18T04:32:27.3975596Z Ran 1 test in 0.003s
2022-05-18T04:32:27.3975761Z 
2022-05-18T04:32:27.3975870Z OK (skipped=1)
2022-05-18T04:32:27.3976025Z 
2022-05-18T04:32:27.3976149Z Generating XML reports...
2022-05-18T04:32:27.4017079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043227.xml
2022-05-18T04:32:28.6748951Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:28.6764160Z 
2022-05-18T04:32:28.6764361Z Running tests...
2022-05-18T04:32:28.6764794Z ----------------------------------------------------------------------
2022-05-18T04:32:28.6789427Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:32:28.6790111Z 
2022-05-18T04:32:28.6790744Z ----------------------------------------------------------------------
2022-05-18T04:32:28.6791368Z Ran 1 test in 0.003s
2022-05-18T04:32:28.6791543Z 
2022-05-18T04:32:28.6791657Z OK (skipped=1)
2022-05-18T04:32:28.6791809Z 
2022-05-18T04:32:28.6792241Z Generating XML reports...
2022-05-18T04:32:28.6833747Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043228.xml
2022-05-18T04:32:29.9400814Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:29.9415396Z 
2022-05-18T04:32:29.9415715Z Running tests...
2022-05-18T04:32:29.9416141Z ----------------------------------------------------------------------
2022-05-18T04:32:29.9440632Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:32:29.9441140Z 
2022-05-18T04:32:29.9441442Z ----------------------------------------------------------------------
2022-05-18T04:32:29.9441770Z Ran 1 test in 0.003s
2022-05-18T04:32:29.9441931Z 
2022-05-18T04:32:29.9442023Z OK (skipped=1)
2022-05-18T04:32:29.9442192Z 
2022-05-18T04:32:29.9442317Z Generating XML reports...
2022-05-18T04:32:29.9483011Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043229.xml
2022-05-18T04:32:31.2111037Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:31.2126557Z 
2022-05-18T04:32:31.2127045Z Running tests...
2022-05-18T04:32:31.2127657Z ----------------------------------------------------------------------
2022-05-18T04:32:31.2151598Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:32:31.2152426Z 
2022-05-18T04:32:31.2152944Z ----------------------------------------------------------------------
2022-05-18T04:32:31.2153288Z Ran 1 test in 0.003s
2022-05-18T04:32:31.2153433Z 
2022-05-18T04:32:31.2153554Z OK (skipped=1)
2022-05-18T04:32:31.2153707Z 
2022-05-18T04:32:31.2153830Z Generating XML reports...
2022-05-18T04:32:31.2196361Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043231.xml
2022-05-18T04:32:32.4942829Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:32.4958196Z 
2022-05-18T04:32:32.4958647Z Running tests...
2022-05-18T04:32:32.4959133Z ----------------------------------------------------------------------
2022-05-18T04:32:32.4983848Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:32:32.4985062Z 
2022-05-18T04:32:32.4985382Z ----------------------------------------------------------------------
2022-05-18T04:32:32.4985695Z Ran 1 test in 0.003s
2022-05-18T04:32:32.4985869Z 
2022-05-18T04:32:32.4985977Z OK (skipped=1)
2022-05-18T04:32:32.4986129Z 
2022-05-18T04:32:32.4986252Z Generating XML reports...
2022-05-18T04:32:32.5026959Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043232.xml
2022-05-18T04:32:33.7498876Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:33.7513180Z 
2022-05-18T04:32:33.7513622Z Running tests...
2022-05-18T04:32:33.7514130Z ----------------------------------------------------------------------
2022-05-18T04:32:33.7537656Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:32:33.7538533Z 
2022-05-18T04:32:33.7538832Z ----------------------------------------------------------------------
2022-05-18T04:32:33.7539454Z Ran 1 test in 0.002s
2022-05-18T04:32:33.7539619Z 
2022-05-18T04:32:33.7539727Z OK (skipped=1)
2022-05-18T04:32:33.7539882Z 
2022-05-18T04:32:33.7540009Z Generating XML reports...
2022-05-18T04:32:33.7579641Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043233.xml
2022-05-18T04:32:34.9931202Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:34.9947835Z 
2022-05-18T04:32:34.9948271Z Running tests...
2022-05-18T04:32:34.9948804Z ----------------------------------------------------------------------
2022-05-18T04:32:34.9973735Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:32:34.9974228Z 
2022-05-18T04:32:34.9974636Z ----------------------------------------------------------------------
2022-05-18T04:32:34.9975261Z Ran 1 test in 0.003s
2022-05-18T04:32:34.9975514Z 
2022-05-18T04:32:34.9975631Z OK (skipped=1)
2022-05-18T04:32:34.9975786Z 
2022-05-18T04:32:34.9975909Z Generating XML reports...
2022-05-18T04:32:35.0018814Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043234.xml
2022-05-18T04:32:36.2674799Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:36.2690714Z 
2022-05-18T04:32:36.2691061Z Running tests...
2022-05-18T04:32:36.2691494Z ----------------------------------------------------------------------
2022-05-18T04:32:36.2714935Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:32:36.2715371Z 
2022-05-18T04:32:36.2715652Z ----------------------------------------------------------------------
2022-05-18T04:32:36.2715990Z Ran 1 test in 0.002s
2022-05-18T04:32:36.2716153Z 
2022-05-18T04:32:36.2716267Z OK (skipped=1)
2022-05-18T04:32:36.2716421Z 
2022-05-18T04:32:36.2716527Z Generating XML reports...
2022-05-18T04:32:36.2759173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043236.xml
2022-05-18T04:32:37.5453959Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:37.5470288Z 
2022-05-18T04:32:37.5470550Z Running tests...
2022-05-18T04:32:37.5470991Z ----------------------------------------------------------------------
2022-05-18T04:32:37.5495039Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:32:37.5495494Z 
2022-05-18T04:32:37.5495784Z ----------------------------------------------------------------------
2022-05-18T04:32:37.5496119Z Ran 1 test in 0.002s
2022-05-18T04:32:37.5496284Z 
2022-05-18T04:32:37.5496392Z OK (skipped=1)
2022-05-18T04:32:37.5496530Z 
2022-05-18T04:32:37.5496655Z Generating XML reports...
2022-05-18T04:32:37.5539881Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043237.xml
2022-05-18T04:32:38.8020001Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:38.8034121Z 
2022-05-18T04:32:38.8034607Z Running tests...
2022-05-18T04:32:38.8035142Z ----------------------------------------------------------------------
2022-05-18T04:32:40.4328906Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:40.4445937Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.641s)
2022-05-18T04:32:40.4446828Z 
2022-05-18T04:32:40.4447112Z ----------------------------------------------------------------------
2022-05-18T04:32:40.4447425Z Ran 1 test in 1.641s
2022-05-18T04:32:40.4447588Z 
2022-05-18T04:32:40.4447702Z OK (skipped=1)
2022-05-18T04:32:40.4447859Z 
2022-05-18T04:32:40.4447984Z Generating XML reports...
2022-05-18T04:32:40.4484895Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043238.xml
2022-05-18T04:32:41.8307781Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:41.8322655Z 
2022-05-18T04:32:41.8323170Z Running tests...
2022-05-18T04:32:41.8323627Z ----------------------------------------------------------------------
2022-05-18T04:32:43.4858865Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:43.5224538Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12149
2022-05-18T04:32:43.5330611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12150
2022-05-18T04:32:44.7135276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:44.7351352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:44.7352196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:44.7439536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:44.7446035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:44.8366592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:46.0342505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77jiiag7
2022-05-18T04:32:46.0343128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77jiiag7/_remote_module_non_scriptable.py
2022-05-18T04:32:46.1472170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3r92th9g
2022-05-18T04:32:46.1473011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3r92th9g/_remote_module_non_scriptable.py
2022-05-18T04:32:47.1420972Z ok (5.310s)
2022-05-18T04:32:47.1421315Z 
2022-05-18T04:32:47.1421810Z ----------------------------------------------------------------------
2022-05-18T04:32:47.1422155Z Ran 1 test in 5.310s
2022-05-18T04:32:47.1422319Z 
2022-05-18T04:32:47.1422628Z OK
2022-05-18T04:32:47.1422782Z 
2022-05-18T04:32:47.1422917Z Generating XML reports...
2022-05-18T04:32:47.1478771Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043241.xml
2022-05-18T04:32:48.5621711Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:48.5636957Z 
2022-05-18T04:32:48.5637199Z Running tests...
2022-05-18T04:32:48.5637787Z ----------------------------------------------------------------------
2022-05-18T04:32:50.1796212Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:50.2155562Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12271
2022-05-18T04:32:50.2265010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12272
2022-05-18T04:32:51.3233130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:51.3758181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:51.3758964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:51.3840427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:51.3846959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:51.4773159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:52.9454413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptx4yf501
2022-05-18T04:32:52.9455367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptx4yf501/_remote_module_non_scriptable.py
2022-05-18T04:32:53.0619722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3jqxugr
2022-05-18T04:32:53.0620944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3jqxugr/_remote_module_non_scriptable.py
2022-05-18T04:32:54.1165008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:54.1165541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:54.1254665Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmptx4yf501'>
2022-05-18T04:32:54.1255141Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:32:54.1255728Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpn3jqxugr'>
2022-05-18T04:32:54.1256162Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:32:54.5365613Z ok (5.973s)
2022-05-18T04:32:54.5366005Z 
2022-05-18T04:32:54.5366418Z ----------------------------------------------------------------------
2022-05-18T04:32:54.5366759Z Ran 1 test in 5.973s
2022-05-18T04:32:54.5366926Z 
2022-05-18T04:32:54.5367024Z OK
2022-05-18T04:32:54.5367142Z 
2022-05-18T04:32:54.5367276Z Generating XML reports...
2022-05-18T04:32:54.5424180Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043248.xml
2022-05-18T04:32:55.9471571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:55.9486445Z 
2022-05-18T04:32:55.9486683Z Running tests...
2022-05-18T04:32:55.9487269Z ----------------------------------------------------------------------
2022-05-18T04:32:55.9560963Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.007s)
2022-05-18T04:32:55.9561839Z 
2022-05-18T04:32:55.9562373Z ----------------------------------------------------------------------
2022-05-18T04:32:55.9562860Z Ran 1 test in 0.007s
2022-05-18T04:32:55.9563025Z 
2022-05-18T04:32:55.9563384Z OK (skipped=1)
2022-05-18T04:32:55.9563556Z 
2022-05-18T04:32:55.9563681Z Generating XML reports...
2022-05-18T04:32:55.9603614Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043255.xml
2022-05-18T04:32:57.2323047Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:57.2338654Z 
2022-05-18T04:32:57.2338934Z Running tests...
2022-05-18T04:32:57.2339393Z ----------------------------------------------------------------------
2022-05-18T04:32:58.8993174Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:58.9361278Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12433
2022-05-18T04:32:58.9468456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12434
2022-05-18T04:33:00.1166579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:00.1271726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:00.1272529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:00.1369083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:00.1376235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:00.2286035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:01.4257077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeiern3wp
2022-05-18T04:33:01.4257695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeiern3wp/_remote_module_non_scriptable.py
2022-05-18T04:33:01.4996032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpncv53hub
2022-05-18T04:33:01.4996643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpncv53hub/_remote_module_non_scriptable.py
2022-05-18T04:33:02.8368853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:33:02.8369418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:33:03.2570237Z ok (6.023s)
2022-05-18T04:33:03.2570432Z 
2022-05-18T04:33:03.2570814Z ----------------------------------------------------------------------
2022-05-18T04:33:03.2571131Z Ran 1 test in 6.023s
2022-05-18T04:33:03.2571300Z 
2022-05-18T04:33:03.2571395Z OK
2022-05-18T04:33:03.2571529Z 
2022-05-18T04:33:03.2571660Z Generating XML reports...
2022-05-18T04:33:03.2628724Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043257.xml
2022-05-18T04:33:04.6801900Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:04.6817440Z 
2022-05-18T04:33:04.6817580Z Running tests...
2022-05-18T04:33:04.6818329Z ----------------------------------------------------------------------
2022-05-18T04:33:06.3021314Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:06.3379785Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12560
2022-05-18T04:33:06.3490237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12561
2022-05-18T04:33:07.5158342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:07.5465824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:07.5466633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:07.5563501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:07.5570024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:07.6477485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:07.6587126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:07.6587838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:07.6588931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:33:07.6589641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:33:07.6591737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:33:07.6692304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:33:07.6693300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:33:07.6694438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:33:09.0045968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmploap0hr5
2022-05-18T04:33:09.0047901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmploap0hr5/_remote_module_non_scriptable.py
2022-05-18T04:33:09.0227605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuft2qgka
2022-05-18T04:33:09.0230238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuft2qgka/_remote_module_non_scriptable.py
2022-05-18T04:33:09.3564882Z ok (4.674s)
2022-05-18T04:33:09.3565136Z 
2022-05-18T04:33:09.3565745Z ----------------------------------------------------------------------
2022-05-18T04:33:09.3566080Z Ran 1 test in 4.675s
2022-05-18T04:33:09.3566260Z 
2022-05-18T04:33:09.3566351Z OK
2022-05-18T04:33:09.3566484Z 
2022-05-18T04:33:09.3566616Z Generating XML reports...
2022-05-18T04:33:09.3623616Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043304.xml
2022-05-18T04:33:10.8087463Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:10.8103282Z 
2022-05-18T04:33:10.8103678Z Running tests...
2022-05-18T04:33:10.8104147Z ----------------------------------------------------------------------
2022-05-18T04:33:12.4494436Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:12.4858546Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12690
2022-05-18T04:33:12.4964122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12691
2022-05-18T04:33:13.6445966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:13.6702343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:13.6703148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:13.6750195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:13.6757057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:13.7714419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:13.7924167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:13.7924708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:13.7925408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:33:13.7926089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:33:13.7927501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:33:13.7928319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:33:13.7929005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:33:13.7929923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:33:15.1294872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnvzph51
2022-05-18T04:33:15.1295791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnvzph51/_remote_module_non_scriptable.py
2022-05-18T04:33:15.1463543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpudu4ls1s
2022-05-18T04:33:15.1466227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpudu4ls1s/_remote_module_non_scriptable.py
2022-05-18T04:33:25.6303909Z [W ProcessGroupNCCL.cpp:865] [Rank 0] Found key in store: NCCLABORTEDCOMM:20ecdac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-05-18T04:33:25.6304926Z [W ProcessGroupNCCL.cpp:865] [Rank 1] Found key in store: NCCLABORTEDCOMM:20ecdac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-05-18T04:33:25.9215131Z ok (15.111s)
2022-05-18T04:33:25.9215458Z 
2022-05-18T04:33:25.9215962Z ----------------------------------------------------------------------
2022-05-18T04:33:25.9216304Z Ran 1 test in 15.111s
2022-05-18T04:33:25.9216472Z 
2022-05-18T04:33:25.9216570Z OK
2022-05-18T04:33:25.9216707Z 
2022-05-18T04:33:25.9216823Z Generating XML reports...
2022-05-18T04:33:25.9272426Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043310.xml
2022-05-18T04:33:27.3673620Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:27.3689205Z 
2022-05-18T04:33:27.3689750Z Running tests...
2022-05-18T04:33:27.3690669Z ----------------------------------------------------------------------
2022-05-18T04:33:29.0133303Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:29.0504318Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12820
2022-05-18T04:33:29.0611968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12821
2022-05-18T04:33:30.2537416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:30.2583712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:30.2584500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:30.2638360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:30.2645615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:30.3599165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:31.5993904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmsx8hap
2022-05-18T04:33:31.5994734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmsx8hap/_remote_module_non_scriptable.py
2022-05-18T04:33:31.6754343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpng8c240g
2022-05-18T04:33:31.6755350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpng8c240g/_remote_module_non_scriptable.py
2022-05-18T04:33:33.7719429Z ok (6.403s)
2022-05-18T04:33:33.7719645Z 
2022-05-18T04:33:33.7720037Z ----------------------------------------------------------------------
2022-05-18T04:33:33.7720392Z Ran 1 test in 6.403s
2022-05-18T04:33:33.7720538Z 
2022-05-18T04:33:33.7720634Z OK
2022-05-18T04:33:33.7720771Z 
2022-05-18T04:33:33.7720932Z Generating XML reports...
2022-05-18T04:33:33.7788789Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043327.xml
2022-05-18T04:33:35.1871771Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:35.1885955Z 
2022-05-18T04:33:35.1886234Z Running tests...
2022-05-18T04:33:35.1886690Z ----------------------------------------------------------------------
2022-05-18T04:33:36.7823965Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:36.8181839Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12947
2022-05-18T04:33:36.8287848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12948
2022-05-18T04:33:38.0062080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:38.0261282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:38.0262077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:38.0264583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:38.0272228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:38.1276253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:39.3611945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkf6sx40
2022-05-18T04:33:39.3612938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkf6sx40/_remote_module_non_scriptable.py
2022-05-18T04:33:39.4318392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7j9ql_sk
2022-05-18T04:33:39.4319463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7j9ql_sk/_remote_module_non_scriptable.py
2022-05-18T04:33:41.4392633Z ok (6.250s)
2022-05-18T04:33:41.4393041Z 
2022-05-18T04:33:41.4393819Z ----------------------------------------------------------------------
2022-05-18T04:33:41.4394305Z Ran 1 test in 6.251s
2022-05-18T04:33:41.4394471Z 
2022-05-18T04:33:41.4394570Z OK
2022-05-18T04:33:41.4394709Z 
2022-05-18T04:33:41.4394851Z Generating XML reports...
2022-05-18T04:33:41.4450938Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043335.xml
2022-05-18T04:33:42.8798764Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:42.8813710Z 
2022-05-18T04:33:42.8814134Z Running tests...
2022-05-18T04:33:42.8815081Z ----------------------------------------------------------------------
2022-05-18T04:33:44.5392466Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:44.5760674Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13074
2022-05-18T04:33:44.5867356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13075
2022-05-18T04:33:45.7648739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:45.7818051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:45.7818842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:45.7851636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:45.7858542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:45.8833834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:47.0578137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpny0dedvq
2022-05-18T04:33:47.0579300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpny0dedvq/_remote_module_non_scriptable.py
2022-05-18T04:33:47.2176287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphktbv8_y
2022-05-18T04:33:47.2177152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphktbv8_y/_remote_module_non_scriptable.py
2022-05-18T04:33:48.8967490Z ok (6.015s)
2022-05-18T04:33:48.8967679Z 
2022-05-18T04:33:48.8968066Z ----------------------------------------------------------------------
2022-05-18T04:33:48.8968417Z Ran 1 test in 6.015s
2022-05-18T04:33:48.8968585Z 
2022-05-18T04:33:48.8968685Z OK
2022-05-18T04:33:48.8968823Z 
2022-05-18T04:33:48.8968933Z Generating XML reports...
2022-05-18T04:33:48.9026300Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043342.xml
2022-05-18T04:33:50.3155969Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:50.3170216Z 
2022-05-18T04:33:50.3170422Z Running tests...
2022-05-18T04:33:50.3170835Z ----------------------------------------------------------------------
2022-05-18T04:33:51.9280562Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:51.9639406Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13197
2022-05-18T04:33:51.9746990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13198
2022-05-18T04:33:53.0981887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:53.1091126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:53.1091947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:53.1184140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:53.1191521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:53.2107028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:54.4100158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5xomjzn
2022-05-18T04:33:54.4100780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5xomjzn/_remote_module_non_scriptable.py
2022-05-18T04:33:54.5043011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvv_ceyd
2022-05-18T04:33:54.5043598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvv_ceyd/_remote_module_non_scriptable.py
2022-05-18T04:33:55.7152164Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:33:55.7198263Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:33:55.7273534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:33:55.7274211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:33:56.0842209Z ok (5.767s)
2022-05-18T04:33:56.0842457Z 
2022-05-18T04:33:56.0842850Z ----------------------------------------------------------------------
2022-05-18T04:33:56.0843200Z Ran 1 test in 5.767s
2022-05-18T04:33:56.0843366Z 
2022-05-18T04:33:56.0843468Z OK
2022-05-18T04:33:56.0843606Z 
2022-05-18T04:33:56.0843721Z Generating XML reports...
2022-05-18T04:33:56.0901226Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043350.xml
2022-05-18T04:33:57.5407135Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:57.5428335Z 
2022-05-18T04:33:57.5428726Z Running tests...
2022-05-18T04:33:57.5429200Z ----------------------------------------------------------------------
2022-05-18T04:33:59.1938931Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:59.2299829Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13324
2022-05-18T04:33:59.2404798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13325
2022-05-18T04:34:00.4133986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:00.4340537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:00.4342026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:00.4440185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:00.4446571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:00.5357213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:01.7341481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkpohu7x
2022-05-18T04:34:01.7342633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkpohu7x/_remote_module_non_scriptable.py
2022-05-18T04:34:01.8559814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8skiwx9
2022-05-18T04:34:01.8560961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8skiwx9/_remote_module_non_scriptable.py
2022-05-18T04:34:02.9180408Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:34:02.9182097Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T04:34:02.9184218Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:34:02.9185698Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T04:34:03.5504203Z ok (6.007s)
2022-05-18T04:34:03.5504454Z 
2022-05-18T04:34:03.5504883Z ----------------------------------------------------------------------
2022-05-18T04:34:03.5505223Z Ran 1 test in 6.008s
2022-05-18T04:34:03.5505395Z 
2022-05-18T04:34:03.5505511Z OK
2022-05-18T04:34:03.5505626Z 
2022-05-18T04:34:03.5505770Z Generating XML reports...
2022-05-18T04:34:03.5564023Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043357.xml
2022-05-18T04:34:04.9971353Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:04.9986567Z 
2022-05-18T04:34:04.9987077Z Running tests...
2022-05-18T04:34:04.9987549Z ----------------------------------------------------------------------
2022-05-18T04:34:06.6383095Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:06.6504470Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.651s)
2022-05-18T04:34:06.6505356Z 
2022-05-18T04:34:06.6505655Z ----------------------------------------------------------------------
2022-05-18T04:34:06.6505972Z Ran 1 test in 1.652s
2022-05-18T04:34:06.6506148Z 
2022-05-18T04:34:06.6506259Z OK (skipped=1)
2022-05-18T04:34:06.6506419Z 
2022-05-18T04:34:06.6506770Z Generating XML reports...
2022-05-18T04:34:06.6546685Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043404.xml
2022-05-18T04:34:08.0510625Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:08.0525547Z 
2022-05-18T04:34:08.0525846Z Running tests...
2022-05-18T04:34:08.0526296Z ----------------------------------------------------------------------
2022-05-18T04:34:09.7128420Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:09.7498756Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13487
2022-05-18T04:34:09.7608783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13488
2022-05-18T04:34:10.8900581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:10.9022951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:10.9023728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:10.9103639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:10.9110421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:11.0038447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:12.1998863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhim_b6e
2022-05-18T04:34:12.1999706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhim_b6e/_remote_module_non_scriptable.py
2022-05-18T04:34:12.3051993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72hgbmfe
2022-05-18T04:34:12.3053357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72hgbmfe/_remote_module_non_scriptable.py
2022-05-18T04:34:13.0851322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:34:13.0852673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:34:13.1190514Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:34:13.1192612Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:34:13.4700965Z ok (5.417s)
2022-05-18T04:34:13.4701165Z 
2022-05-18T04:34:13.4701569Z ----------------------------------------------------------------------
2022-05-18T04:34:13.4701913Z Ran 1 test in 5.418s
2022-05-18T04:34:13.4702082Z 
2022-05-18T04:34:13.4702195Z OK
2022-05-18T04:34:13.4702336Z 
2022-05-18T04:34:13.4702476Z Generating XML reports...
2022-05-18T04:34:13.4759371Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043408.xml
2022-05-18T04:34:14.8897173Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:14.8912228Z 
2022-05-18T04:34:14.8912751Z Running tests...
2022-05-18T04:34:14.8913245Z ----------------------------------------------------------------------
2022-05-18T04:34:16.5121063Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:16.5483071Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13617
2022-05-18T04:34:16.5593082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13618
2022-05-18T04:34:17.6954227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:17.6995423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:17.6996421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:17.7055494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:17.7061899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:17.8011339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:18.9964169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbx6g5_30
2022-05-18T04:34:18.9964777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbx6g5_30/_remote_module_non_scriptable.py
2022-05-18T04:34:19.1235573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekzsp3gu
2022-05-18T04:34:19.1236240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekzsp3gu/_remote_module_non_scriptable.py
2022-05-18T04:34:20.3680354Z ok (5.476s)
2022-05-18T04:34:20.3680714Z 
2022-05-18T04:34:20.3681168Z ----------------------------------------------------------------------
2022-05-18T04:34:20.3681513Z Ran 1 test in 5.477s
2022-05-18T04:34:20.3681676Z 
2022-05-18T04:34:20.3681751Z OK
2022-05-18T04:34:20.3681885Z 
2022-05-18T04:34:20.3682016Z Generating XML reports...
2022-05-18T04:34:20.3740734Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043414.xml
2022-05-18T04:34:21.8226951Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:21.8241654Z 
2022-05-18T04:34:21.8242050Z Running tests...
2022-05-18T04:34:21.8242572Z ----------------------------------------------------------------------
2022-05-18T04:34:23.4609628Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:23.4971910Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13740
2022-05-18T04:34:23.5076097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13741
2022-05-18T04:34:24.6801219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:24.7065920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:24.7066723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:24.7105038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:24.7111780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:24.8081688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:26.0017862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps4oouo9m
2022-05-18T04:34:26.0018470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps4oouo9m/_remote_module_non_scriptable.py
2022-05-18T04:34:26.1120407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1qpihc26
2022-05-18T04:34:26.1121421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1qpihc26/_remote_module_non_scriptable.py
2022-05-18T04:34:26.1821454Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:34:26.1822391Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T04:34:26.1823554Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:34:26.1824385Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T04:34:26.4676082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:34:26.4676873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:34:26.4736473Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:34:26.4738073Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:34:26.4848685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:34:26.4849424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:34:26.4916675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:34:26.4917338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:34:26.8158116Z ok (4.991s)
2022-05-18T04:34:26.8158473Z 
2022-05-18T04:34:26.8159080Z ----------------------------------------------------------------------
2022-05-18T04:34:26.8159427Z Ran 1 test in 4.992s
2022-05-18T04:34:26.8159592Z 
2022-05-18T04:34:26.8159672Z OK
2022-05-18T04:34:26.8159810Z 
2022-05-18T04:34:26.8160003Z Generating XML reports...
2022-05-18T04:34:26.8216480Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043421.xml
2022-05-18T04:34:28.2520137Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:28.2535351Z 
2022-05-18T04:34:28.2535832Z Running tests...
2022-05-18T04:34:28.2536340Z ----------------------------------------------------------------------
2022-05-18T04:34:29.8976940Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:29.9339650Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13866
2022-05-18T04:34:29.9446898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13867
2022-05-18T04:34:31.0938350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:31.0955409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:31.0956392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:31.1039557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:31.1046647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:31.1971466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:32.4157796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8si8gbj8
2022-05-18T04:34:32.4158414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8si8gbj8/_remote_module_non_scriptable.py
2022-05-18T04:34:32.4936010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkq1a2wed
2022-05-18T04:34:32.4937061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkq1a2wed/_remote_module_non_scriptable.py
2022-05-18T04:34:32.5637921Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:34:32.5638884Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T04:34:32.5640044Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:34:32.5640876Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T04:34:32.8535044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:34:32.8535611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:34:33.1526769Z ok (4.899s)
2022-05-18T04:34:33.1526979Z 
2022-05-18T04:34:33.1527358Z ----------------------------------------------------------------------
2022-05-18T04:34:33.1527866Z Ran 1 test in 4.899s
2022-05-18T04:34:33.1528028Z 
2022-05-18T04:34:33.1528132Z OK
2022-05-18T04:34:33.1528266Z 
2022-05-18T04:34:33.1528392Z Generating XML reports...
2022-05-18T04:34:33.1584977Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043428.xml
2022-05-18T04:34:34.5922688Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:34.5938056Z 
2022-05-18T04:34:34.5938294Z Running tests...
2022-05-18T04:34:34.5938830Z ----------------------------------------------------------------------
2022-05-18T04:34:36.2372147Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:36.2495623Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.655s)
2022-05-18T04:34:36.2496209Z 
2022-05-18T04:34:36.2496487Z ----------------------------------------------------------------------
2022-05-18T04:34:36.2496798Z Ran 1 test in 1.656s
2022-05-18T04:34:36.2496960Z 
2022-05-18T04:34:36.2497069Z OK (skipped=1)
2022-05-18T04:34:36.2499052Z 
2022-05-18T04:34:36.2499506Z Generating XML reports...
2022-05-18T04:34:36.2536415Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043434.xml
2022-05-18T04:34:37.6218017Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:37.6235479Z 
2022-05-18T04:34:37.6235815Z Running tests...
2022-05-18T04:34:37.6236513Z ----------------------------------------------------------------------
2022-05-18T04:34:39.2501426Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:39.2866297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14028
2022-05-18T04:34:39.2971704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14029
2022-05-18T04:34:40.4521314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:40.4717749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:40.4718555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:40.4723791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:40.4730696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:40.5732669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:41.7918832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6o2emyqo
2022-05-18T04:34:41.7919479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6o2emyqo/_remote_module_non_scriptable.py
2022-05-18T04:34:41.8448403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrmws727
2022-05-18T04:34:41.8450035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrmws727/_remote_module_non_scriptable.py
2022-05-18T04:34:42.2844039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:34:42.2844582Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:34:43.4070539Z ok (5.783s)
2022-05-18T04:34:43.4070981Z 
2022-05-18T04:34:43.4071699Z ----------------------------------------------------------------------
2022-05-18T04:34:43.4072079Z Ran 1 test in 5.784s
2022-05-18T04:34:43.4072244Z 
2022-05-18T04:34:43.4072589Z OK
2022-05-18T04:34:43.4072727Z 
2022-05-18T04:34:43.4072860Z Generating XML reports...
2022-05-18T04:34:43.4128767Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043437.xml
2022-05-18T04:34:44.8304772Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:44.8319366Z 
2022-05-18T04:34:44.8319608Z Running tests...
2022-05-18T04:34:44.8320037Z ----------------------------------------------------------------------
2022-05-18T04:34:46.4499019Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:46.4860261Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14158
2022-05-18T04:34:46.4970837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14159
2022-05-18T04:34:47.6757568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:47.6964837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:47.6965652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:47.7061545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:47.7068138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:47.7979913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:49.0288216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj42y1njx
2022-05-18T04:34:49.0288836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj42y1njx/_remote_module_non_scriptable.py
2022-05-18T04:34:49.0709939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4vukk116
2022-05-18T04:34:49.0712102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4vukk116/_remote_module_non_scriptable.py
2022-05-18T04:34:50.5067482Z ok (5.674s)
2022-05-18T04:34:50.5067837Z 
2022-05-18T04:34:50.5068351Z ----------------------------------------------------------------------
2022-05-18T04:34:50.5068694Z Ran 1 test in 5.675s
2022-05-18T04:34:50.5068839Z 
2022-05-18T04:34:50.5068932Z OK
2022-05-18T04:34:50.5069066Z 
2022-05-18T04:34:50.5069203Z Generating XML reports...
2022-05-18T04:34:50.5125611Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043444.xml
2022-05-18T04:34:51.9436690Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:51.9451723Z 
2022-05-18T04:34:51.9451983Z Running tests...
2022-05-18T04:34:51.9452621Z ----------------------------------------------------------------------
2022-05-18T04:34:53.5962746Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:53.6330677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14281
2022-05-18T04:34:53.6438128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14282
2022-05-18T04:34:54.7897996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:54.8008743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:54.8009812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:54.8100445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:54.8107049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:54.9024315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:56.1126791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxw6fn6gj
2022-05-18T04:34:56.1127452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxw6fn6gj/_remote_module_non_scriptable.py
2022-05-18T04:34:56.2185173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0imra4o
2022-05-18T04:34:56.2186537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0imra4o/_remote_module_non_scriptable.py
2022-05-18T04:34:57.6533674Z ok (5.708s)
2022-05-18T04:34:57.6534555Z 
2022-05-18T04:34:57.6535220Z ----------------------------------------------------------------------
2022-05-18T04:34:57.6535621Z Ran 1 test in 5.708s
2022-05-18T04:34:57.6535786Z 
2022-05-18T04:34:57.6535880Z OK
2022-05-18T04:34:57.6536015Z 
2022-05-18T04:34:57.6536148Z Generating XML reports...
2022-05-18T04:34:57.6593944Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043451.xml
2022-05-18T04:34:59.1076417Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:59.1091556Z 
2022-05-18T04:34:59.1092016Z Running tests...
2022-05-18T04:34:59.1092501Z ----------------------------------------------------------------------
2022-05-18T04:35:00.7428154Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:00.7791336Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14404
2022-05-18T04:35:00.7896720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14405
2022-05-18T04:35:01.9809032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:01.9988242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:01.9989074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:02.0010666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:02.0017717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:02.1004020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:03.3044953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphlqoxi2d
2022-05-18T04:35:03.3045548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphlqoxi2d/_remote_module_non_scriptable.py
2022-05-18T04:35:03.4080086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bq8xtu2
2022-05-18T04:35:03.4081140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bq8xtu2/_remote_module_non_scriptable.py
2022-05-18T04:35:04.7494824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:35:04.7495407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:35:05.0998018Z ok (5.990s)
2022-05-18T04:35:05.0998406Z 
2022-05-18T04:35:05.0999060Z ----------------------------------------------------------------------
2022-05-18T04:35:05.0999653Z Ran 1 test in 5.991s
2022-05-18T04:35:05.0999969Z 
2022-05-18T04:35:05.1000139Z OK
2022-05-18T04:35:05.1000631Z 
2022-05-18T04:35:05.1001041Z Generating XML reports...
2022-05-18T04:35:05.1057967Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043459.xml
2022-05-18T04:35:06.5387003Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:06.5402214Z 
2022-05-18T04:35:06.5402449Z Running tests...
2022-05-18T04:35:06.5403164Z ----------------------------------------------------------------------
2022-05-18T04:35:08.1891303Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:08.2011187Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.661s)
2022-05-18T04:35:08.2011858Z 
2022-05-18T04:35:08.2012122Z ----------------------------------------------------------------------
2022-05-18T04:35:08.2012451Z Ran 1 test in 1.661s
2022-05-18T04:35:08.2012614Z 
2022-05-18T04:35:08.2012721Z OK (skipped=1)
2022-05-18T04:35:08.2012875Z 
2022-05-18T04:35:08.2013002Z Generating XML reports...
2022-05-18T04:35:08.2051994Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043506.xml
2022-05-18T04:35:09.6000418Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:09.6016045Z 
2022-05-18T04:35:09.6016280Z Running tests...
2022-05-18T04:35:09.6016767Z ----------------------------------------------------------------------
2022-05-18T04:35:11.2766269Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:11.3128938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14567
2022-05-18T04:35:11.3235784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14568
2022-05-18T04:35:12.5000440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:12.5050470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:12.5051263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:12.5101747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:12.5108625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:12.6065217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:13.8337610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohjih5vm
2022-05-18T04:35:13.8338204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohjih5vm/_remote_module_non_scriptable.py
2022-05-18T04:35:13.9136621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fq3zqkl
2022-05-18T04:35:13.9137968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fq3zqkl/_remote_module_non_scriptable.py
2022-05-18T04:35:14.2723172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:35:14.2723787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:35:14.2958614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:35:14.2959411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:35:14.3052214Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp7fq3zqkl'>
2022-05-18T04:35:14.3052828Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:35:14.6318901Z ok (5.030s)
2022-05-18T04:35:14.6319240Z 
2022-05-18T04:35:14.6319781Z ----------------------------------------------------------------------
2022-05-18T04:35:14.6320174Z Ran 1 test in 5.030s
2022-05-18T04:35:14.6320342Z 
2022-05-18T04:35:14.6320437Z OK
2022-05-18T04:35:14.6320557Z 
2022-05-18T04:35:14.6320686Z Generating XML reports...
2022-05-18T04:35:14.6377488Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043509.xml
2022-05-18T04:35:16.0515804Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:16.0529869Z 
2022-05-18T04:35:16.0530280Z Running tests...
2022-05-18T04:35:16.0531024Z ----------------------------------------------------------------------
2022-05-18T04:35:17.6697785Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:17.7062229Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14693
2022-05-18T04:35:17.7172729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14694
2022-05-18T04:35:18.8631173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:18.8969981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:18.8971271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:18.9035624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:18.9042085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:18.9985860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:20.2222904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoan2hu0i
2022-05-18T04:35:20.2223535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoan2hu0i/_remote_module_non_scriptable.py
2022-05-18T04:35:20.3037224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejljgjef
2022-05-18T04:35:20.3038119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejljgjef/_remote_module_non_scriptable.py
2022-05-18T04:35:21.9267837Z ok (5.873s)
2022-05-18T04:35:21.9268076Z 
2022-05-18T04:35:21.9268453Z ----------------------------------------------------------------------
2022-05-18T04:35:21.9268792Z Ran 1 test in 5.874s
2022-05-18T04:35:21.9268961Z 
2022-05-18T04:35:21.9269036Z OK
2022-05-18T04:35:21.9269170Z 
2022-05-18T04:35:21.9269304Z Generating XML reports...
2022-05-18T04:35:21.9325167Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043516.xml
2022-05-18T04:35:23.3471284Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:23.3485461Z 
2022-05-18T04:35:23.3485798Z Running tests...
2022-05-18T04:35:23.3486528Z ----------------------------------------------------------------------
2022-05-18T04:35:24.9824830Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:25.0189841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14820
2022-05-18T04:35:25.0297954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14821
2022-05-18T04:35:26.2064685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:26.2235485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:26.2236277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:26.2267402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:26.2274365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:26.2277368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:35:26.3248620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:26.3253146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:35:26.3253849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:35:26.3295201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:35:26.6348634Z ok (3.286s)
2022-05-18T04:35:26.6348851Z 
2022-05-18T04:35:26.6349242Z ----------------------------------------------------------------------
2022-05-18T04:35:26.6349589Z Ran 1 test in 3.286s
2022-05-18T04:35:26.6349754Z 
2022-05-18T04:35:26.6349847Z OK
2022-05-18T04:35:26.6349981Z 
2022-05-18T04:35:26.6350112Z Generating XML reports...
2022-05-18T04:35:26.6406100Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043523.xml
2022-05-18T04:35:28.0272538Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:28.0287681Z 
2022-05-18T04:35:28.0288132Z Running tests...
2022-05-18T04:35:28.0288597Z ----------------------------------------------------------------------
2022-05-18T04:35:29.6322264Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:29.6684429Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14937
2022-05-18T04:35:29.6791829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14938
2022-05-18T04:35:30.8301610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:30.8380957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:30.8381762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:30.8402625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:30.8409346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:30.8412824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:35:30.9392391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:30.9396115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:35:30.9397179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:35:30.9426732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:35:31.1841110Z ok (3.155s)
2022-05-18T04:35:31.1841326Z 
2022-05-18T04:35:31.1841704Z ----------------------------------------------------------------------
2022-05-18T04:35:31.1842044Z Ran 1 test in 3.155s
2022-05-18T04:35:31.1842212Z 
2022-05-18T04:35:31.1842312Z OK
2022-05-18T04:35:31.1843083Z 
2022-05-18T04:35:31.1843355Z Generating XML reports...
2022-05-18T04:35:31.1901789Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043528.xml
2022-05-18T04:35:32.6348607Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:32.6363891Z 
2022-05-18T04:35:32.6364042Z Running tests...
2022-05-18T04:35:32.6364483Z ----------------------------------------------------------------------
2022-05-18T04:35:34.2772288Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:34.3144996Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15054
2022-05-18T04:35:34.3250332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15055
2022-05-18T04:35:35.4832067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:35.5028903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:35.5029714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:35.5034552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:35.5041616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:35.6043701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:36.8091080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7uhpihl
2022-05-18T04:35:36.8091706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7uhpihl/_remote_module_non_scriptable.py
2022-05-18T04:35:36.8646507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7zxxzf8
2022-05-18T04:35:36.8647485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7zxxzf8/_remote_module_non_scriptable.py
2022-05-18T04:35:37.2339997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:35:37.2340593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:35:37.2406915Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:35:37.2410187Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:35:37.5330615Z ok (4.896s)
2022-05-18T04:35:37.5331133Z 
2022-05-18T04:35:37.5331801Z ----------------------------------------------------------------------
2022-05-18T04:35:37.5332159Z Ran 1 test in 4.897s
2022-05-18T04:35:37.5332325Z 
2022-05-18T04:35:37.5332434Z OK
2022-05-18T04:35:37.5332558Z 
2022-05-18T04:35:37.5332698Z Generating XML reports...
2022-05-18T04:35:37.5389546Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043532.xml
2022-05-18T04:35:38.9778254Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:38.9794299Z 
2022-05-18T04:35:38.9794454Z Running tests...
2022-05-18T04:35:38.9795219Z ----------------------------------------------------------------------
2022-05-18T04:35:40.6356795Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:40.6725309Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15180
2022-05-18T04:35:40.6834488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15181
2022-05-18T04:35:41.8332198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:41.8571418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:41.8572473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:41.8635999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:41.8643220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:41.9586305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:43.1930573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmyyjksm_
2022-05-18T04:35:43.1931490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmyyjksm_/_remote_module_non_scriptable.py
2022-05-18T04:35:43.2443697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp917ajj5h
2022-05-18T04:35:43.2444732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp917ajj5h/_remote_module_non_scriptable.py
2022-05-18T04:35:43.6013846Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:35:43.6173869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:35:43.6174391Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:35:43.8917555Z ok (4.912s)
2022-05-18T04:35:43.8917771Z 
2022-05-18T04:35:43.8918161Z ----------------------------------------------------------------------
2022-05-18T04:35:43.8918486Z Ran 1 test in 4.912s
2022-05-18T04:35:43.8918651Z 
2022-05-18T04:35:43.8918745Z OK
2022-05-18T04:35:43.8918878Z 
2022-05-18T04:35:43.8919011Z Generating XML reports...
2022-05-18T04:35:43.8976270Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043538.xml
2022-05-18T04:35:45.3349318Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:45.3364388Z 
2022-05-18T04:35:45.3364729Z Running tests...
2022-05-18T04:35:45.3365146Z ----------------------------------------------------------------------
2022-05-18T04:35:46.9831526Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:47.0204991Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15306
2022-05-18T04:35:47.0315107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15307
2022-05-18T04:35:48.2396957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:48.2588432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:48.2589229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:48.2599570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:48.2606201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:48.3600267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:48.5366293Z ok (3.200s)
2022-05-18T04:35:48.5366694Z 
2022-05-18T04:35:48.5367366Z ----------------------------------------------------------------------
2022-05-18T04:35:48.5368355Z Ran 1 test in 3.200s
2022-05-18T04:35:48.5368659Z 
2022-05-18T04:35:48.5368824Z OK
2022-05-18T04:35:48.5369059Z 
2022-05-18T04:35:48.5369306Z Generating XML reports...
2022-05-18T04:35:48.5426670Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043545.xml
2022-05-18T04:35:49.9588864Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:49.9603770Z 
2022-05-18T04:35:49.9604257Z Running tests...
2022-05-18T04:35:49.9604754Z ----------------------------------------------------------------------
2022-05-18T04:35:49.9624543Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:35:49.9624849Z 
2022-05-18T04:35:49.9625130Z ----------------------------------------------------------------------
2022-05-18T04:35:49.9625450Z Ran 1 test in 0.002s
2022-05-18T04:35:49.9625627Z 
2022-05-18T04:35:49.9625734Z OK (skipped=1)
2022-05-18T04:35:49.9625888Z 
2022-05-18T04:35:49.9625995Z Generating XML reports...
2022-05-18T04:35:49.9668721Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043549.xml
2022-05-18T04:35:51.2403957Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:51.2418934Z 
2022-05-18T04:35:51.2419482Z Running tests...
2022-05-18T04:35:51.2419988Z ----------------------------------------------------------------------
2022-05-18T04:35:51.2447780Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.003s)
2022-05-18T04:35:51.2448097Z 
2022-05-18T04:35:51.2448409Z ----------------------------------------------------------------------
2022-05-18T04:35:51.2448737Z Ran 1 test in 0.003s
2022-05-18T04:35:51.2448899Z 
2022-05-18T04:35:51.2448989Z OK (skipped=1)
2022-05-18T04:35:51.2449154Z 
2022-05-18T04:35:51.2449282Z Generating XML reports...
2022-05-18T04:35:51.2492677Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043551.xml
2022-05-18T04:35:52.5185077Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:52.5200692Z 
2022-05-18T04:35:52.5201160Z Running tests...
2022-05-18T04:35:52.5201671Z ----------------------------------------------------------------------
2022-05-18T04:35:54.1812441Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:54.2173590Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15489
2022-05-18T04:35:54.2280675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15490
2022-05-18T04:35:55.4027128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:55.4081729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:55.4082576Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:55.4128239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:55.4134986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:55.5097545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:59.5399315Z ok (7.019s)
2022-05-18T04:35:59.5399569Z 
2022-05-18T04:35:59.5399965Z ----------------------------------------------------------------------
2022-05-18T04:35:59.5400322Z Ran 1 test in 7.020s
2022-05-18T04:35:59.5400486Z 
2022-05-18T04:35:59.5400585Z OK
2022-05-18T04:35:59.5400702Z 
2022-05-18T04:35:59.5400838Z Generating XML reports...
2022-05-18T04:35:59.5458956Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043552.xml
2022-05-18T04:36:00.9802575Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:00.9817771Z 
2022-05-18T04:36:00.9818272Z Running tests...
2022-05-18T04:36:00.9818780Z ----------------------------------------------------------------------
2022-05-18T04:36:00.9838034Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:36:00.9838452Z 
2022-05-18T04:36:00.9838926Z ----------------------------------------------------------------------
2022-05-18T04:36:00.9839267Z Ran 1 test in 0.002s
2022-05-18T04:36:00.9839428Z 
2022-05-18T04:36:00.9839537Z OK (skipped=1)
2022-05-18T04:36:00.9839709Z 
2022-05-18T04:36:00.9839820Z Generating XML reports...
2022-05-18T04:36:00.9882637Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043600.xml
2022-05-18T04:36:02.2596478Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:02.2611872Z 
2022-05-18T04:36:02.2612112Z Running tests...
2022-05-18T04:36:02.2612541Z ----------------------------------------------------------------------
2022-05-18T04:36:02.2632804Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:36:02.2633117Z 
2022-05-18T04:36:02.2633406Z ----------------------------------------------------------------------
2022-05-18T04:36:02.2633731Z Ran 1 test in 0.002s
2022-05-18T04:36:02.2633892Z 
2022-05-18T04:36:02.2633983Z OK (skipped=1)
2022-05-18T04:36:02.2634135Z 
2022-05-18T04:36:02.2634263Z Generating XML reports...
2022-05-18T04:36:02.2677401Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043602.xml
2022-05-18T04:36:03.5254608Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:03.5270869Z 
2022-05-18T04:36:03.5271464Z Running tests...
2022-05-18T04:36:03.5271964Z ----------------------------------------------------------------------
2022-05-18T04:36:05.1410967Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:05.1774701Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15687
2022-05-18T04:36:05.1883809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15688
2022-05-18T04:36:06.3426400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:06.3681416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:06.3682238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:06.3730330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:06.3737654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:06.4696661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:09.1988032Z ok (5.671s)
2022-05-18T04:36:09.1988251Z 
2022-05-18T04:36:09.1988632Z ----------------------------------------------------------------------
2022-05-18T04:36:09.1988970Z Ran 1 test in 5.672s
2022-05-18T04:36:09.1989117Z 
2022-05-18T04:36:09.1989210Z OK
2022-05-18T04:36:09.1989343Z 
2022-05-18T04:36:09.1991951Z Generating XML reports...
2022-05-18T04:36:09.2045990Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043603.xml
2022-05-18T04:36:10.6392777Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:10.6407794Z 
2022-05-18T04:36:10.6408329Z Running tests...
2022-05-18T04:36:10.6408843Z ----------------------------------------------------------------------
2022-05-18T04:36:12.2822007Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:12.3192518Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15811
2022-05-18T04:36:12.3300142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15812
2022-05-18T04:36:13.4772973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:13.4888453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:13.4889268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:13.4975233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:13.4982049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:13.5903115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:13.6103757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:36:13.6104271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:36:13.6104974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:36:13.6105656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:36:16.0815715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:36:16.0816277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:36:16.0817080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:36:16.0817756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:36:16.1272630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:36:16.1273161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:36:16.1273896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:36:16.1274774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:36:16.5401445Z ok (5.899s)
2022-05-18T04:36:16.5401665Z 
2022-05-18T04:36:16.5402068Z ----------------------------------------------------------------------
2022-05-18T04:36:16.5402406Z Ran 1 test in 5.899s
2022-05-18T04:36:16.5402570Z 
2022-05-18T04:36:16.5402663Z OK
2022-05-18T04:36:16.5402797Z 
2022-05-18T04:36:16.5402914Z Generating XML reports...
2022-05-18T04:36:16.5459565Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043610.xml
2022-05-18T04:36:17.9684741Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:17.9699374Z 
2022-05-18T04:36:17.9699574Z Running tests...
2022-05-18T04:36:17.9700014Z ----------------------------------------------------------------------
2022-05-18T04:36:19.5843923Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:19.6205915Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15974
2022-05-18T04:36:19.6317122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15975
2022-05-18T04:36:20.7121836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:20.7744776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:20.7745586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:20.7830172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:20.7836892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:20.7839792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:36:20.8756010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:20.8761111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:36:20.8761798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:36:20.8857653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:36:21.1369040Z ok (3.167s)
2022-05-18T04:36:21.1369227Z 
2022-05-18T04:36:21.1369955Z ----------------------------------------------------------------------
2022-05-18T04:36:21.1408104Z Ran 1 test in 3.167s
2022-05-18T04:36:21.1408333Z 
2022-05-18T04:36:21.1408436Z OK
2022-05-18T04:36:21.1408581Z 
2022-05-18T04:36:21.1408717Z Generating XML reports...
2022-05-18T04:36:21.1427415Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043617.xml
2022-05-18T04:36:22.5646777Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:22.5661482Z 
2022-05-18T04:36:22.5661920Z Running tests...
2022-05-18T04:36:22.5662425Z ----------------------------------------------------------------------
2022-05-18T04:36:24.1936462Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:24.2301137Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16091
2022-05-18T04:36:24.2406746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16092
2022-05-18T04:36:25.3977163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:25.4398989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:25.4399988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:25.4485144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:25.4491706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:25.5413431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:28.0498692Z ok (5.483s)
2022-05-18T04:36:28.0498910Z 
2022-05-18T04:36:28.0499288Z ----------------------------------------------------------------------
2022-05-18T04:36:28.0499629Z Ran 1 test in 5.484s
2022-05-18T04:36:28.0499797Z 
2022-05-18T04:36:28.0499893Z OK
2022-05-18T04:36:28.0500028Z 
2022-05-18T04:36:28.0500161Z Generating XML reports...
2022-05-18T04:36:28.0556055Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043622.xml
2022-05-18T04:36:29.4872047Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:29.4887338Z 
2022-05-18T04:36:29.4887784Z Running tests...
2022-05-18T04:36:29.4888249Z ----------------------------------------------------------------------
2022-05-18T04:36:31.1408344Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:31.1777293Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16214
2022-05-18T04:36:31.1884673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16215
2022-05-18T04:36:32.3626683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:32.3692326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:32.3693199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:32.3728138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:32.3735120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:32.4703588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:32.8938453Z ok (3.405s)
2022-05-18T04:36:32.8938700Z 
2022-05-18T04:36:32.8939095Z ----------------------------------------------------------------------
2022-05-18T04:36:32.8939418Z Ran 1 test in 3.405s
2022-05-18T04:36:32.8939588Z 
2022-05-18T04:36:32.8939686Z OK
2022-05-18T04:36:32.8939824Z 
2022-05-18T04:36:32.8939957Z Generating XML reports...
2022-05-18T04:36:32.8996354Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043629.xml
2022-05-18T04:36:34.3295846Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:34.3311040Z 
2022-05-18T04:36:34.3311281Z Running tests...
2022-05-18T04:36:34.3311699Z ----------------------------------------------------------------------
2022-05-18T04:36:35.9601529Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:35.9969659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16327
2022-05-18T04:36:36.0079068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16328
2022-05-18T04:36:37.1522962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:37.1587917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:37.1588715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:37.1624403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:37.1630909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:37.1634026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:36:37.2601655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:37.2605122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:36:37.2605824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:36:37.2653369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:36:37.5131158Z ok (3.182s)
2022-05-18T04:36:37.5131555Z 
2022-05-18T04:36:37.5132251Z ----------------------------------------------------------------------
2022-05-18T04:36:37.5132870Z Ran 1 test in 3.182s
2022-05-18T04:36:37.5133180Z 
2022-05-18T04:36:37.5133718Z OK
2022-05-18T04:36:37.5133973Z 
2022-05-18T04:36:37.5134212Z Generating XML reports...
2022-05-18T04:36:37.5191682Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043634.xml
2022-05-18T04:36:38.9622531Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:38.9637809Z 
2022-05-18T04:36:38.9637978Z Running tests...
2022-05-18T04:36:38.9638787Z ----------------------------------------------------------------------
2022-05-18T04:36:40.6187346Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:40.6549315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16444
2022-05-18T04:36:40.6657440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16445
2022-05-18T04:36:41.8148212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:41.8401040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:41.8401849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:41.8451963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:41.8459021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:41.8462303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:36:41.9414487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:41.9417721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:36:41.9418433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:36:41.9482650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:36:42.1707519Z ok (3.207s)
2022-05-18T04:36:42.1707725Z 
2022-05-18T04:36:42.1708083Z ----------------------------------------------------------------------
2022-05-18T04:36:42.1708416Z Ran 1 test in 3.207s
2022-05-18T04:36:42.1708589Z 
2022-05-18T04:36:42.1708687Z OK
2022-05-18T04:36:42.1708826Z 
2022-05-18T04:36:42.1708960Z Generating XML reports...
2022-05-18T04:36:42.1765975Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043638.xml
2022-05-18T04:36:43.5944478Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:43.5960358Z 
2022-05-18T04:36:43.5960503Z Running tests...
2022-05-18T04:36:43.5961183Z ----------------------------------------------------------------------
2022-05-18T04:36:45.2369981Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:45.2733838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16561
2022-05-18T04:36:45.2839050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16562
2022-05-18T04:36:46.4480916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:46.4572499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:46.4573286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:46.4582220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:46.4589591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:46.5587799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:47.7858295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4dnmkihb
2022-05-18T04:36:47.7858904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4dnmkihb/_remote_module_non_scriptable.py
2022-05-18T04:36:47.8421242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_lo_4ed
2022-05-18T04:36:47.8422530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_lo_4ed/_remote_module_non_scriptable.py
2022-05-18T04:36:48.4918500Z ok (4.895s)
2022-05-18T04:36:48.4918714Z 
2022-05-18T04:36:48.4919095Z ----------------------------------------------------------------------
2022-05-18T04:36:48.4919455Z Ran 1 test in 4.896s
2022-05-18T04:36:48.4919623Z 
2022-05-18T04:36:48.4919718Z OK
2022-05-18T04:36:48.4919856Z 
2022-05-18T04:36:48.4919991Z Generating XML reports...
2022-05-18T04:36:48.4977085Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043643.xml
2022-05-18T04:36:49.9366938Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:49.9382568Z 
2022-05-18T04:36:49.9382819Z Running tests...
2022-05-18T04:36:49.9383259Z ----------------------------------------------------------------------
2022-05-18T04:36:49.9410886Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.003s)
2022-05-18T04:36:49.9411178Z 
2022-05-18T04:36:49.9411526Z ----------------------------------------------------------------------
2022-05-18T04:36:49.9412049Z Ran 1 test in 0.003s
2022-05-18T04:36:49.9412214Z 
2022-05-18T04:36:49.9412327Z OK (skipped=1)
2022-05-18T04:36:49.9412482Z 
2022-05-18T04:36:49.9412609Z Generating XML reports...
2022-05-18T04:36:49.9455842Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043649.xml
2022-05-18T04:36:51.2181789Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:51.2197176Z 
2022-05-18T04:36:51.2197414Z Running tests...
2022-05-18T04:36:51.2197852Z ----------------------------------------------------------------------
2022-05-18T04:36:51.2217406Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-05-18T04:36:51.2217699Z 
2022-05-18T04:36:51.2217978Z ----------------------------------------------------------------------
2022-05-18T04:36:51.2218287Z Ran 1 test in 0.002s
2022-05-18T04:36:51.2218453Z 
2022-05-18T04:36:51.2218566Z OK (skipped=1)
2022-05-18T04:36:51.2218722Z 
2022-05-18T04:36:51.2218851Z Generating XML reports...
2022-05-18T04:36:51.2261799Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043651.xml
2022-05-18T04:36:52.4567668Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:52.4583568Z 
2022-05-18T04:36:52.4583938Z Running tests...
2022-05-18T04:36:52.4584434Z ----------------------------------------------------------------------
2022-05-18T04:36:52.4603755Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-05-18T04:36:52.4604067Z 
2022-05-18T04:36:52.4604331Z ----------------------------------------------------------------------
2022-05-18T04:36:52.4604667Z Ran 1 test in 0.002s
2022-05-18T04:36:52.4604837Z 
2022-05-18T04:36:52.4604947Z OK (skipped=1)
2022-05-18T04:36:52.4605105Z 
2022-05-18T04:36:52.4605231Z Generating XML reports...
2022-05-18T04:36:52.4647764Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043652.xml
2022-05-18T04:36:53.7346045Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:53.7361633Z 
2022-05-18T04:36:53.7362057Z Running tests...
2022-05-18T04:36:53.7362480Z ----------------------------------------------------------------------
2022-05-18T04:36:53.7383513Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-05-18T04:36:53.7383825Z 
2022-05-18T04:36:53.7384101Z ----------------------------------------------------------------------
2022-05-18T04:36:53.7384430Z Ran 1 test in 0.002s
2022-05-18T04:36:53.7384595Z 
2022-05-18T04:36:53.7384688Z OK (skipped=1)
2022-05-18T04:36:53.7384845Z 
2022-05-18T04:36:53.7384972Z Generating XML reports...
2022-05-18T04:36:53.7427362Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043653.xml
2022-05-18T04:36:55.0149502Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:55.0164684Z 
2022-05-18T04:36:55.0164801Z Running tests...
2022-05-18T04:36:55.0165506Z ----------------------------------------------------------------------
2022-05-18T04:36:55.0187101Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T04:36:55.0187466Z 
2022-05-18T04:36:55.0187748Z ----------------------------------------------------------------------
2022-05-18T04:36:55.0188060Z Ran 1 test in 0.002s
2022-05-18T04:36:55.0188225Z 
2022-05-18T04:36:55.0188341Z OK (skipped=1)
2022-05-18T04:36:55.0188501Z 
2022-05-18T04:36:55.0188627Z Generating XML reports...
2022-05-18T04:36:55.0231332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043655.xml
2022-05-18T04:36:56.2963441Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:56.2979312Z 
2022-05-18T04:36:56.2979733Z Running tests...
2022-05-18T04:36:56.2980218Z ----------------------------------------------------------------------
2022-05-18T04:36:56.3000961Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T04:36:56.3001341Z 
2022-05-18T04:36:56.3001624Z ----------------------------------------------------------------------
2022-05-18T04:36:56.3001958Z Ran 1 test in 0.002s
2022-05-18T04:36:56.3002103Z 
2022-05-18T04:36:56.3002215Z OK (skipped=1)
2022-05-18T04:36:56.3002369Z 
2022-05-18T04:36:56.3002496Z Generating XML reports...
2022-05-18T04:36:56.3044326Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043656.xml
2022-05-18T04:36:57.5803297Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:57.5819804Z 
2022-05-18T04:36:57.5819969Z Running tests...
2022-05-18T04:36:57.5820420Z ----------------------------------------------------------------------
2022-05-18T04:36:57.5845794Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:36:57.5846153Z 
2022-05-18T04:36:57.5846405Z ----------------------------------------------------------------------
2022-05-18T04:36:57.5846737Z Ran 1 test in 0.003s
2022-05-18T04:36:57.5846903Z 
2022-05-18T04:36:57.5847013Z OK (skipped=1)
2022-05-18T04:36:57.5847175Z 
2022-05-18T04:36:57.5847300Z Generating XML reports...
2022-05-18T04:36:57.5889680Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043657.xml
2022-05-18T04:36:58.8601848Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:58.8617878Z 
2022-05-18T04:36:58.8618143Z Running tests...
2022-05-18T04:36:58.8618598Z ----------------------------------------------------------------------
2022-05-18T04:36:58.8648147Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.003s)
2022-05-18T04:36:58.8648775Z 
2022-05-18T04:36:58.8649057Z ----------------------------------------------------------------------
2022-05-18T04:36:58.8649397Z Ran 1 test in 0.003s
2022-05-18T04:36:58.8649901Z 
2022-05-18T04:36:58.8650024Z OK (skipped=1)
2022-05-18T04:36:58.8650184Z 
2022-05-18T04:36:58.8650310Z Generating XML reports...
2022-05-18T04:36:58.8692026Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043658.xml
2022-05-18T04:37:00.1243979Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:00.1260086Z 
2022-05-18T04:37:00.1260356Z Running tests...
2022-05-18T04:37:00.1261082Z ----------------------------------------------------------------------
2022-05-18T04:37:00.1283848Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:37:00.1284211Z 
2022-05-18T04:37:00.1284629Z ----------------------------------------------------------------------
2022-05-18T04:37:00.1285088Z Ran 1 test in 0.002s
2022-05-18T04:37:00.1285253Z 
2022-05-18T04:37:00.1285363Z OK (skipped=1)
2022-05-18T04:37:00.1285499Z 
2022-05-18T04:37:00.1285624Z Generating XML reports...
2022-05-18T04:37:00.1327587Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043700.xml
2022-05-18T04:37:01.3818709Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:01.3833890Z 
2022-05-18T04:37:01.3834356Z Running tests...
2022-05-18T04:37:01.3834840Z ----------------------------------------------------------------------
2022-05-18T04:37:01.3857911Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:37:01.3858427Z 
2022-05-18T04:37:01.3858712Z ----------------------------------------------------------------------
2022-05-18T04:37:01.3859054Z Ran 1 test in 0.002s
2022-05-18T04:37:01.3859200Z 
2022-05-18T04:37:01.3859312Z OK (skipped=1)
2022-05-18T04:37:01.3859467Z 
2022-05-18T04:37:01.3859598Z Generating XML reports...
2022-05-18T04:37:01.3900915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043701.xml
2022-05-18T04:37:02.6615206Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:02.6630259Z 
2022-05-18T04:37:02.6630695Z Running tests...
2022-05-18T04:37:02.6631196Z ----------------------------------------------------------------------
2022-05-18T04:37:02.6654101Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:37:02.6654469Z 
2022-05-18T04:37:02.6654743Z ----------------------------------------------------------------------
2022-05-18T04:37:02.6655086Z Ran 1 test in 0.002s
2022-05-18T04:37:02.6655250Z 
2022-05-18T04:37:02.6655362Z OK (skipped=1)
2022-05-18T04:37:02.6655519Z 
2022-05-18T04:37:02.6655623Z Generating XML reports...
2022-05-18T04:37:02.6697503Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043702.xml
2022-05-18T04:37:03.9372001Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:03.9387620Z 
2022-05-18T04:37:03.9387939Z Running tests...
2022-05-18T04:37:03.9388354Z ----------------------------------------------------------------------
2022-05-18T04:37:05.6164653Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:05.6531958Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17072
2022-05-18T04:37:05.6640356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17073
2022-05-18T04:37:06.8024051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:06.8110829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:06.8111637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:06.8124973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:06.8132319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:06.9126388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:08.5716037Z ok (4.632s)
2022-05-18T04:37:08.5716278Z 
2022-05-18T04:37:08.5716659Z ----------------------------------------------------------------------
2022-05-18T04:37:08.5717005Z Ran 1 test in 4.633s
2022-05-18T04:37:08.5717185Z 
2022-05-18T04:37:08.5717259Z OK
2022-05-18T04:37:08.5717395Z 
2022-05-18T04:37:08.5718872Z Generating XML reports...
2022-05-18T04:37:08.5774445Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043703.xml
2022-05-18T04:37:10.0062625Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:10.0077352Z 
2022-05-18T04:37:10.0077769Z Running tests...
2022-05-18T04:37:10.0078230Z ----------------------------------------------------------------------
2022-05-18T04:37:11.6248192Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:11.6610356Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17194
2022-05-18T04:37:11.6719692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17195
2022-05-18T04:37:12.8344693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:12.8565340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:12.8566361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:12.8649475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:12.8656314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:12.9580801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:14.5793913Z ok (4.571s)
2022-05-18T04:37:14.5794322Z 
2022-05-18T04:37:14.5795490Z ----------------------------------------------------------------------
2022-05-18T04:37:14.5795902Z Ran 1 test in 4.572s
2022-05-18T04:37:14.5796070Z 
2022-05-18T04:37:14.5796164Z OK
2022-05-18T04:37:14.5796289Z 
2022-05-18T04:37:14.5796424Z Generating XML reports...
2022-05-18T04:37:14.5851731Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043710.xml
2022-05-18T04:37:16.0455839Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:16.0471010Z 
2022-05-18T04:37:16.0471220Z Running tests...
2022-05-18T04:37:16.0471665Z ----------------------------------------------------------------------
2022-05-18T04:37:17.7059829Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:17.7427760Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17316
2022-05-18T04:37:17.7536368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17317
2022-05-18T04:37:18.9413303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:18.9622068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:18.9622861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:18.9719722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:18.9726554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:19.0636970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:21.7633131Z ok (5.716s)
2022-05-18T04:37:21.7633388Z 
2022-05-18T04:37:21.7633791Z ----------------------------------------------------------------------
2022-05-18T04:37:21.7634129Z Ran 1 test in 5.716s
2022-05-18T04:37:21.7634298Z 
2022-05-18T04:37:21.7634400Z OK
2022-05-18T04:37:21.7634516Z 
2022-05-18T04:37:21.7634652Z Generating XML reports...
2022-05-18T04:37:21.7692193Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043716.xml
2022-05-18T04:37:23.2031047Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:23.2045878Z 
2022-05-18T04:37:23.2046125Z Running tests...
2022-05-18T04:37:23.2046545Z ----------------------------------------------------------------------
2022-05-18T04:37:24.8526148Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:24.8890191Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17439
2022-05-18T04:37:24.8997167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17440
2022-05-18T04:37:26.0752337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:26.0824715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:26.0825521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:26.0853563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:26.0860621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:26.1839048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:27.9075998Z ok (4.703s)
2022-05-18T04:37:27.9076242Z 
2022-05-18T04:37:27.9077370Z ----------------------------------------------------------------------
2022-05-18T04:37:27.9077750Z Ran 1 test in 4.703s
2022-05-18T04:37:27.9078200Z 
2022-05-18T04:37:27.9078291Z OK
2022-05-18T04:37:27.9078436Z 
2022-05-18T04:37:27.9078570Z Generating XML reports...
2022-05-18T04:37:27.9135987Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043723.xml
2022-05-18T04:37:29.3252404Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:29.3267717Z 
2022-05-18T04:37:29.3267977Z Running tests...
2022-05-18T04:37:29.3268392Z ----------------------------------------------------------------------
2022-05-18T04:37:30.9361287Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:30.9723037Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17561
2022-05-18T04:37:30.9832186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17562
2022-05-18T04:37:32.1398226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:32.1575628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:32.1576692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:32.1601005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:32.1607977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:32.2586996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:34.5919035Z ok (5.265s)
2022-05-18T04:37:34.5919272Z 
2022-05-18T04:37:34.5919673Z ----------------------------------------------------------------------
2022-05-18T04:37:34.5920010Z Ran 1 test in 5.265s
2022-05-18T04:37:34.5920175Z 
2022-05-18T04:37:34.5920271Z OK
2022-05-18T04:37:34.5920400Z 
2022-05-18T04:37:34.5920539Z Generating XML reports...
2022-05-18T04:37:34.5977304Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043729.xml
2022-05-18T04:37:36.0317007Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:36.0331845Z 
2022-05-18T04:37:36.0331992Z Running tests...
2022-05-18T04:37:36.0332679Z ----------------------------------------------------------------------
2022-05-18T04:37:36.0356754Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:37:36.0357059Z 
2022-05-18T04:37:36.0357340Z ----------------------------------------------------------------------
2022-05-18T04:37:36.0357670Z Ran 1 test in 0.003s
2022-05-18T04:37:36.0357840Z 
2022-05-18T04:37:36.0357931Z OK (skipped=1)
2022-05-18T04:37:36.0358087Z 
2022-05-18T04:37:36.0358214Z Generating XML reports...
2022-05-18T04:37:36.0401332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043736.xml
2022-05-18T04:37:37.3022745Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:37.3038232Z 
2022-05-18T04:37:37.3038520Z Running tests...
2022-05-18T04:37:37.3038972Z ----------------------------------------------------------------------
2022-05-18T04:37:37.3066904Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-05-18T04:37:37.3067416Z 
2022-05-18T04:37:37.3067855Z ----------------------------------------------------------------------
2022-05-18T04:37:37.3068200Z Ran 1 test in 0.003s
2022-05-18T04:37:37.3068378Z 
2022-05-18T04:37:37.3068492Z OK (skipped=1)
2022-05-18T04:37:37.3068649Z 
2022-05-18T04:37:37.3068779Z Generating XML reports...
2022-05-18T04:37:37.3111385Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043737.xml
2022-05-18T04:37:38.5898252Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:38.5912823Z 
2022-05-18T04:37:38.5913142Z Running tests...
2022-05-18T04:37:38.5913585Z ----------------------------------------------------------------------
2022-05-18T04:37:38.5938153Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:37:38.5938511Z 
2022-05-18T04:37:38.5939090Z ----------------------------------------------------------------------
2022-05-18T04:37:38.5939714Z Ran 1 test in 0.003s
2022-05-18T04:37:38.5939896Z 
2022-05-18T04:37:38.5940047Z OK (skipped=1)
2022-05-18T04:37:38.5940205Z 
2022-05-18T04:37:38.5940313Z Generating XML reports...
2022-05-18T04:37:38.5982401Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043738.xml
2022-05-18T04:37:39.8702476Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:39.8717744Z 
2022-05-18T04:37:39.8718070Z Running tests...
2022-05-18T04:37:39.8718803Z ----------------------------------------------------------------------
2022-05-18T04:37:41.5298805Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:41.5667304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17799
2022-05-18T04:37:41.5775684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17800
2022-05-18T04:37:42.6954821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:42.7218254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:42.7219051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:42.7258267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:42.7265385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:42.8231652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:42.9825094Z ok (3.110s)
2022-05-18T04:37:42.9825311Z 
2022-05-18T04:37:42.9825683Z ----------------------------------------------------------------------
2022-05-18T04:37:42.9826020Z Ran 1 test in 3.111s
2022-05-18T04:37:42.9826191Z 
2022-05-18T04:37:42.9826288Z OK
2022-05-18T04:37:42.9826425Z 
2022-05-18T04:37:42.9826560Z Generating XML reports...
2022-05-18T04:37:42.9882778Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043739.xml
2022-05-18T04:37:44.3784380Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:44.3798726Z 
2022-05-18T04:37:44.3798849Z Running tests...
2022-05-18T04:37:44.3799537Z ----------------------------------------------------------------------
2022-05-18T04:37:45.9931248Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:46.0290685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17912
2022-05-18T04:37:46.0401361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17913
2022-05-18T04:37:47.2243013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:47.2399444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:47.2400237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:47.2445886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:47.2452206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:47.3412505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:47.5450114Z ok (3.165s)
2022-05-18T04:37:47.5450751Z 
2022-05-18T04:37:47.5451196Z ----------------------------------------------------------------------
2022-05-18T04:37:47.5451538Z Ran 1 test in 3.165s
2022-05-18T04:37:47.5451704Z 
2022-05-18T04:37:47.5451780Z OK
2022-05-18T04:37:47.5451920Z 
2022-05-18T04:37:47.5452060Z Generating XML reports...
2022-05-18T04:37:47.5509074Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043744.xml
2022-05-18T04:37:48.9619568Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:48.9636565Z 
2022-05-18T04:37:48.9636997Z Running tests...
2022-05-18T04:37:48.9637485Z ----------------------------------------------------------------------
2022-05-18T04:37:48.9660312Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:37:48.9660965Z 
2022-05-18T04:37:48.9661626Z ----------------------------------------------------------------------
2022-05-18T04:37:48.9662293Z Ran 1 test in 0.002s
2022-05-18T04:37:48.9662596Z 
2022-05-18T04:37:48.9662823Z OK (skipped=1)
2022-05-18T04:37:48.9663118Z 
2022-05-18T04:37:48.9663353Z Generating XML reports...
2022-05-18T04:37:48.9707338Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043748.xml
2022-05-18T04:37:50.2421575Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:50.2436553Z 
2022-05-18T04:37:50.2436874Z Running tests...
2022-05-18T04:37:50.2437311Z ----------------------------------------------------------------------
2022-05-18T04:37:50.2458717Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:37:50.2459081Z 
2022-05-18T04:37:50.2459356Z ----------------------------------------------------------------------
2022-05-18T04:37:50.2459683Z Ran 1 test in 0.002s
2022-05-18T04:37:50.2459846Z 
2022-05-18T04:37:50.2459959Z OK (skipped=1)
2022-05-18T04:37:50.2460116Z 
2022-05-18T04:37:50.2460223Z Generating XML reports...
2022-05-18T04:37:50.2502460Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043750.xml
2022-05-18T04:37:51.5161009Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:51.5176834Z 
2022-05-18T04:37:51.5177323Z Running tests...
2022-05-18T04:37:51.5177863Z ----------------------------------------------------------------------
2022-05-18T04:37:53.1824819Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:53.2195736Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18095
2022-05-18T04:37:53.2305071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18096
2022-05-18T04:37:54.3279897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:54.3769720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:54.3770790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:54.3786141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:54.3793423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:54.4785829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:55.6510521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppp207z8f
2022-05-18T04:37:55.6511401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppp207z8f/_remote_module_non_scriptable.py
2022-05-18T04:37:55.7838760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbma9cqw
2022-05-18T04:37:55.7839714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbma9cqw/_remote_module_non_scriptable.py
2022-05-18T04:37:57.4403497Z ok (5.922s)
2022-05-18T04:37:57.4403807Z 
2022-05-18T04:37:57.4404346Z ----------------------------------------------------------------------
2022-05-18T04:37:57.4404670Z Ran 1 test in 5.923s
2022-05-18T04:37:57.4404834Z 
2022-05-18T04:37:57.4404935Z OK
2022-05-18T04:37:57.4405072Z 
2022-05-18T04:37:57.4405227Z Generating XML reports...
2022-05-18T04:37:57.4461970Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043751.xml
2022-05-18T04:37:58.8762606Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:58.8778788Z 
2022-05-18T04:37:58.8779117Z Running tests...
2022-05-18T04:37:58.8779530Z ----------------------------------------------------------------------
2022-05-18T04:38:00.5184568Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:00.5559899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18222
2022-05-18T04:38:00.5668074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18223
2022-05-18T04:38:01.7139324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:01.7280054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:01.7280830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:01.7341996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:01.7348977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:01.8295782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:03.0258313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqc4ztp35
2022-05-18T04:38:03.0259174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqc4ztp35/_remote_module_non_scriptable.py
2022-05-18T04:38:03.1504636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8uaok4dc
2022-05-18T04:38:03.1506656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8uaok4dc/_remote_module_non_scriptable.py
2022-05-18T04:38:04.8770367Z ok (5.999s)
2022-05-18T04:38:04.8770863Z 
2022-05-18T04:38:04.8771272Z ----------------------------------------------------------------------
2022-05-18T04:38:04.8771592Z Ran 1 test in 5.999s
2022-05-18T04:38:04.8771758Z 
2022-05-18T04:38:04.8771853Z OK
2022-05-18T04:38:04.8771986Z 
2022-05-18T04:38:04.8772122Z Generating XML reports...
2022-05-18T04:38:04.8828324Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043758.xml
2022-05-18T04:38:06.3184330Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:06.3199094Z 
2022-05-18T04:38:06.3199336Z Running tests...
2022-05-18T04:38:06.3199897Z ----------------------------------------------------------------------
2022-05-18T04:38:07.9739597Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:08.0109756Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18349
2022-05-18T04:38:08.0218543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18350
2022-05-18T04:38:09.1390120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:09.1640393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:09.1641196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:09.1693910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:09.1700207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:09.2656301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:11.8310067Z ok (5.511s)
2022-05-18T04:38:11.8310552Z 
2022-05-18T04:38:11.8311180Z ----------------------------------------------------------------------
2022-05-18T04:38:11.8311823Z Ran 1 test in 5.511s
2022-05-18T04:38:11.8311995Z 
2022-05-18T04:38:11.8312100Z OK
2022-05-18T04:38:11.8312237Z 
2022-05-18T04:38:11.8312351Z Generating XML reports...
2022-05-18T04:38:11.8368451Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043806.xml
2022-05-18T04:38:13.2801448Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:13.2816794Z 
2022-05-18T04:38:13.2817051Z Running tests...
2022-05-18T04:38:13.2817475Z ----------------------------------------------------------------------
2022-05-18T04:38:14.9230015Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:14.9607239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18472
2022-05-18T04:38:14.9717341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18473
2022-05-18T04:38:16.0973875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:16.1112799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:16.1113874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:16.1176805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:16.1183414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:16.2128396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:18.7808883Z ok (5.499s)
2022-05-18T04:38:18.7809309Z 
2022-05-18T04:38:18.7810364Z ----------------------------------------------------------------------
2022-05-18T04:38:18.7810741Z Ran 1 test in 5.499s
2022-05-18T04:38:18.7810916Z 
2022-05-18T04:38:18.7811009Z OK
2022-05-18T04:38:18.7811145Z 
2022-05-18T04:38:18.7811279Z Generating XML reports...
2022-05-18T04:38:18.7878907Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043813.xml
2022-05-18T04:38:20.2303462Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:20.2319073Z 
2022-05-18T04:38:20.2319211Z Running tests...
2022-05-18T04:38:20.2319797Z ----------------------------------------------------------------------
2022-05-18T04:38:21.8788440Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:21.8908678Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.659s)
2022-05-18T04:38:21.8909261Z 
2022-05-18T04:38:21.8909555Z ----------------------------------------------------------------------
2022-05-18T04:38:21.8909885Z Ran 1 test in 1.659s
2022-05-18T04:38:21.8910028Z 
2022-05-18T04:38:21.8910139Z OK (skipped=1)
2022-05-18T04:38:21.8910293Z 
2022-05-18T04:38:21.8910417Z Generating XML reports...
2022-05-18T04:38:21.8948626Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043820.xml
2022-05-18T04:38:23.2911486Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:23.2926754Z 
2022-05-18T04:38:23.2927005Z Running tests...
2022-05-18T04:38:23.2927630Z ----------------------------------------------------------------------
2022-05-18T04:38:24.9469991Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:24.9591586Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.666s)
2022-05-18T04:38:24.9592185Z 
2022-05-18T04:38:24.9592443Z ----------------------------------------------------------------------
2022-05-18T04:38:24.9592775Z Ran 1 test in 1.666s
2022-05-18T04:38:24.9592944Z 
2022-05-18T04:38:24.9593052Z OK (skipped=1)
2022-05-18T04:38:24.9593209Z 
2022-05-18T04:38:24.9593332Z Generating XML reports...
2022-05-18T04:38:24.9632042Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043823.xml
2022-05-18T04:38:26.3697870Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:26.3713064Z 
2022-05-18T04:38:26.3713517Z Running tests...
2022-05-18T04:38:26.3714042Z ----------------------------------------------------------------------
2022-05-18T04:38:28.0259668Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:28.0631801Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18667
2022-05-18T04:38:28.0742174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18668
2022-05-18T04:38:29.2142594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:29.2269198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:29.2270009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:29.2344995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:29.2351415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:29.3284925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:29.4790950Z skip: Need at least 4 CUDA devices (3.107s)
2022-05-18T04:38:29.4791217Z 
2022-05-18T04:38:29.4791833Z ----------------------------------------------------------------------
2022-05-18T04:38:29.4792184Z Ran 1 test in 3.108s
2022-05-18T04:38:29.4792351Z 
2022-05-18T04:38:29.4792454Z OK (skipped=1)
2022-05-18T04:38:29.4792609Z 
2022-05-18T04:38:29.4792738Z Generating XML reports...
2022-05-18T04:38:29.4849124Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043826.xml
2022-05-18T04:38:30.8859797Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:30.8874205Z 
2022-05-18T04:38:30.8874467Z Running tests...
2022-05-18T04:38:30.8874930Z ----------------------------------------------------------------------
2022-05-18T04:38:32.5107146Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:32.5472324Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18780
2022-05-18T04:38:32.5586287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18781
2022-05-18T04:38:33.7064473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:33.7088017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:33.7088844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:33.7168231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:33.7175217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:33.8102913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:33.9634527Z skip: Need at least 4 CUDA devices (3.076s)
2022-05-18T04:38:33.9634781Z 
2022-05-18T04:38:33.9635160Z ----------------------------------------------------------------------
2022-05-18T04:38:33.9635500Z Ran 1 test in 3.076s
2022-05-18T04:38:33.9635664Z 
2022-05-18T04:38:33.9635758Z OK (skipped=1)
2022-05-18T04:38:33.9635914Z 
2022-05-18T04:38:33.9636039Z Generating XML reports...
2022-05-18T04:38:33.9693788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043830.xml
2022-05-18T04:38:35.3872353Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:35.3887246Z 
2022-05-18T04:38:35.3887766Z Running tests...
2022-05-18T04:38:35.3888266Z ----------------------------------------------------------------------
2022-05-18T04:38:35.3910275Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:38:35.3910730Z 
2022-05-18T04:38:35.3911023Z ----------------------------------------------------------------------
2022-05-18T04:38:35.3911354Z Ran 1 test in 0.002s
2022-05-18T04:38:35.3911517Z 
2022-05-18T04:38:35.3911626Z OK (skipped=1)
2022-05-18T04:38:35.3911761Z 
2022-05-18T04:38:35.3911884Z Generating XML reports...
2022-05-18T04:38:35.3954082Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043835.xml
2022-05-18T04:38:36.6662387Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:36.6677209Z 
2022-05-18T04:38:36.6677478Z Running tests...
2022-05-18T04:38:36.6677902Z ----------------------------------------------------------------------
2022-05-18T04:38:36.6699700Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:38:36.6700025Z 
2022-05-18T04:38:36.6700308Z ----------------------------------------------------------------------
2022-05-18T04:38:36.6700638Z Ran 1 test in 0.002s
2022-05-18T04:38:36.6700802Z 
2022-05-18T04:38:36.6700900Z OK (skipped=1)
2022-05-18T04:38:36.6701055Z 
2022-05-18T04:38:36.6701179Z Generating XML reports...
2022-05-18T04:38:36.6743150Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043836.xml
2022-05-18T04:38:37.9442184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:37.9457606Z 
2022-05-18T04:38:37.9457766Z Running tests...
2022-05-18T04:38:37.9458185Z ----------------------------------------------------------------------
2022-05-18T04:38:37.9481166Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:38:37.9481502Z 
2022-05-18T04:38:37.9481779Z ----------------------------------------------------------------------
2022-05-18T04:38:37.9482105Z Ran 1 test in 0.002s
2022-05-18T04:38:37.9482267Z 
2022-05-18T04:38:37.9482357Z OK (skipped=1)
2022-05-18T04:38:37.9482512Z 
2022-05-18T04:38:37.9483826Z Generating XML reports...
2022-05-18T04:38:37.9525050Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043837.xml
2022-05-18T04:38:39.2341800Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:39.2356931Z 
2022-05-18T04:38:39.2357202Z Running tests...
2022-05-18T04:38:39.2357675Z ----------------------------------------------------------------------
2022-05-18T04:38:39.2379996Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:38:39.2380611Z 
2022-05-18T04:38:39.2380905Z ----------------------------------------------------------------------
2022-05-18T04:38:39.2381237Z Ran 1 test in 0.002s
2022-05-18T04:38:39.2381397Z 
2022-05-18T04:38:39.2381514Z OK (skipped=1)
2022-05-18T04:38:39.2381670Z 
2022-05-18T04:38:39.2381778Z Generating XML reports...
2022-05-18T04:38:39.2424053Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043839.xml
2022-05-18T04:38:40.5304150Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:40.5319628Z 
2022-05-18T04:38:40.5320067Z Running tests...
2022-05-18T04:38:40.5320548Z ----------------------------------------------------------------------
2022-05-18T04:38:40.5342493Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:38:40.5342818Z 
2022-05-18T04:38:40.5343113Z ----------------------------------------------------------------------
2022-05-18T04:38:40.5343439Z Ran 1 test in 0.002s
2022-05-18T04:38:40.5343608Z 
2022-05-18T04:38:40.5343725Z OK (skipped=1)
2022-05-18T04:38:40.5343877Z 
2022-05-18T04:38:40.5344002Z Generating XML reports...
2022-05-18T04:38:40.5387871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043840.xml
2022-05-18T04:38:41.8163319Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:41.8178936Z 
2022-05-18T04:38:41.8179235Z Running tests...
2022-05-18T04:38:41.8179670Z ----------------------------------------------------------------------
2022-05-18T04:38:41.8200458Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:38:41.8200767Z 
2022-05-18T04:38:41.8201047Z ----------------------------------------------------------------------
2022-05-18T04:38:41.8201378Z Ran 1 test in 0.002s
2022-05-18T04:38:41.8201550Z 
2022-05-18T04:38:41.8201664Z OK (skipped=1)
2022-05-18T04:38:41.8201820Z 
2022-05-18T04:38:41.8201927Z Generating XML reports...
2022-05-18T04:38:41.8244640Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043841.xml
2022-05-18T04:38:43.0973574Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:43.0988426Z 
2022-05-18T04:38:43.0988690Z Running tests...
2022-05-18T04:38:43.0989111Z ----------------------------------------------------------------------
2022-05-18T04:38:43.1013264Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:38:43.1013700Z 
2022-05-18T04:38:43.1014434Z ----------------------------------------------------------------------
2022-05-18T04:38:43.1014790Z Ran 1 test in 0.002s
2022-05-18T04:38:43.1014953Z 
2022-05-18T04:38:43.1015043Z OK (skipped=1)
2022-05-18T04:38:43.1015209Z 
2022-05-18T04:38:43.1015334Z Generating XML reports...
2022-05-18T04:38:43.1057749Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043843.xml
2022-05-18T04:38:44.3856166Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:44.3871172Z 
2022-05-18T04:38:44.3871365Z Running tests...
2022-05-18T04:38:44.3871808Z ----------------------------------------------------------------------
2022-05-18T04:38:44.3894761Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:38:44.3895080Z 
2022-05-18T04:38:44.3895362Z ----------------------------------------------------------------------
2022-05-18T04:38:44.3895689Z Ran 1 test in 0.002s
2022-05-18T04:38:44.3895851Z 
2022-05-18T04:38:44.3895960Z OK (skipped=1)
2022-05-18T04:38:44.3896121Z 
2022-05-18T04:38:44.3896246Z Generating XML reports...
2022-05-18T04:38:44.3938466Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043844.xml
2022-05-18T04:38:45.6653137Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:45.6668282Z 
2022-05-18T04:38:45.6668475Z Running tests...
2022-05-18T04:38:45.6668907Z ----------------------------------------------------------------------
2022-05-18T04:38:45.6690062Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:38:45.6690344Z 
2022-05-18T04:38:45.6690880Z ----------------------------------------------------------------------
2022-05-18T04:38:45.6691226Z Ran 1 test in 0.002s
2022-05-18T04:38:45.6691388Z 
2022-05-18T04:38:45.6691514Z OK (skipped=1)
2022-05-18T04:38:45.6691669Z 
2022-05-18T04:38:45.6691776Z Generating XML reports...
2022-05-18T04:38:45.6734901Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043845.xml
2022-05-18T04:38:46.9452659Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:46.9468462Z 
2022-05-18T04:38:46.9468990Z Running tests...
2022-05-18T04:38:46.9469478Z ----------------------------------------------------------------------
2022-05-18T04:38:46.9491081Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:38:46.9491669Z 
2022-05-18T04:38:46.9491999Z ----------------------------------------------------------------------
2022-05-18T04:38:46.9492332Z Ran 1 test in 0.002s
2022-05-18T04:38:46.9492581Z 
2022-05-18T04:38:46.9492672Z OK (skipped=1)
2022-05-18T04:38:46.9492826Z 
2022-05-18T04:38:46.9492950Z Generating XML reports...
2022-05-18T04:38:46.9535895Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043846.xml
2022-05-18T04:38:48.2329064Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:48.2344638Z 
2022-05-18T04:38:48.2344903Z Running tests...
2022-05-18T04:38:48.2345342Z ----------------------------------------------------------------------
2022-05-18T04:38:49.9123639Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:49.9494772Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19243
2022-05-18T04:38:49.9603155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19244
2022-05-18T04:38:51.1149423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:51.1733343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:51.1734145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:51.1756713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:51.1763790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:51.2748740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:53.4687138Z ok (5.234s)
2022-05-18T04:38:53.4687361Z 
2022-05-18T04:38:53.4687735Z ----------------------------------------------------------------------
2022-05-18T04:38:53.4688054Z Ran 1 test in 5.234s
2022-05-18T04:38:53.4688223Z 
2022-05-18T04:38:53.4688323Z OK
2022-05-18T04:38:53.4688457Z 
2022-05-18T04:38:53.4688589Z Generating XML reports...
2022-05-18T04:38:53.4745782Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043848.xml
2022-05-18T04:38:54.8752212Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:54.8767037Z 
2022-05-18T04:38:54.8767313Z Running tests...
2022-05-18T04:38:54.8767758Z ----------------------------------------------------------------------
2022-05-18T04:38:54.8791195Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:38:54.8791492Z 
2022-05-18T04:38:54.8791775Z ----------------------------------------------------------------------
2022-05-18T04:38:54.8792107Z Ran 1 test in 0.002s
2022-05-18T04:38:54.8792269Z 
2022-05-18T04:38:54.8792378Z OK (skipped=1)
2022-05-18T04:38:54.8792532Z 
2022-05-18T04:38:54.8792637Z Generating XML reports...
2022-05-18T04:38:54.8835068Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043854.xml
2022-05-18T04:38:56.1085829Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:56.1102227Z 
2022-05-18T04:38:56.1102669Z Running tests...
2022-05-18T04:38:56.1103112Z ----------------------------------------------------------------------
2022-05-18T04:38:56.1125668Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:38:56.1125969Z 
2022-05-18T04:38:56.1126250Z ----------------------------------------------------------------------
2022-05-18T04:38:56.1126581Z Ran 1 test in 0.002s
2022-05-18T04:38:56.1126741Z 
2022-05-18T04:38:56.1126849Z OK (skipped=1)
2022-05-18T04:38:56.1126986Z 
2022-05-18T04:38:56.1127594Z Generating XML reports...
2022-05-18T04:38:56.1171888Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043856.xml
2022-05-18T04:38:57.3968388Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:38:57.3985526Z 
2022-05-18T04:38:57.3986002Z Running tests...
2022-05-18T04:38:57.3986515Z ----------------------------------------------------------------------
2022-05-18T04:38:59.0604207Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:59.0975602Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19439
2022-05-18T04:38:59.1084887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19440
2022-05-18T04:39:00.2750024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:00.3184672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:00.3185471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:00.3256745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:00.3262947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:00.4200286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:02.5183234Z ok (5.119s)
2022-05-18T04:39:02.5183434Z 
2022-05-18T04:39:02.5183822Z ----------------------------------------------------------------------
2022-05-18T04:39:02.5184158Z Ran 1 test in 5.120s
2022-05-18T04:39:02.5184323Z 
2022-05-18T04:39:02.5184419Z OK
2022-05-18T04:39:02.5184551Z 
2022-05-18T04:39:02.5184667Z Generating XML reports...
2022-05-18T04:39:02.5241286Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043857.xml
2022-05-18T04:39:03.9662398Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:03.9677675Z 
2022-05-18T04:39:03.9678190Z Running tests...
2022-05-18T04:39:03.9678673Z ----------------------------------------------------------------------
2022-05-18T04:39:05.6227837Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:05.6597982Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19565
2022-05-18T04:39:05.6706400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19566
2022-05-18T04:39:06.7606689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:06.8261176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:06.8261976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:06.8315646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:06.8323349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:06.9276577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:09.0791362Z ok (5.111s)
2022-05-18T04:39:09.0791574Z 
2022-05-18T04:39:09.0791986Z ----------------------------------------------------------------------
2022-05-18T04:39:09.0792308Z Ran 1 test in 5.111s
2022-05-18T04:39:09.0792472Z 
2022-05-18T04:39:09.0792567Z OK
2022-05-18T04:39:09.0792703Z 
2022-05-18T04:39:09.0792840Z Generating XML reports...
2022-05-18T04:39:09.0850905Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043903.xml
2022-05-18T04:39:10.4983486Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:10.4997736Z 
2022-05-18T04:39:10.4998137Z Running tests...
2022-05-18T04:39:10.4998637Z ----------------------------------------------------------------------
2022-05-18T04:39:10.5020111Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:39:10.5020447Z 
2022-05-18T04:39:10.5020735Z ----------------------------------------------------------------------
2022-05-18T04:39:10.5021067Z Ran 1 test in 0.002s
2022-05-18T04:39:10.5021230Z 
2022-05-18T04:39:10.5021319Z OK (skipped=1)
2022-05-18T04:39:10.5021478Z 
2022-05-18T04:39:10.5021605Z Generating XML reports...
2022-05-18T04:39:10.5062932Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043910.xml
2022-05-18T04:39:11.7815348Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:11.7830408Z 
2022-05-18T04:39:11.7830549Z Running tests...
2022-05-18T04:39:11.7831922Z ----------------------------------------------------------------------
2022-05-18T04:39:11.7851091Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:39:11.7851423Z 
2022-05-18T04:39:11.7851718Z ----------------------------------------------------------------------
2022-05-18T04:39:11.7852056Z Ran 1 test in 0.002s
2022-05-18T04:39:11.7852218Z 
2022-05-18T04:39:11.7852307Z OK (skipped=1)
2022-05-18T04:39:11.7852461Z 
2022-05-18T04:39:11.7852585Z Generating XML reports...
2022-05-18T04:39:11.7895156Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043911.xml
2022-05-18T04:39:13.0527386Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:13.0543005Z 
2022-05-18T04:39:13.0543251Z Running tests...
2022-05-18T04:39:13.0543683Z ----------------------------------------------------------------------
2022-05-18T04:39:13.0570587Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.003s)
2022-05-18T04:39:13.0570902Z 
2022-05-18T04:39:13.0571182Z ----------------------------------------------------------------------
2022-05-18T04:39:13.0571495Z Ran 1 test in 0.003s
2022-05-18T04:39:13.0571665Z 
2022-05-18T04:39:13.0572040Z OK (skipped=1)
2022-05-18T04:39:13.0572200Z 
2022-05-18T04:39:13.0572324Z Generating XML reports...
2022-05-18T04:39:13.0613748Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043913.xml
2022-05-18T04:39:14.2987152Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:14.3002018Z 
2022-05-18T04:39:14.3002560Z Running tests...
2022-05-18T04:39:14.3003067Z ----------------------------------------------------------------------
2022-05-18T04:39:14.3022678Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:39:14.3023194Z 
2022-05-18T04:39:14.3023501Z ----------------------------------------------------------------------
2022-05-18T04:39:14.3023812Z Ran 1 test in 0.002s
2022-05-18T04:39:14.3023979Z 
2022-05-18T04:39:14.3024092Z OK (skipped=1)
2022-05-18T04:39:14.3024249Z 
2022-05-18T04:39:14.3024386Z Generating XML reports...
2022-05-18T04:39:14.3066926Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043914.xml
2022-05-18T04:39:15.5697212Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:15.5712163Z 
2022-05-18T04:39:15.5712388Z Running tests...
2022-05-18T04:39:15.5712828Z ----------------------------------------------------------------------
2022-05-18T04:39:17.2445472Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:17.2806274Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19831
2022-05-18T04:39:17.2914098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19832
2022-05-18T04:39:18.4712469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:18.5095627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:18.5096651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:18.5116970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:18.5123487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:18.6111448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:22.6035140Z ok (7.032s)
2022-05-18T04:39:22.6035360Z 
2022-05-18T04:39:22.6035755Z ----------------------------------------------------------------------
2022-05-18T04:39:22.6036352Z Ran 1 test in 7.032s
2022-05-18T04:39:22.6036532Z 
2022-05-18T04:39:22.6037605Z OK
2022-05-18T04:39:22.6038055Z 
2022-05-18T04:39:22.6038363Z Generating XML reports...
2022-05-18T04:39:22.6092769Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043915.xml
2022-05-18T04:39:24.0537586Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:24.0553241Z 
2022-05-18T04:39:24.0553775Z Running tests...
2022-05-18T04:39:24.0554261Z ----------------------------------------------------------------------
2022-05-18T04:39:25.6975550Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:25.7340513Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19959
2022-05-18T04:39:25.7447388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19960
2022-05-18T04:39:26.8928208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:26.9032060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:26.9033109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:26.9130490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:26.9137275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:27.0046661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:30.9562700Z ok (6.901s)
2022-05-18T04:39:30.9562921Z 
2022-05-18T04:39:30.9563303Z ----------------------------------------------------------------------
2022-05-18T04:39:30.9563641Z Ran 1 test in 6.901s
2022-05-18T04:39:30.9563796Z 
2022-05-18T04:39:30.9563890Z OK
2022-05-18T04:39:30.9564041Z 
2022-05-18T04:39:30.9564180Z Generating XML reports...
2022-05-18T04:39:30.9620438Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043924.xml
2022-05-18T04:39:32.3990922Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:32.4006032Z 
2022-05-18T04:39:32.4006350Z Running tests...
2022-05-18T04:39:32.4006784Z ----------------------------------------------------------------------
2022-05-18T04:39:32.4026668Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:39:32.4026958Z 
2022-05-18T04:39:32.4027239Z ----------------------------------------------------------------------
2022-05-18T04:39:32.4027567Z Ran 1 test in 0.002s
2022-05-18T04:39:32.4027731Z 
2022-05-18T04:39:32.4027838Z OK (skipped=1)
2022-05-18T04:39:32.4027995Z 
2022-05-18T04:39:32.4028102Z Generating XML reports...
2022-05-18T04:39:32.4070785Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043932.xml
2022-05-18T04:39:33.6344496Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:33.6359685Z 
2022-05-18T04:39:33.6360147Z Running tests...
2022-05-18T04:39:33.6360649Z ----------------------------------------------------------------------
2022-05-18T04:39:33.6379759Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:39:33.6380108Z 
2022-05-18T04:39:33.6380430Z ----------------------------------------------------------------------
2022-05-18T04:39:33.6380796Z Ran 1 test in 0.002s
2022-05-18T04:39:33.6380966Z 
2022-05-18T04:39:33.6381076Z OK (skipped=1)
2022-05-18T04:39:33.6381232Z 
2022-05-18T04:39:33.6381373Z Generating XML reports...
2022-05-18T04:39:33.6423862Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043933.xml
2022-05-18T04:39:34.9219695Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:34.9234784Z 
2022-05-18T04:39:34.9235189Z Running tests...
2022-05-18T04:39:34.9235690Z ----------------------------------------------------------------------
2022-05-18T04:39:34.9262102Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.003s)
2022-05-18T04:39:34.9262430Z 
2022-05-18T04:39:34.9262681Z ----------------------------------------------------------------------
2022-05-18T04:39:34.9263011Z Ran 1 test in 0.003s
2022-05-18T04:39:34.9263172Z 
2022-05-18T04:39:34.9263281Z OK (skipped=1)
2022-05-18T04:39:34.9263434Z 
2022-05-18T04:39:34.9263561Z Generating XML reports...
2022-05-18T04:39:34.9306844Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043934.xml
2022-05-18T04:39:36.2069836Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:36.2084668Z 
2022-05-18T04:39:36.2084817Z Running tests...
2022-05-18T04:39:36.2085258Z ----------------------------------------------------------------------
2022-05-18T04:39:36.2105247Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:39:36.2105573Z 
2022-05-18T04:39:36.2105858Z ----------------------------------------------------------------------
2022-05-18T04:39:36.2106180Z Ran 1 test in 0.002s
2022-05-18T04:39:36.2106341Z 
2022-05-18T04:39:36.2106450Z OK (skipped=1)
2022-05-18T04:39:36.2106587Z 
2022-05-18T04:39:36.2106716Z Generating XML reports...
2022-05-18T04:39:36.2148717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043936.xml
2022-05-18T04:39:37.4618638Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:37.4635754Z 
2022-05-18T04:39:37.4636190Z Running tests...
2022-05-18T04:39:37.4636697Z ----------------------------------------------------------------------
2022-05-18T04:39:37.4656891Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-05-18T04:39:37.4657228Z 
2022-05-18T04:39:37.4657510Z ----------------------------------------------------------------------
2022-05-18T04:39:37.4657835Z Ran 1 test in 0.002s
2022-05-18T04:39:37.4657994Z 
2022-05-18T04:39:37.4658086Z OK (skipped=1)
2022-05-18T04:39:37.4658245Z 
2022-05-18T04:39:37.4658370Z Generating XML reports...
2022-05-18T04:39:37.4701979Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043937.xml
2022-05-18T04:39:38.7388537Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:38.7403640Z 
2022-05-18T04:39:38.7403869Z Running tests...
2022-05-18T04:39:38.7404304Z ----------------------------------------------------------------------
2022-05-18T04:39:38.7424368Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-05-18T04:39:38.7424737Z 
2022-05-18T04:39:38.7425014Z ----------------------------------------------------------------------
2022-05-18T04:39:38.7425326Z Ran 1 test in 0.002s
2022-05-18T04:39:38.7425489Z 
2022-05-18T04:39:38.7425598Z OK (skipped=1)
2022-05-18T04:39:38.7425754Z 
2022-05-18T04:39:38.7425878Z Generating XML reports...
2022-05-18T04:39:38.7468553Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043938.xml
2022-05-18T04:39:40.0118283Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:40.0133623Z 
2022-05-18T04:39:40.0133936Z Running tests...
2022-05-18T04:39:40.0134610Z ----------------------------------------------------------------------
2022-05-18T04:39:40.0155885Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-05-18T04:39:40.0156271Z 
2022-05-18T04:39:40.0156603Z ----------------------------------------------------------------------
2022-05-18T04:39:40.0156932Z Ran 1 test in 0.002s
2022-05-18T04:39:40.0157093Z 
2022-05-18T04:39:40.0157198Z OK (skipped=1)
2022-05-18T04:39:40.0157334Z 
2022-05-18T04:39:40.0157455Z Generating XML reports...
2022-05-18T04:39:40.0198920Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043940.xml
2022-05-18T04:39:41.2546931Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:41.2562053Z 
2022-05-18T04:39:41.2562311Z Running tests...
2022-05-18T04:39:41.2562748Z ----------------------------------------------------------------------
2022-05-18T04:39:41.2583064Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:39:41.2583404Z 
2022-05-18T04:39:41.2583966Z ----------------------------------------------------------------------
2022-05-18T04:39:41.2584301Z Ran 1 test in 0.002s
2022-05-18T04:39:41.2584462Z 
2022-05-18T04:39:41.2584567Z OK (skipped=1)
2022-05-18T04:39:41.2584721Z 
2022-05-18T04:39:41.2584844Z Generating XML reports...
2022-05-18T04:39:41.2626384Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043941.xml
2022-05-18T04:39:42.5331695Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:42.5346977Z 
2022-05-18T04:39:42.5347313Z Running tests...
2022-05-18T04:39:42.5347798Z ----------------------------------------------------------------------
2022-05-18T04:39:44.2166254Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:44.2537094Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20367
2022-05-18T04:39:44.2644592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20368
2022-05-18T04:39:45.4568784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:45.4569336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:45.4570609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:45.4571286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:45.4577572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:45.4578378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:47.1719087Z ok (4.637s)
2022-05-18T04:39:47.1719283Z 
2022-05-18T04:39:47.1719679Z ----------------------------------------------------------------------
2022-05-18T04:39:47.1720033Z Ran 1 test in 4.637s
2022-05-18T04:39:47.1720202Z 
2022-05-18T04:39:47.1720295Z OK
2022-05-18T04:39:47.1720429Z 
2022-05-18T04:39:47.1720545Z Generating XML reports...
2022-05-18T04:39:47.1777374Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043942.xml
2022-05-18T04:39:48.6247993Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:48.6263550Z 
2022-05-18T04:39:48.6263787Z Running tests...
2022-05-18T04:39:48.6264225Z ----------------------------------------------------------------------
2022-05-18T04:39:50.2658552Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:50.3022834Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20489
2022-05-18T04:39:50.3129976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20490
2022-05-18T04:39:51.4978812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:51.5351872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:51.5352673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:51.5383747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:51.5390359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:51.6367407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:53.7211871Z ok (5.094s)
2022-05-18T04:39:53.7212093Z 
2022-05-18T04:39:53.7212480Z ----------------------------------------------------------------------
2022-05-18T04:39:53.7212798Z Ran 1 test in 5.095s
2022-05-18T04:39:53.7213241Z 
2022-05-18T04:39:53.7213334Z OK
2022-05-18T04:39:53.7213475Z 
2022-05-18T04:39:53.7213606Z Generating XML reports...
2022-05-18T04:39:53.7269227Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043948.xml
2022-05-18T04:39:55.1677609Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:39:55.1692878Z 
2022-05-18T04:39:55.1693133Z Running tests...
2022-05-18T04:39:55.1693799Z ----------------------------------------------------------------------
2022-05-18T04:39:56.8164568Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:56.8534787Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20615
2022-05-18T04:39:56.8643556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20616
2022-05-18T04:39:58.0064564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:58.0188707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:58.0189797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:58.0266995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:58.0273741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:58.1203621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:00.2729710Z ok (5.103s)
2022-05-18T04:40:00.2730214Z 
2022-05-18T04:40:00.2730598Z ----------------------------------------------------------------------
2022-05-18T04:40:00.2730940Z Ran 1 test in 5.104s
2022-05-18T04:40:00.2731105Z 
2022-05-18T04:40:00.2731214Z OK
2022-05-18T04:40:00.2731348Z 
2022-05-18T04:40:00.2731480Z Generating XML reports...
2022-05-18T04:40:00.2789641Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043955.xml
2022-05-18T04:40:01.6926074Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:40:01.6943807Z 
2022-05-18T04:40:01.6944089Z Running tests...
2022-05-18T04:40:01.6944553Z ----------------------------------------------------------------------
2022-05-18T04:40:01.6965337Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:40:01.6965661Z 
2022-05-18T04:40:01.6966211Z ----------------------------------------------------------------------
2022-05-18T04:40:01.6966559Z Ran 1 test in 0.002s
2022-05-18T04:40:01.6966719Z 
2022-05-18T04:40:01.6966827Z OK (skipped=1)
2022-05-18T04:40:01.6966980Z 
2022-05-18T04:40:01.6967086Z Generating XML reports...
2022-05-18T04:40:01.7007754Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044001.xml
2022-05-18T04:40:02.9665496Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:40:02.9680619Z 
2022-05-18T04:40:02.9680862Z Running tests...
2022-05-18T04:40:02.9681307Z ----------------------------------------------------------------------
2022-05-18T04:40:02.9701210Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:40:02.9701823Z 
2022-05-18T04:40:02.9702169Z ----------------------------------------------------------------------
2022-05-18T04:40:02.9702507Z Ran 1 test in 0.002s
2022-05-18T04:40:02.9702683Z 
2022-05-18T04:40:02.9702793Z OK (skipped=1)
2022-05-18T04:40:02.9702932Z 
2022-05-18T04:40:02.9703054Z Generating XML reports...
2022-05-18T04:40:02.9744426Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044002.xml
2022-05-18T04:40:04.2523254Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:40:04.2539345Z 
2022-05-18T04:40:04.2539782Z Running tests...
2022-05-18T04:40:04.2540286Z ----------------------------------------------------------------------
2022-05-18T04:40:04.2560333Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:40:04.2560684Z 
2022-05-18T04:40:04.2560950Z ----------------------------------------------------------------------
2022-05-18T04:40:04.2561274Z Ran 1 test in 0.002s
2022-05-18T04:40:04.2561434Z 
2022-05-18T04:40:04.2561542Z OK (skipped=1)
2022-05-18T04:40:04.2561716Z 
2022-05-18T04:40:04.2561842Z Generating XML reports...
2022-05-18T04:40:04.2605878Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044004.xml
2022-05-18T04:40:05.5290767Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:40:05.5305975Z 
2022-05-18T04:40:05.5306419Z Running tests...
2022-05-18T04:40:05.5306910Z ----------------------------------------------------------------------
2022-05-18T04:40:05.5328064Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:40:05.5328401Z 
2022-05-18T04:40:05.5328682Z ----------------------------------------------------------------------
2022-05-18T04:40:05.5328992Z Ran 1 test in 0.002s
2022-05-18T04:40:05.5329156Z 
2022-05-18T04:40:05.5329273Z OK (skipped=1)
2022-05-18T04:40:05.5329429Z 
2022-05-18T04:40:05.5329846Z Generating XML reports...
2022-05-18T04:40:05.5371677Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044005.xml
2022-05-18T04:40:06.8158520Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:40:06.8174534Z 
2022-05-18T04:40:06.8174930Z Running tests...
2022-05-18T04:40:06.8175392Z ----------------------------------------------------------------------
2022-05-18T04:40:06.8195577Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-05-18T04:40:06.8195906Z 
2022-05-18T04:40:06.8196184Z ----------------------------------------------------------------------
2022-05-18T04:40:06.8196507Z Ran 1 test in 0.002s
2022-05-18T04:40:06.8196651Z 
2022-05-18T04:40:06.8196766Z OK (skipped=1)
2022-05-18T04:40:06.8196917Z 
2022-05-18T04:40:06.8197042Z Generating XML reports...
2022-05-18T04:40:06.8240592Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044006.xml
2022-05-18T04:40:08.0990906Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:40:08.1006348Z 
2022-05-18T04:40:08.1006792Z Running tests...
2022-05-18T04:40:08.1007291Z ----------------------------------------------------------------------
2022-05-18T04:40:08.1027645Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-05-18T04:40:08.1027980Z 
2022-05-18T04:40:08.1028244Z ----------------------------------------------------------------------
2022-05-18T04:40:08.1028571Z Ran 1 test in 0.002s
2022-05-18T04:40:08.1028734Z 
2022-05-18T04:40:08.1028850Z OK (skipped=1)
2022-05-18T04:40:08.1029003Z 
2022-05-18T04:40:08.1029126Z Generating XML reports...
2022-05-18T04:40:08.1072285Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044008.xml
2022-05-18T04:40:09.3557408Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:40:09.3571761Z 
2022-05-18T04:40:09.3572167Z Running tests...
2022-05-18T04:40:09.3572612Z ----------------------------------------------------------------------
2022-05-18T04:40:11.0056015Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:11.0418589Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20951
2022-05-18T04:40:11.0527436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20952
2022-05-18T04:40:12.1752127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:12.1840806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:12.1841940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:12.1853388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:12.1859870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:12.2857718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:15.2624906Z ok (5.905s)
2022-05-18T04:40:15.2625197Z 
2022-05-18T04:40:15.2625582Z ----------------------------------------------------------------------
2022-05-18T04:40:15.2625919Z Ran 1 test in 5.905s
2022-05-18T04:40:15.2626088Z 
2022-05-18T04:40:15.2626183Z OK
2022-05-18T04:40:15.2626304Z 
2022-05-18T04:40:15.2626438Z Generating XML reports...
2022-05-18T04:40:15.2682342Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044009.xml
2022-05-18T04:40:16.6778168Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:40:16.6792183Z 
2022-05-18T04:40:16.6792384Z Running tests...
2022-05-18T04:40:16.6792804Z ----------------------------------------------------------------------
2022-05-18T04:40:16.6816597Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-05-18T04:40:16.6816924Z 
2022-05-18T04:40:16.6817208Z ----------------------------------------------------------------------
2022-05-18T04:40:16.6817535Z Ran 1 test in 0.002s
2022-05-18T04:40:16.6817695Z 
2022-05-18T04:40:16.6817787Z OK (skipped=1)
2022-05-18T04:40:16.6817945Z 
2022-05-18T04:40:16.6818069Z Generating XML reports...
2022-05-18T04:40:16.6858929Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044016.xml
2022-05-18T04:40:17.9333856Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:40:17.9349592Z 
2022-05-18T04:40:17.9349761Z Running tests...
2022-05-18T04:40:17.9350201Z ----------------------------------------------------------------------
2022-05-18T04:40:19.5534418Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:19.5898414Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21113
2022-05-18T04:40:19.6011203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21114
2022-05-18T04:40:20.7629324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:20.7806381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:20.7807186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:20.7831310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:20.7837708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:20.8821773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:22.1097361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehh8sd16
2022-05-18T04:40:22.1097979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehh8sd16/_remote_module_non_scriptable.py
2022-05-18T04:40:22.1689149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7t6qna0
2022-05-18T04:40:22.1690377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7t6qna0/_remote_module_non_scriptable.py
2022-05-18T04:40:23.5104459Z ok (5.575s)
2022-05-18T04:40:23.5104659Z 
2022-05-18T04:40:23.5105053Z ----------------------------------------------------------------------
2022-05-18T04:40:23.5105370Z Ran 1 test in 5.576s
2022-05-18T04:40:23.5105557Z 
2022-05-18T04:40:23.5105652Z OK
2022-05-18T04:40:23.5105786Z 
2022-05-18T04:40:23.5105919Z Generating XML reports...
2022-05-18T04:40:23.5163025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044017.xml
2022-05-18T04:40:24.9531674Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:40:24.9548085Z 
2022-05-18T04:40:24.9548338Z Running tests...
2022-05-18T04:40:24.9548758Z ----------------------------------------------------------------------
2022-05-18T04:40:26.6024623Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:26.6399929Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21236
2022-05-18T04:40:26.6509947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21237
2022-05-18T04:40:27.8399208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:27.8563849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:27.8564656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:27.8601936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:27.8608742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:27.9578895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:29.1701582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3qs_ldg
2022-05-18T04:40:29.1702170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3qs_ldg/_remote_module_non_scriptable.py
2022-05-18T04:40:29.2313095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsr8wq0dl
2022-05-18T04:40:29.2313955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsr8wq0dl/_remote_module_non_scriptable.py
2022-05-18T04:40:30.5411732Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:40:30.5498873Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:40:30.9612872Z ok (6.006s)
2022-05-18T04:40:30.9613086Z 
2022-05-18T04:40:30.9613502Z ----------------------------------------------------------------------
2022-05-18T04:40:30.9613847Z Ran 1 test in 6.006s
2022-05-18T04:40:30.9614019Z 
2022-05-18T04:40:30.9614115Z OK
2022-05-18T04:40:30.9614250Z 
2022-05-18T04:40:30.9614386Z Generating XML reports...
2022-05-18T04:40:30.9672170Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044024.xml
2022-05-18T04:40:32.4033904Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:40:32.4049291Z 
2022-05-18T04:40:32.4049835Z Running tests...
2022-05-18T04:40:32.4050609Z ----------------------------------------------------------------------
2022-05-18T04:40:34.0608035Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:34.0980938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21363
2022-05-18T04:40:34.1089426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21364
2022-05-18T04:40:35.2620223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:35.2948896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:35.2949695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:35.3026439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:35.3033298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:35.3960438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:35.4071128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:40:35.4071668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:40:35.4072370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:35.4073120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:35.4074545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:40:35.4176444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:40:35.4177153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:40:35.4177848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:40:36.7616094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8plp2n4
2022-05-18T04:40:36.7616709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8plp2n4/_remote_module_non_scriptable.py
2022-05-18T04:40:36.7731953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7pd9hves
2022-05-18T04:40:36.7735124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7pd9hves/_remote_module_non_scriptable.py
2022-05-18T04:40:42.4271486Z ok (10.022s)
2022-05-18T04:40:42.4271904Z 
2022-05-18T04:40:42.4272680Z ----------------------------------------------------------------------
2022-05-18T04:40:42.4273081Z Ran 1 test in 10.022s
2022-05-18T04:40:42.4273247Z 
2022-05-18T04:40:42.4273344Z OK
2022-05-18T04:40:42.4273515Z 
2022-05-18T04:40:42.4273868Z Generating XML reports...
2022-05-18T04:40:42.4329346Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044032.xml
2022-05-18T04:40:43.8615848Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:40:43.8630361Z 
2022-05-18T04:40:43.8630903Z Running tests...
2022-05-18T04:40:43.8631420Z ----------------------------------------------------------------------
2022-05-18T04:40:45.5219845Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:45.5590117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21493
2022-05-18T04:40:45.5700599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21494
2022-05-18T04:40:46.7127008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:46.7176397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:46.7177459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:46.7228514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:46.7234789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:46.8189887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:46.8399918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:40:46.8400654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:40:46.8401345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:46.8402044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:46.8403375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:40:46.8504543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:40:46.8505476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:40:46.8506167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:40:48.1744199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb105artf
2022-05-18T04:40:48.1744811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb105artf/_remote_module_non_scriptable.py
2022-05-18T04:40:48.2011012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1krdh074
2022-05-18T04:40:48.2013895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1krdh074/_remote_module_non_scriptable.py
2022-05-18T04:40:53.7871168Z ok (9.924s)
2022-05-18T04:40:53.7871443Z 
2022-05-18T04:40:53.7872136Z ----------------------------------------------------------------------
2022-05-18T04:40:53.7872829Z Ran 1 test in 9.924s
2022-05-18T04:40:53.7873042Z 
2022-05-18T04:40:53.7873137Z OK
2022-05-18T04:40:53.7873256Z 
2022-05-18T04:40:53.7873393Z Generating XML reports...
2022-05-18T04:40:53.7929431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044043.xml
2022-05-18T04:40:54.1926096Z Running distributed tests for the nccl backend with file init_method
2022-05-18T04:40:54.1928301Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:40:54.192496]
2022-05-18T04:40:55.3411684Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:40:55.3456557Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-05-18T04:40:55.3491104Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3491614Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3492014Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3492531Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3492989Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3493438Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3493922Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3494410Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3494903Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3495438Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3495992Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3496514Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3497028Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3497541Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3498013Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3498580Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3499025Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3499453Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3499875Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3500300Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3500776Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3501246Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3501650Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3502026Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3502454Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3502870Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3503267Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3503685Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3504092Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3504455Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3504842Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3505237Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3505626Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3505995Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3506399Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3506818Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3507214Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3507641Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3508079Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3508499Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3508941Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3509360Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3509862Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3510291Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3510719Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3511134Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3511552Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3511986Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3512393Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3512801Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3513202Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3513623Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3514023Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3514420Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3514825Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3515291Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3515679Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3516056Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3516450Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3516823Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3517174Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3517556Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3517957Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3518344Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3518734Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3519112Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3519473Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3519862Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3520249Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3520638Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3521030Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3521411Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3521781Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3522137Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3522516Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3522915Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3523293Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3523680Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3524066Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3524464Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3524874Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3525305Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3525742Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3526174Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3526623Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3527122Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3527572Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3527986Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3528419Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3528853Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3583079Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3583578Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3584066Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3584538Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3585022Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3585450Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3585850Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3586418Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3586782Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3587154Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3587547Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3587942Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3588347Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3588744Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3589162Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3589574Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3589992Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3590411Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3590821Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3591267Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3591700Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3592116Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3592549Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3592974Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3593415Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3593843Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3594276Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3594674Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3595044Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3595442Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3595838Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3596226Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3596634Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3597100Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3597629Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3598089Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3598508Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3599025Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3599472Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3599941Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3600433Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3600894Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3601316Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3601775Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3602193Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3602568Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3602970Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3603393Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3603817Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3604253Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3604773Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3605204Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3605666Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3606180Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3606761Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3607415Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3608048Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3608691Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3609332Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3610560Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3611158Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3611748Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3612281Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3612762Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3613186Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3613559Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3613945Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3614327Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3614712Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3615121Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3615546Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3616073Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3616542Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3616948Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3617312Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3617715Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3618144Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3618548Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3618951Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3619359Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3619772Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3620182Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3620590Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3620984Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3621453Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3621856Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3622243Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3622649Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3623082Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3623497Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3623862Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3624235Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3624646Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3625046Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3625407Z test_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3625756Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3626115Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3626483Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3626834Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3627189Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3627560Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3627916Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3628265Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3628611Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3628968Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3629346Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3629730Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3630082Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3630402Z test_isend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3630760Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3631147Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3631530Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3631969Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3632403Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3632849Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3633262Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3633679Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3634138Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3634534Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3634941Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3635340Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3635722Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3636115Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3636497Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3636869Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3637327Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3637802Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3638337Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3638761Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3639204Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3639646Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3640061Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3640454Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3640867Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3641291Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3641716Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3642183Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3642686Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3643131Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3643501Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3643889Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3644280Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3644644Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3645013Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3645396Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3645757Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3646114Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3646468Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3646818Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3647183Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3647536Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3647891Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3648249Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3648616Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3648968Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3649308Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3650381Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3650770Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3651128Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3651513Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3651882Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3652253Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3652597Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3652960Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3653369Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3653790Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3654203Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3654584Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3654961Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3655373Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3655861Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3656243Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3656625Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3657040Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3657439Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3657818Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3658211Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3658596Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3658961Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3659366Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3659805Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:55.3660235Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:40:56.4964932Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:40:56.4980590Z 
2022-05-18T04:40:56.4981003Z Running tests...
2022-05-18T04:40:56.4981506Z ----------------------------------------------------------------------
2022-05-18T04:40:58.1352258Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:58.1716401Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21658
2022-05-18T04:40:58.1819430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21659
2022-05-18T04:40:59.3338443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:59.3502293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:59.3503134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:59.3541224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:59.3548491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:59.4517128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:00.6723722Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:41:00.6725083Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:41:00.7269572Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:41:00.7270618Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:41:01.7302554Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:41:01.7303556Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:41:01.7304675Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:41:01.7305509Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:41:01.7448257Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:41:01.7449418Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:41:01.7453824Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:41:01.7454677Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:41:01.7596881Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:41:01.7597746Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:41:01.7601737Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:41:01.7602590Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:41:02.0912708Z ok (5.593s)
2022-05-18T04:41:02.0913647Z 
2022-05-18T04:41:02.0914055Z ----------------------------------------------------------------------
2022-05-18T04:41:02.0917768Z Ran 1 test in 5.593s
2022-05-18T04:41:02.0918200Z 
2022-05-18T04:41:02.0918386Z OK
2022-05-18T04:41:02.0918657Z 
2022-05-18T04:41:02.0918796Z Generating XML reports...
2022-05-18T04:41:02.0977675Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044056.xml
2022-05-18T04:41:03.5547764Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:41:03.5563684Z 
2022-05-18T04:41:03.5563937Z Running tests...
2022-05-18T04:41:03.5564400Z ----------------------------------------------------------------------
2022-05-18T04:41:03.5611879Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.005s)
2022-05-18T04:41:03.5612752Z 
2022-05-18T04:41:03.5613050Z ----------------------------------------------------------------------
2022-05-18T04:41:03.5613492Z Ran 1 test in 0.005s
2022-05-18T04:41:03.5613787Z 
2022-05-18T04:41:03.5613923Z OK (skipped=1)
2022-05-18T04:41:03.5614080Z 
2022-05-18T04:41:03.5614205Z Generating XML reports...
2022-05-18T04:41:03.5657143Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044103.xml
2022-05-18T04:41:04.8372971Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:41:04.8388238Z 
2022-05-18T04:41:04.8388698Z Running tests...
2022-05-18T04:41:04.8389204Z ----------------------------------------------------------------------
2022-05-18T04:41:06.4998497Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:06.5362837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21816
2022-05-18T04:41:06.5468247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21817
2022-05-18T04:41:07.6918941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:07.7061704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:07.7062503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:07.7121448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:07.7128060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:07.8074840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:07.9517475Z ok (3.113s)
2022-05-18T04:41:07.9517691Z 
2022-05-18T04:41:07.9518246Z ----------------------------------------------------------------------
2022-05-18T04:41:07.9518586Z Ran 1 test in 3.113s
2022-05-18T04:41:07.9518756Z 
2022-05-18T04:41:07.9518852Z OK
2022-05-18T04:41:07.9518987Z 
2022-05-18T04:41:07.9519126Z Generating XML reports...
2022-05-18T04:41:07.9586698Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044104.xml
2022-05-18T04:41:09.3598173Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:41:09.3612889Z 
2022-05-18T04:41:09.3613361Z Running tests...
2022-05-18T04:41:09.3613872Z ----------------------------------------------------------------------
2022-05-18T04:41:10.9711797Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:10.9829454Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.621s)
2022-05-18T04:41:10.9830044Z 
2022-05-18T04:41:10.9830333Z ----------------------------------------------------------------------
2022-05-18T04:41:10.9830664Z Ran 1 test in 1.622s
2022-05-18T04:41:10.9830828Z 
2022-05-18T04:41:10.9830918Z OK (skipped=1)
2022-05-18T04:41:10.9831070Z 
2022-05-18T04:41:10.9831195Z Generating XML reports...
2022-05-18T04:41:10.9868550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044109.xml
2022-05-18T04:41:12.3806160Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:41:12.3822256Z 
2022-05-18T04:41:12.3822726Z Running tests...
2022-05-18T04:41:12.3823452Z ----------------------------------------------------------------------
2022-05-18T04:41:12.3842171Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-05-18T04:41:12.3842534Z 
2022-05-18T04:41:12.3842897Z ----------------------------------------------------------------------
2022-05-18T04:41:12.3843408Z Ran 1 test in 0.002s
2022-05-18T04:41:12.3843577Z 
2022-05-18T04:41:12.3843687Z OK (skipped=1)
2022-05-18T04:41:12.3843844Z 
2022-05-18T04:41:12.3843969Z Generating XML reports...
2022-05-18T04:41:12.3885915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044112.xml
2022-05-18T04:41:13.6631377Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:41:13.6646354Z 
2022-05-18T04:41:13.6646675Z Running tests...
2022-05-18T04:41:13.6647357Z ----------------------------------------------------------------------
2022-05-18T04:41:13.6666108Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-05-18T04:41:13.6666880Z 
2022-05-18T04:41:13.6667436Z ----------------------------------------------------------------------
2022-05-18T04:41:13.6667825Z Ran 1 test in 0.002s
2022-05-18T04:41:13.6667986Z 
2022-05-18T04:41:13.6668095Z OK (skipped=1)
2022-05-18T04:41:13.6668250Z 
2022-05-18T04:41:13.6668376Z Generating XML reports...
2022-05-18T04:41:13.6710351Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044113.xml
2022-05-18T04:41:14.9215324Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:41:14.9229143Z 
2022-05-18T04:41:14.9229592Z Running tests...
2022-05-18T04:41:14.9230557Z ----------------------------------------------------------------------
2022-05-18T04:41:16.5287088Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:16.5644727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22035
2022-05-18T04:41:16.5750803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22036
2022-05-18T04:41:17.7555276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:17.7682113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:17.7683516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:17.7760377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:17.7766664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:17.8698339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:19.0661171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4r01mbfo
2022-05-18T04:41:19.0662095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4r01mbfo/_remote_module_non_scriptable.py
2022-05-18T04:41:19.1906537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1w1ydb9
2022-05-18T04:41:19.1907706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1w1ydb9/_remote_module_non_scriptable.py
2022-05-18T04:41:20.7868473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:20.7871341Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:20.8094595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:20.8099153Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:20.8390288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:20.8395569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:20.8615102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:20.8619615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:20.9888687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:20.9893466Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:21.0113137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:21.0117772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:21.5863492Z ok (6.663s)
2022-05-18T04:41:21.5863731Z 
2022-05-18T04:41:21.5864352Z ----------------------------------------------------------------------
2022-05-18T04:41:21.5864733Z Ran 1 test in 6.663s
2022-05-18T04:41:21.5864901Z 
2022-05-18T04:41:21.5864975Z OK
2022-05-18T04:41:21.5865110Z 
2022-05-18T04:41:21.5865245Z Generating XML reports...
2022-05-18T04:41:21.5922686Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044114.xml
2022-05-18T04:41:23.0367759Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:41:23.0383999Z 
2022-05-18T04:41:23.0384246Z Running tests...
2022-05-18T04:41:23.0384687Z ----------------------------------------------------------------------
2022-05-18T04:41:24.7105820Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:24.7465187Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22162
2022-05-18T04:41:24.7571511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22163
2022-05-18T04:41:25.9240724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:25.9296898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:25.9298392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:25.9341626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:25.9348064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:26.0312996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:27.2203680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplywx0105
2022-05-18T04:41:27.2204820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplywx0105/_remote_module_non_scriptable.py
2022-05-18T04:41:27.2903981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgwg6qqx
2022-05-18T04:41:27.2905113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgwg6qqx/_remote_module_non_scriptable.py
2022-05-18T04:41:28.3423673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:28.3424690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:28.3562876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:28.3564368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:28.6665202Z ok (5.628s)
2022-05-18T04:41:28.6665438Z 
2022-05-18T04:41:28.6665809Z ----------------------------------------------------------------------
2022-05-18T04:41:28.6666148Z Ran 1 test in 5.628s
2022-05-18T04:41:28.6666315Z 
2022-05-18T04:41:28.6666418Z OK
2022-05-18T04:41:28.6666555Z 
2022-05-18T04:41:28.6666690Z Generating XML reports...
2022-05-18T04:41:28.6722515Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044123.xml
2022-05-18T04:41:30.1031058Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:41:30.1045599Z 
2022-05-18T04:41:30.1046037Z Running tests...
2022-05-18T04:41:30.1046524Z ----------------------------------------------------------------------
2022-05-18T04:41:31.7389906Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:31.7758408Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22289
2022-05-18T04:41:31.7865417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22290
2022-05-18T04:41:32.9012713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:32.9340808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:32.9341612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:32.9417560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:32.9424024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:33.0355489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:34.2010605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvso78i1d
2022-05-18T04:41:34.2011425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvso78i1d/_remote_module_non_scriptable.py
2022-05-18T04:41:34.3352868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr64l3j3b
2022-05-18T04:41:34.3353466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr64l3j3b/_remote_module_non_scriptable.py
2022-05-18T04:41:35.2462392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:35.2462937Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:35.2618898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:35.2619412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:35.5957177Z ok (5.491s)
2022-05-18T04:41:35.5957404Z 
2022-05-18T04:41:35.5957790Z ----------------------------------------------------------------------
2022-05-18T04:41:35.5958135Z Ran 1 test in 5.491s
2022-05-18T04:41:35.5958308Z 
2022-05-18T04:41:35.5958404Z OK
2022-05-18T04:41:35.5958540Z 
2022-05-18T04:41:35.5958676Z Generating XML reports...
2022-05-18T04:41:35.6015091Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044130.xml
2022-05-18T04:41:37.0393367Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:41:37.0408359Z 
2022-05-18T04:41:37.0408801Z Running tests...
2022-05-18T04:41:37.0409272Z ----------------------------------------------------------------------
2022-05-18T04:41:38.6871715Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:38.7231525Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22416
2022-05-18T04:41:38.7337938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22417
2022-05-18T04:41:39.8527395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:39.8816556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:39.8817352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:39.8831055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:39.8837573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:39.9832646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:41.1564951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxvt8pth
2022-05-18T04:41:41.1565544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxvt8pth/_remote_module_non_scriptable.py
2022-05-18T04:41:41.3138912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplc9k_8jz
2022-05-18T04:41:41.3139719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplc9k_8jz/_remote_module_non_scriptable.py
2022-05-18T04:41:42.1949637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:42.1950229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:42.7431796Z ok (5.702s)
2022-05-18T04:41:42.7432042Z 
2022-05-18T04:41:42.7432434Z ----------------------------------------------------------------------
2022-05-18T04:41:42.7432777Z Ran 1 test in 5.702s
2022-05-18T04:41:42.7432993Z 
2022-05-18T04:41:42.7433097Z OK
2022-05-18T04:41:42.7433212Z 
2022-05-18T04:41:42.7433351Z Generating XML reports...
2022-05-18T04:41:42.7490596Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044137.xml
2022-05-18T04:41:44.1651813Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:41:44.1666962Z 
2022-05-18T04:41:44.1667281Z Running tests...
2022-05-18T04:41:44.1667731Z ----------------------------------------------------------------------
2022-05-18T04:41:45.7743013Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:45.8102657Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22543
2022-05-18T04:41:45.8209532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22544
2022-05-18T04:41:46.9724752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:46.9904236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:46.9905084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:46.9926802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:46.9933543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:47.0919177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:48.2832337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1m74pwi3
2022-05-18T04:41:48.2833132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1m74pwi3/_remote_module_non_scriptable.py
2022-05-18T04:41:48.3712205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jvr9rfh
2022-05-18T04:41:48.3713399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jvr9rfh/_remote_module_non_scriptable.py
2022-05-18T04:41:49.8530790Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:49.8531315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:49.8747384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:49.8749866Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:50.3316210Z ok (6.165s)
2022-05-18T04:41:50.3316513Z 
2022-05-18T04:41:50.3317246Z ----------------------------------------------------------------------
2022-05-18T04:41:50.3317842Z Ran 1 test in 6.165s
2022-05-18T04:41:50.3318011Z 
2022-05-18T04:41:50.3318564Z OK
2022-05-18T04:41:50.3318824Z 
2022-05-18T04:41:50.3319898Z Generating XML reports...
2022-05-18T04:41:50.3374458Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044144.xml
2022-05-18T04:41:51.7579488Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:41:51.7594296Z 
2022-05-18T04:41:51.7594567Z Running tests...
2022-05-18T04:41:51.7595008Z ----------------------------------------------------------------------
2022-05-18T04:41:53.3918752Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:53.4278565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22670
2022-05-18T04:41:53.4382191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22671
2022-05-18T04:41:54.5980499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:54.6173129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:54.6173932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:54.6182948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:54.6189424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:54.7189610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:55.9125930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2kdv3hn
2022-05-18T04:41:55.9126793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2kdv3hn/_remote_module_non_scriptable.py
2022-05-18T04:41:56.0265711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtzjx12m
2022-05-18T04:41:56.0266269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtzjx12m/_remote_module_non_scriptable.py
2022-05-18T04:41:57.3744756Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:57.3746182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:57.3923162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:57.3925792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:41:57.7482186Z ok (5.988s)
2022-05-18T04:41:57.7482509Z 
2022-05-18T04:41:57.7483253Z ----------------------------------------------------------------------
2022-05-18T04:41:57.7483861Z Ran 1 test in 5.989s
2022-05-18T04:41:57.7484031Z 
2022-05-18T04:41:57.7484127Z OK
2022-05-18T04:41:57.7484264Z 
2022-05-18T04:41:57.7484400Z Generating XML reports...
2022-05-18T04:41:57.7542650Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044151.xml
2022-05-18T04:41:59.1682968Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:41:59.1698144Z 
2022-05-18T04:41:59.1698464Z Running tests...
2022-05-18T04:41:59.1698909Z ----------------------------------------------------------------------
2022-05-18T04:42:00.7996208Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:00.8361679Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22797
2022-05-18T04:42:00.8468556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22798
2022-05-18T04:42:02.0147277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:02.0270104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:02.0270952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:02.0350024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:02.0356542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:02.1285159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:03.3248218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjm6e90i
2022-05-18T04:42:03.3248828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjm6e90i/_remote_module_non_scriptable.py
2022-05-18T04:42:03.4489145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxn2x__fq
2022-05-18T04:42:03.4490235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxn2x__fq/_remote_module_non_scriptable.py
2022-05-18T04:42:04.5059985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:42:04.5060540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:42:04.5202818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:42:04.5204367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:42:04.8564547Z ok (5.686s)
2022-05-18T04:42:04.8564888Z 
2022-05-18T04:42:04.8565401Z ----------------------------------------------------------------------
2022-05-18T04:42:04.8565878Z Ran 1 test in 5.687s
2022-05-18T04:42:04.8566028Z 
2022-05-18T04:42:04.8566124Z OK
2022-05-18T04:42:04.8566260Z 
2022-05-18T04:42:04.8566393Z Generating XML reports...
2022-05-18T04:42:04.8623083Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044159.xml
2022-05-18T04:42:06.2928131Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:42:06.2943617Z 
2022-05-18T04:42:06.2944013Z Running tests...
2022-05-18T04:42:06.2944466Z ----------------------------------------------------------------------
2022-05-18T04:42:07.9462630Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:07.9583168Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.664s)
2022-05-18T04:42:07.9583822Z 
2022-05-18T04:42:07.9584108Z ----------------------------------------------------------------------
2022-05-18T04:42:07.9584446Z Ran 1 test in 1.664s
2022-05-18T04:42:07.9584607Z 
2022-05-18T04:42:07.9584713Z OK (skipped=1)
2022-05-18T04:42:07.9584868Z 
2022-05-18T04:42:07.9584994Z Generating XML reports...
2022-05-18T04:42:07.9622632Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044206.xml
2022-05-18T04:42:09.3562192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:42:09.3578075Z 
2022-05-18T04:42:09.3578455Z Running tests...
2022-05-18T04:42:09.3578876Z ----------------------------------------------------------------------
2022-05-18T04:42:10.9999145Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:11.0367601Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22960
2022-05-18T04:42:11.0476794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22961
2022-05-18T04:42:12.2149846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:12.2346341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:12.2347177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:12.2352197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:12.2359245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:12.3359111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:12.5525465Z ok (3.194s)
2022-05-18T04:42:12.5525693Z 
2022-05-18T04:42:12.5526071Z ----------------------------------------------------------------------
2022-05-18T04:42:12.5526406Z Ran 1 test in 3.195s
2022-05-18T04:42:12.5526577Z 
2022-05-18T04:42:12.5526652Z OK
2022-05-18T04:42:12.5526806Z 
2022-05-18T04:42:12.5526940Z Generating XML reports...
2022-05-18T04:42:12.5584581Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044209.xml
2022-05-18T04:42:13.9804369Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:42:13.9819717Z 
2022-05-18T04:42:13.9820165Z Running tests...
2022-05-18T04:42:13.9820670Z ----------------------------------------------------------------------
2022-05-18T04:42:15.6340187Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:15.6461062Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.664s)
2022-05-18T04:42:15.6461650Z 
2022-05-18T04:42:15.6461927Z ----------------------------------------------------------------------
2022-05-18T04:42:15.6462255Z Ran 1 test in 1.664s
2022-05-18T04:42:15.6462428Z 
2022-05-18T04:42:15.6462540Z OK (skipped=1)
2022-05-18T04:42:15.6462695Z 
2022-05-18T04:42:15.6462801Z Generating XML reports...
2022-05-18T04:42:15.6501040Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044213.xml
2022-05-18T04:42:17.0415403Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:42:17.0429934Z 
2022-05-18T04:42:17.0430165Z Running tests...
2022-05-18T04:42:17.0430605Z ----------------------------------------------------------------------
2022-05-18T04:42:18.6978206Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:18.7338851Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23109
2022-05-18T04:42:18.7445671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23110
2022-05-18T04:42:19.9385809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:19.9428248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:19.9429022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:19.9487123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:19.9493623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:20.0443120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:22.7540380Z ok (5.711s)
2022-05-18T04:42:22.7540588Z 
2022-05-18T04:42:22.7541248Z ----------------------------------------------------------------------
2022-05-18T04:42:22.7541612Z Ran 1 test in 5.711s
2022-05-18T04:42:22.7541779Z 
2022-05-18T04:42:22.7541874Z OK
2022-05-18T04:42:22.7542031Z 
2022-05-18T04:42:22.7542148Z Generating XML reports...
2022-05-18T04:42:22.7598293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044217.xml
2022-05-18T04:42:24.1838046Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:42:24.1854022Z 
2022-05-18T04:42:24.1854570Z Running tests...
2022-05-18T04:42:24.1855046Z ----------------------------------------------------------------------
2022-05-18T04:42:24.1876550Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-05-18T04:42:24.1877053Z 
2022-05-18T04:42:24.1877350Z ----------------------------------------------------------------------
2022-05-18T04:42:24.1877717Z Ran 1 test in 0.002s
2022-05-18T04:42:24.1877867Z 
2022-05-18T04:42:24.1877983Z OK (skipped=1)
2022-05-18T04:42:24.1878142Z 
2022-05-18T04:42:24.1878268Z Generating XML reports...
2022-05-18T04:42:24.1920761Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044224.xml
2022-05-18T04:42:25.4595138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:42:25.4610385Z 
2022-05-18T04:42:25.4610746Z Running tests...
2022-05-18T04:42:25.4611189Z ----------------------------------------------------------------------
2022-05-18T04:42:25.4630229Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:42:27.1016492Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:27.1384924Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23267
2022-05-18T04:42:27.1491633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23268
2022-05-18T04:42:28.3273418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:28.3628507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:28.3629291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:28.3678494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:28.3684960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:28.4643655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:29.6786983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmsk4b83
2022-05-18T04:42:29.6787924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmsk4b83/_remote_module_non_scriptable.py
2022-05-18T04:42:29.7648247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjys2a4v8
2022-05-18T04:42:29.7649217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjys2a4v8/_remote_module_non_scriptable.py
2022-05-18T04:42:31.3589734Z ok (5.898s)
2022-05-18T04:42:31.3590052Z 
2022-05-18T04:42:31.3590458Z ----------------------------------------------------------------------
2022-05-18T04:42:31.3590789Z Ran 1 test in 5.898s
2022-05-18T04:42:31.3590957Z 
2022-05-18T04:42:31.3591051Z OK
2022-05-18T04:42:31.3591188Z 
2022-05-18T04:42:31.3591322Z Generating XML reports...
2022-05-18T04:42:31.3648496Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044225.xml
2022-05-18T04:42:32.8148201Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:42:32.8163163Z 
2022-05-18T04:42:32.8163890Z Running tests...
2022-05-18T04:42:32.8164440Z ----------------------------------------------------------------------
2022-05-18T04:42:32.8188527Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:42:34.4830839Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:34.5200703Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23394
2022-05-18T04:42:34.5308387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23395
2022-05-18T04:42:35.6919726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:35.7163666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:35.7164706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:35.7223412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:35.7230595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:35.8179325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:37.0082651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5uel5cne
2022-05-18T04:42:37.0083264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5uel5cne/_remote_module_non_scriptable.py
2022-05-18T04:42:37.1100882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7x0aj7a2
2022-05-18T04:42:37.1101933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7x0aj7a2/_remote_module_non_scriptable.py
2022-05-18T04:42:38.4529158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:42:38.4529991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:42:38.8408114Z ok (6.024s)
2022-05-18T04:42:38.8408309Z 
2022-05-18T04:42:38.8408684Z ----------------------------------------------------------------------
2022-05-18T04:42:38.8409046Z Ran 1 test in 6.024s
2022-05-18T04:42:38.8409210Z 
2022-05-18T04:42:38.8409284Z OK
2022-05-18T04:42:38.8409425Z 
2022-05-18T04:42:38.8409923Z Generating XML reports...
2022-05-18T04:42:38.8468557Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044232.xml
2022-05-18T04:42:40.2788094Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:42:40.2803413Z 
2022-05-18T04:42:40.2803843Z Running tests...
2022-05-18T04:42:40.2804347Z ----------------------------------------------------------------------
2022-05-18T04:42:40.2830224Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:42:41.9202239Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:41.9571829Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23521
2022-05-18T04:42:41.9675703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23522
2022-05-18T04:42:43.1002223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:43.1196092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:43.1196903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:43.1203875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:43.1210744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:43.2211995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:44.4166154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphy9vq5ox
2022-05-18T04:42:44.4166799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphy9vq5ox/_remote_module_non_scriptable.py
2022-05-18T04:42:44.5285550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnzat9tjj
2022-05-18T04:42:44.5286156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnzat9tjj/_remote_module_non_scriptable.py
2022-05-18T04:42:45.7439288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:42:45.7439883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:42:46.0770995Z ok (5.796s)
2022-05-18T04:42:46.0772556Z 
2022-05-18T04:42:46.0773193Z ----------------------------------------------------------------------
2022-05-18T04:42:46.0773559Z Ran 1 test in 5.797s
2022-05-18T04:42:46.0773729Z 
2022-05-18T04:42:46.0773806Z OK
2022-05-18T04:42:46.0773942Z 
2022-05-18T04:42:46.0774364Z Generating XML reports...
2022-05-18T04:42:46.0830818Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044240.xml
2022-05-18T04:42:47.5237542Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:42:47.5253626Z 
2022-05-18T04:42:47.5253995Z Running tests...
2022-05-18T04:42:47.5254413Z ----------------------------------------------------------------------
2022-05-18T04:42:47.5273763Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:42:49.1867727Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:49.2234531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23648
2022-05-18T04:42:49.2342621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23649
2022-05-18T04:42:50.3632222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:50.3766891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:50.3767696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:50.3834617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:50.3841459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:50.4782774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:51.6956893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehly8tjd
2022-05-18T04:42:51.6957740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehly8tjd/_remote_module_non_scriptable.py
2022-05-18T04:42:51.7969355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp941m85tu
2022-05-18T04:42:51.7970699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp941m85tu/_remote_module_non_scriptable.py
2022-05-18T04:42:53.5444692Z ok (6.019s)
2022-05-18T04:42:53.5444962Z 
2022-05-18T04:42:53.5445580Z ----------------------------------------------------------------------
2022-05-18T04:42:53.5446193Z Ran 1 test in 6.019s
2022-05-18T04:42:53.5446469Z 
2022-05-18T04:42:53.5448445Z OK
2022-05-18T04:42:53.5448901Z 
2022-05-18T04:42:53.5449201Z Generating XML reports...
2022-05-18T04:42:53.5503309Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044247.xml
2022-05-18T04:42:54.9826093Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:42:54.9841017Z 
2022-05-18T04:42:54.9841335Z Running tests...
2022-05-18T04:42:54.9841793Z ----------------------------------------------------------------------
2022-05-18T04:42:54.9862209Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:42:54.9862675Z 
2022-05-18T04:42:54.9862959Z ----------------------------------------------------------------------
2022-05-18T04:42:54.9863277Z Ran 1 test in 0.002s
2022-05-18T04:42:54.9863442Z 
2022-05-18T04:42:54.9863552Z OK (skipped=1)
2022-05-18T04:42:54.9863709Z 
2022-05-18T04:42:54.9863837Z Generating XML reports...
2022-05-18T04:42:54.9906416Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044254.xml
2022-05-18T04:42:56.2645676Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:42:56.2661445Z 
2022-05-18T04:42:56.2661621Z Running tests...
2022-05-18T04:42:56.2662833Z ----------------------------------------------------------------------
2022-05-18T04:42:56.2683199Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:42:56.2684310Z 
2022-05-18T04:42:56.2684920Z ----------------------------------------------------------------------
2022-05-18T04:42:56.2685586Z Ran 1 test in 0.002s
2022-05-18T04:42:56.2685752Z 
2022-05-18T04:42:56.2685843Z OK (skipped=1)
2022-05-18T04:42:56.2685998Z 
2022-05-18T04:42:56.2686122Z Generating XML reports...
2022-05-18T04:42:56.2727850Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044256.xml
2022-05-18T04:42:57.5392159Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:42:57.5406982Z 
2022-05-18T04:42:57.5407173Z Running tests...
2022-05-18T04:42:57.5408121Z ----------------------------------------------------------------------
2022-05-18T04:42:57.5428386Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:42:57.5428967Z 
2022-05-18T04:42:57.5429453Z ----------------------------------------------------------------------
2022-05-18T04:42:57.5430149Z Ran 1 test in 0.002s
2022-05-18T04:42:57.5430472Z 
2022-05-18T04:42:57.5430674Z OK (skipped=1)
2022-05-18T04:42:57.5430929Z 
2022-05-18T04:42:57.5431055Z Generating XML reports...
2022-05-18T04:42:57.5473868Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044257.xml
2022-05-18T04:42:58.8142745Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:42:58.8157696Z 
2022-05-18T04:42:58.8158151Z Running tests...
2022-05-18T04:42:58.8158660Z ----------------------------------------------------------------------
2022-05-18T04:42:58.8179074Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:42:58.8179419Z 
2022-05-18T04:42:58.8179705Z ----------------------------------------------------------------------
2022-05-18T04:42:58.8180031Z Ran 1 test in 0.002s
2022-05-18T04:42:58.8180193Z 
2022-05-18T04:42:58.8180306Z OK (skipped=1)
2022-05-18T04:42:58.8180463Z 
2022-05-18T04:42:58.8180590Z Generating XML reports...
2022-05-18T04:42:58.8223281Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044258.xml
2022-05-18T04:43:00.0908313Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:00.0923124Z 
2022-05-18T04:43:00.0923507Z Running tests...
2022-05-18T04:43:00.0923998Z ----------------------------------------------------------------------
2022-05-18T04:43:00.0944976Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:43:00.0945806Z 
2022-05-18T04:43:00.0946124Z ----------------------------------------------------------------------
2022-05-18T04:43:00.0946456Z Ran 1 test in 0.002s
2022-05-18T04:43:00.0946620Z 
2022-05-18T04:43:00.0946738Z OK (skipped=1)
2022-05-18T04:43:00.0946892Z 
2022-05-18T04:43:00.0947021Z Generating XML reports...
2022-05-18T04:43:00.0989146Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044300.xml
2022-05-18T04:43:01.3576669Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:01.3590880Z 
2022-05-18T04:43:01.3591025Z Running tests...
2022-05-18T04:43:01.3591730Z ----------------------------------------------------------------------
2022-05-18T04:43:01.3624168Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s)
2022-05-18T04:43:01.3624838Z 
2022-05-18T04:43:01.3625123Z ----------------------------------------------------------------------
2022-05-18T04:43:01.3625458Z Ran 1 test in 0.003s
2022-05-18T04:43:01.3625890Z 
2022-05-18T04:43:01.3626000Z OK (skipped=1)
2022-05-18T04:43:01.3626156Z 
2022-05-18T04:43:01.3626283Z Generating XML reports...
2022-05-18T04:43:01.3666556Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044301.xml
2022-05-18T04:43:02.6107457Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:02.6123709Z 
2022-05-18T04:43:02.6124141Z Running tests...
2022-05-18T04:43:02.6124581Z ----------------------------------------------------------------------
2022-05-18T04:43:02.6145457Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:43:02.6146218Z 
2022-05-18T04:43:02.6146694Z ----------------------------------------------------------------------
2022-05-18T04:43:02.6147029Z Ran 1 test in 0.002s
2022-05-18T04:43:02.6147196Z 
2022-05-18T04:43:02.6147305Z OK (skipped=1)
2022-05-18T04:43:02.6147448Z 
2022-05-18T04:43:02.6147586Z Generating XML reports...
2022-05-18T04:43:02.6190818Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044302.xml
2022-05-18T04:43:03.8899933Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:03.8914705Z 
2022-05-18T04:43:03.8915238Z Running tests...
2022-05-18T04:43:03.8915728Z ----------------------------------------------------------------------
2022-05-18T04:43:05.5322498Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:05.5694316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24020
2022-05-18T04:43:05.5801706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24021
2022-05-18T04:43:06.7487653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:43:06.7756142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:43:06.7756973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:06.7791206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:06.7797804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:06.8771333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:10.7930302Z ok (6.901s)
2022-05-18T04:43:10.7930550Z 
2022-05-18T04:43:10.7930956Z ----------------------------------------------------------------------
2022-05-18T04:43:10.7931536Z Ran 1 test in 6.902s
2022-05-18T04:43:10.7931728Z 
2022-05-18T04:43:10.7931822Z OK
2022-05-18T04:43:10.7931958Z 
2022-05-18T04:43:10.7932096Z Generating XML reports...
2022-05-18T04:43:10.7989004Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044303.xml
2022-05-18T04:43:12.2378196Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:12.2393102Z 
2022-05-18T04:43:12.2393484Z Running tests...
2022-05-18T04:43:12.2393915Z ----------------------------------------------------------------------
2022-05-18T04:43:13.8930617Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:13.9292532Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24148
2022-05-18T04:43:13.9401430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24149
2022-05-18T04:43:15.0831866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:43:15.0881013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:43:15.0882045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:15.0933312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:15.0940295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:15.1897096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:19.0518252Z ok (6.812s)
2022-05-18T04:43:19.0518476Z 
2022-05-18T04:43:19.0518873Z ----------------------------------------------------------------------
2022-05-18T04:43:19.0519192Z Ran 1 test in 6.812s
2022-05-18T04:43:19.0519358Z 
2022-05-18T04:43:19.0519467Z OK
2022-05-18T04:43:19.0519604Z 
2022-05-18T04:43:19.0519737Z Generating XML reports...
2022-05-18T04:43:19.0577364Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044312.xml
2022-05-18T04:43:20.4992324Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:20.5007546Z 
2022-05-18T04:43:20.5007788Z Running tests...
2022-05-18T04:43:20.5008232Z ----------------------------------------------------------------------
2022-05-18T04:43:20.5028285Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:43:20.5029075Z 
2022-05-18T04:43:20.5029550Z ----------------------------------------------------------------------
2022-05-18T04:43:20.5029888Z Ran 1 test in 0.002s
2022-05-18T04:43:20.5030082Z 
2022-05-18T04:43:20.5030196Z OK (skipped=1)
2022-05-18T04:43:20.5030331Z 
2022-05-18T04:43:20.5030481Z Generating XML reports...
2022-05-18T04:43:20.5072588Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044320.xml
2022-05-18T04:43:21.7474328Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:21.7489582Z 
2022-05-18T04:43:21.7489736Z Running tests...
2022-05-18T04:43:21.7490182Z ----------------------------------------------------------------------
2022-05-18T04:43:21.7511344Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:43:21.7511661Z 
2022-05-18T04:43:21.7511953Z ----------------------------------------------------------------------
2022-05-18T04:43:21.7512264Z Ran 1 test in 0.002s
2022-05-18T04:43:21.7512428Z 
2022-05-18T04:43:21.7512545Z OK (skipped=1)
2022-05-18T04:43:21.7512704Z 
2022-05-18T04:43:21.7512833Z Generating XML reports...
2022-05-18T04:43:21.7556705Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044321.xml
2022-05-18T04:43:23.0426750Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:23.0441876Z 
2022-05-18T04:43:23.0442020Z Running tests...
2022-05-18T04:43:23.0442779Z ----------------------------------------------------------------------
2022-05-18T04:43:24.7292560Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:24.7666777Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24346
2022-05-18T04:43:24.7776539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24347
2022-05-18T04:43:25.9250464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:43:25.9334257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:43:25.9335066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:25.9351428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:25.9358446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:26.0350069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:28.1863954Z ok (5.142s)
2022-05-18T04:43:28.1864192Z 
2022-05-18T04:43:28.1864571Z ----------------------------------------------------------------------
2022-05-18T04:43:28.1864913Z Ran 1 test in 5.142s
2022-05-18T04:43:28.1865086Z 
2022-05-18T04:43:28.1865181Z OK
2022-05-18T04:43:28.1865296Z 
2022-05-18T04:43:28.1865433Z Generating XML reports...
2022-05-18T04:43:28.1922230Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044323.xml
2022-05-18T04:43:29.6363157Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:29.6378894Z 
2022-05-18T04:43:29.6379167Z Running tests...
2022-05-18T04:43:29.6379881Z ----------------------------------------------------------------------
2022-05-18T04:43:31.2845888Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:31.3216008Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24472
2022-05-18T04:43:31.3322737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24473
2022-05-18T04:43:32.4819783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:43:32.4888781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:43:32.4889822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:32.4920970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:32.4927470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:32.5904389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:34.7407806Z ok (5.103s)
2022-05-18T04:43:34.7408261Z 
2022-05-18T04:43:34.7408762Z ----------------------------------------------------------------------
2022-05-18T04:43:34.7409365Z Ran 1 test in 5.103s
2022-05-18T04:43:34.7409854Z 
2022-05-18T04:43:34.7409962Z OK
2022-05-18T04:43:34.7410106Z 
2022-05-18T04:43:34.7410220Z Generating XML reports...
2022-05-18T04:43:34.7469533Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044329.xml
2022-05-18T04:43:36.1889419Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:36.1904066Z 
2022-05-18T04:43:36.1904734Z Running tests...
2022-05-18T04:43:36.1905355Z ----------------------------------------------------------------------
2022-05-18T04:43:37.8429068Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:37.8796736Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24598
2022-05-18T04:43:37.8904180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24599
2022-05-18T04:43:38.9800193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:43:39.0394059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:43:39.0394898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:39.0407512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:39.0414972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:39.1408769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:41.8009273Z ok (5.610s)
2022-05-18T04:43:41.8009488Z 
2022-05-18T04:43:41.8010119Z ----------------------------------------------------------------------
2022-05-18T04:43:41.8010460Z Ran 1 test in 5.610s
2022-05-18T04:43:41.8010625Z 
2022-05-18T04:43:41.8010702Z OK
2022-05-18T04:43:41.8010837Z 
2022-05-18T04:43:41.8010970Z Generating XML reports...
2022-05-18T04:43:41.8068051Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044336.xml
2022-05-18T04:43:43.2421331Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:43.2436257Z 
2022-05-18T04:43:43.2436562Z Running tests...
2022-05-18T04:43:43.2437011Z ----------------------------------------------------------------------
2022-05-18T04:43:44.8824949Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:44.9194065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24721
2022-05-18T04:43:44.9298641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24722
2022-05-18T04:43:46.0516219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:43:46.0697941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:43:46.0698864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:46.0718828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:46.0725205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:46.1709925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:46.1844408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:43:46.1844923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:43:46.1845590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:43:46.1846273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:43:49.5803686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:43:49.5804504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:43:49.5805298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:43:49.5805995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:43:49.6247850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:43:49.6248410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:43:49.6249139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:43:49.6250207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:43:50.0415663Z ok (6.798s)
2022-05-18T04:43:50.0415900Z 
2022-05-18T04:43:50.0416327Z ----------------------------------------------------------------------
2022-05-18T04:43:50.0416646Z Ran 1 test in 6.798s
2022-05-18T04:43:50.0416811Z 
2022-05-18T04:43:50.0417198Z OK
2022-05-18T04:43:50.0417339Z 
2022-05-18T04:43:50.0417484Z Generating XML reports...
2022-05-18T04:43:50.0474327Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044343.xml
2022-05-18T04:43:51.4690737Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:51.4705434Z 
2022-05-18T04:43:51.4705916Z Running tests...
2022-05-18T04:43:51.4706406Z ----------------------------------------------------------------------
2022-05-18T04:43:51.4726450Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:43:51.4726806Z 
2022-05-18T04:43:51.4727096Z ----------------------------------------------------------------------
2022-05-18T04:43:51.4727422Z Ran 1 test in 0.002s
2022-05-18T04:43:51.4727582Z 
2022-05-18T04:43:51.4727690Z OK (skipped=1)
2022-05-18T04:43:51.4727828Z 
2022-05-18T04:43:51.4727965Z Generating XML reports...
2022-05-18T04:43:51.4768328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044351.xml
2022-05-18T04:43:52.7365772Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:52.7380775Z 
2022-05-18T04:43:52.7381049Z Running tests...
2022-05-18T04:43:52.7381492Z ----------------------------------------------------------------------
2022-05-18T04:43:52.7402875Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:43:52.7403226Z 
2022-05-18T04:43:52.7403497Z ----------------------------------------------------------------------
2022-05-18T04:43:52.7403837Z Ran 1 test in 0.002s
2022-05-18T04:43:52.7404003Z 
2022-05-18T04:43:52.7404112Z OK (skipped=1)
2022-05-18T04:43:52.7404246Z 
2022-05-18T04:43:52.7404371Z Generating XML reports...
2022-05-18T04:43:52.7448948Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044352.xml
2022-05-18T04:43:53.9736379Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:53.9751993Z 
2022-05-18T04:43:53.9752434Z Running tests...
2022-05-18T04:43:53.9752920Z ----------------------------------------------------------------------
2022-05-18T04:43:53.9774438Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:43:53.9774792Z 
2022-05-18T04:43:53.9775059Z ----------------------------------------------------------------------
2022-05-18T04:43:53.9775369Z Ran 1 test in 0.002s
2022-05-18T04:43:53.9775839Z 
2022-05-18T04:43:53.9775968Z OK (skipped=1)
2022-05-18T04:43:53.9776127Z 
2022-05-18T04:43:53.9776252Z Generating XML reports...
2022-05-18T04:43:53.9817784Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044353.xml
2022-05-18T04:43:55.2237837Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:55.2252169Z 
2022-05-18T04:43:55.2252375Z Running tests...
2022-05-18T04:43:55.2252817Z ----------------------------------------------------------------------
2022-05-18T04:43:55.2272944Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:43:55.2273286Z 
2022-05-18T04:43:55.2273557Z ----------------------------------------------------------------------
2022-05-18T04:43:55.2273867Z Ran 1 test in 0.002s
2022-05-18T04:43:55.2274029Z 
2022-05-18T04:43:55.2274138Z OK (skipped=1)
2022-05-18T04:43:55.2274308Z 
2022-05-18T04:43:55.2274434Z Generating XML reports...
2022-05-18T04:43:55.2315230Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044355.xml
2022-05-18T04:43:56.5043960Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:56.5060711Z 
2022-05-18T04:43:56.5061242Z Running tests...
2022-05-18T04:43:56.5061748Z ----------------------------------------------------------------------
2022-05-18T04:43:56.5083034Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:43:56.5083375Z 
2022-05-18T04:43:56.5083685Z ----------------------------------------------------------------------
2022-05-18T04:43:56.5084241Z Ran 1 test in 0.002s
2022-05-18T04:43:56.5084412Z 
2022-05-18T04:43:56.5084522Z OK (skipped=1)
2022-05-18T04:43:56.5084685Z 
2022-05-18T04:43:56.5084810Z Generating XML reports...
2022-05-18T04:43:56.5127576Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044356.xml
2022-05-18T04:43:57.7660491Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:57.7674397Z 
2022-05-18T04:43:57.7674659Z Running tests...
2022-05-18T04:43:57.7675099Z ----------------------------------------------------------------------
2022-05-18T04:43:57.7695975Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:43:57.7696317Z 
2022-05-18T04:43:57.7696588Z ----------------------------------------------------------------------
2022-05-18T04:43:57.7696919Z Ran 1 test in 0.002s
2022-05-18T04:43:57.7697084Z 
2022-05-18T04:43:57.7697191Z OK (skipped=1)
2022-05-18T04:43:57.7697329Z 
2022-05-18T04:43:57.7697453Z Generating XML reports...
2022-05-18T04:43:57.7738031Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044357.xml
2022-05-18T04:43:59.0143932Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:43:59.0157981Z 
2022-05-18T04:43:59.0158234Z Running tests...
2022-05-18T04:43:59.0158668Z ----------------------------------------------------------------------
2022-05-18T04:43:59.0179764Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:43:59.0180104Z 
2022-05-18T04:43:59.0180350Z ----------------------------------------------------------------------
2022-05-18T04:43:59.0180670Z Ran 1 test in 0.002s
2022-05-18T04:43:59.0180834Z 
2022-05-18T04:43:59.0180947Z OK (skipped=1)
2022-05-18T04:43:59.0181102Z 
2022-05-18T04:43:59.0181226Z Generating XML reports...
2022-05-18T04:43:59.0222431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044359.xml
2022-05-18T04:44:00.2906808Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:00.2921827Z 
2022-05-18T04:44:00.2922253Z Running tests...
2022-05-18T04:44:00.2922728Z ----------------------------------------------------------------------
2022-05-18T04:44:00.2943961Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:44:00.2944302Z 
2022-05-18T04:44:00.2944567Z ----------------------------------------------------------------------
2022-05-18T04:44:00.2944892Z Ran 1 test in 0.002s
2022-05-18T04:44:00.2945037Z 
2022-05-18T04:44:00.2945155Z OK (skipped=1)
2022-05-18T04:44:00.2945311Z 
2022-05-18T04:44:00.2945433Z Generating XML reports...
2022-05-18T04:44:00.2988561Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044400.xml
2022-05-18T04:44:01.5708596Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:01.5723750Z 
2022-05-18T04:44:01.5724030Z Running tests...
2022-05-18T04:44:01.5724464Z ----------------------------------------------------------------------
2022-05-18T04:44:01.5746375Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:44:01.5746705Z 
2022-05-18T04:44:01.5746969Z ----------------------------------------------------------------------
2022-05-18T04:44:01.5747296Z Ran 1 test in 0.002s
2022-05-18T04:44:01.5747455Z 
2022-05-18T04:44:01.5747564Z OK (skipped=1)
2022-05-18T04:44:01.5747701Z 
2022-05-18T04:44:01.5749535Z Generating XML reports...
2022-05-18T04:44:01.5791856Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044401.xml
2022-05-18T04:44:02.8574819Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:02.8591567Z 
2022-05-18T04:44:02.8592215Z Running tests...
2022-05-18T04:44:02.8593118Z ----------------------------------------------------------------------
2022-05-18T04:44:02.8613917Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:02.8614280Z 
2022-05-18T04:44:02.8614557Z ----------------------------------------------------------------------
2022-05-18T04:44:02.8614887Z Ran 1 test in 0.002s
2022-05-18T04:44:02.8615049Z 
2022-05-18T04:44:02.8615158Z OK (skipped=1)
2022-05-18T04:44:02.8615313Z 
2022-05-18T04:44:02.8615437Z Generating XML reports...
2022-05-18T04:44:02.8657997Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044402.xml
2022-05-18T04:44:04.1466217Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:04.1481112Z 
2022-05-18T04:44:04.1481597Z Running tests...
2022-05-18T04:44:04.1482086Z ----------------------------------------------------------------------
2022-05-18T04:44:04.1503815Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:44:04.1504279Z 
2022-05-18T04:44:04.1504559Z ----------------------------------------------------------------------
2022-05-18T04:44:04.1504886Z Ran 1 test in 0.002s
2022-05-18T04:44:04.1505049Z 
2022-05-18T04:44:04.1505156Z OK (skipped=1)
2022-05-18T04:44:04.1505292Z 
2022-05-18T04:44:04.1505414Z Generating XML reports...
2022-05-18T04:44:04.1547603Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044404.xml
2022-05-18T04:44:05.4361635Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:05.4376634Z 
2022-05-18T04:44:05.4377135Z Running tests...
2022-05-18T04:44:05.4377918Z ----------------------------------------------------------------------
2022-05-18T04:44:05.4397889Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:44:05.4398377Z 
2022-05-18T04:44:05.4398791Z ----------------------------------------------------------------------
2022-05-18T04:44:05.4399126Z Ran 1 test in 0.002s
2022-05-18T04:44:05.4399270Z 
2022-05-18T04:44:05.4399380Z OK (skipped=1)
2022-05-18T04:44:05.4399532Z 
2022-05-18T04:44:05.4399655Z Generating XML reports...
2022-05-18T04:44:05.4441614Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044405.xml
2022-05-18T04:44:06.7123873Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:06.7139391Z 
2022-05-18T04:44:06.7139852Z Running tests...
2022-05-18T04:44:06.7140583Z ----------------------------------------------------------------------
2022-05-18T04:44:06.7161140Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:44:06.7161481Z 
2022-05-18T04:44:06.7162287Z ----------------------------------------------------------------------
2022-05-18T04:44:06.7162605Z Ran 1 test in 0.002s
2022-05-18T04:44:06.7162767Z 
2022-05-18T04:44:06.7162875Z OK (skipped=1)
2022-05-18T04:44:06.7163028Z 
2022-05-18T04:44:06.7163151Z Generating XML reports...
2022-05-18T04:44:06.7205160Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044406.xml
2022-05-18T04:44:07.9612054Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:07.9627386Z 
2022-05-18T04:44:07.9627770Z Running tests...
2022-05-18T04:44:07.9628286Z ----------------------------------------------------------------------
2022-05-18T04:44:07.9652507Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:07.9652843Z 
2022-05-18T04:44:07.9653126Z ----------------------------------------------------------------------
2022-05-18T04:44:07.9653440Z Ran 1 test in 0.003s
2022-05-18T04:44:07.9653605Z 
2022-05-18T04:44:07.9653719Z OK (skipped=1)
2022-05-18T04:44:07.9653871Z 
2022-05-18T04:44:07.9653994Z Generating XML reports...
2022-05-18T04:44:07.9696394Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044407.xml
2022-05-18T04:44:09.2349246Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:09.2364521Z 
2022-05-18T04:44:09.2365068Z Running tests...
2022-05-18T04:44:09.2365575Z ----------------------------------------------------------------------
2022-05-18T04:44:09.2385884Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:09.2386552Z 
2022-05-18T04:44:09.2387203Z ----------------------------------------------------------------------
2022-05-18T04:44:09.2387696Z Ran 1 test in 0.002s
2022-05-18T04:44:09.2387861Z 
2022-05-18T04:44:09.2387982Z OK (skipped=1)
2022-05-18T04:44:09.2388121Z 
2022-05-18T04:44:09.2388246Z Generating XML reports...
2022-05-18T04:44:09.2430245Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044409.xml
2022-05-18T04:44:10.4935202Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:10.4949472Z 
2022-05-18T04:44:10.4949918Z Running tests...
2022-05-18T04:44:10.4950418Z ----------------------------------------------------------------------
2022-05-18T04:44:10.4970154Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:10.4970874Z 
2022-05-18T04:44:10.4971446Z ----------------------------------------------------------------------
2022-05-18T04:44:10.4971801Z Ran 1 test in 0.002s
2022-05-18T04:44:10.4971963Z 
2022-05-18T04:44:10.4972055Z OK (skipped=1)
2022-05-18T04:44:10.4972212Z 
2022-05-18T04:44:10.4972346Z Generating XML reports...
2022-05-18T04:44:10.5012616Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044410.xml
2022-05-18T04:44:11.7666896Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:11.7681836Z 
2022-05-18T04:44:11.7682208Z Running tests...
2022-05-18T04:44:11.7682630Z ----------------------------------------------------------------------
2022-05-18T04:44:11.7704794Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:11.7705122Z 
2022-05-18T04:44:11.7705401Z ----------------------------------------------------------------------
2022-05-18T04:44:11.7705740Z Ran 1 test in 0.002s
2022-05-18T04:44:11.7705885Z 
2022-05-18T04:44:11.7705991Z OK (skipped=1)
2022-05-18T04:44:11.7706145Z 
2022-05-18T04:44:11.7706269Z Generating XML reports...
2022-05-18T04:44:11.7749373Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044411.xml
2022-05-18T04:44:13.0164559Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:13.0181489Z 
2022-05-18T04:44:13.0181932Z Running tests...
2022-05-18T04:44:13.0182431Z ----------------------------------------------------------------------
2022-05-18T04:44:13.0204139Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:13.0204619Z 
2022-05-18T04:44:13.0205264Z ----------------------------------------------------------------------
2022-05-18T04:44:13.0205933Z Ran 1 test in 0.002s
2022-05-18T04:44:13.0206249Z 
2022-05-18T04:44:13.0206477Z OK (skipped=1)
2022-05-18T04:44:13.0206792Z 
2022-05-18T04:44:13.0207006Z Generating XML reports...
2022-05-18T04:44:13.0251638Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044413.xml
2022-05-18T04:44:14.2958876Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:14.2974966Z 
2022-05-18T04:44:14.2975180Z Running tests...
2022-05-18T04:44:14.2975886Z ----------------------------------------------------------------------
2022-05-18T04:44:14.2997320Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:14.2997635Z 
2022-05-18T04:44:14.2997966Z ----------------------------------------------------------------------
2022-05-18T04:44:14.2998528Z Ran 1 test in 0.002s
2022-05-18T04:44:14.2998723Z 
2022-05-18T04:44:14.2998835Z OK (skipped=1)
2022-05-18T04:44:14.2998993Z 
2022-05-18T04:44:14.2999122Z Generating XML reports...
2022-05-18T04:44:14.3042195Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044414.xml
2022-05-18T04:44:15.5689297Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:15.5705138Z 
2022-05-18T04:44:15.5705423Z Running tests...
2022-05-18T04:44:15.5706116Z ----------------------------------------------------------------------
2022-05-18T04:44:15.5727121Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:15.5727430Z 
2022-05-18T04:44:15.5728151Z ----------------------------------------------------------------------
2022-05-18T04:44:15.5728534Z Ran 1 test in 0.002s
2022-05-18T04:44:15.5728697Z 
2022-05-18T04:44:15.5728813Z OK (skipped=1)
2022-05-18T04:44:15.5728968Z 
2022-05-18T04:44:15.5729095Z Generating XML reports...
2022-05-18T04:44:15.5772250Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044415.xml
2022-05-18T04:44:16.8437193Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:16.8453598Z 
2022-05-18T04:44:16.8454013Z Running tests...
2022-05-18T04:44:16.8454496Z ----------------------------------------------------------------------
2022-05-18T04:44:16.8477094Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:16.8477412Z 
2022-05-18T04:44:16.8477673Z ----------------------------------------------------------------------
2022-05-18T04:44:16.8478007Z Ran 1 test in 0.002s
2022-05-18T04:44:16.8478175Z 
2022-05-18T04:44:16.8478285Z OK (skipped=1)
2022-05-18T04:44:16.8478445Z 
2022-05-18T04:44:16.8478569Z Generating XML reports...
2022-05-18T04:44:16.8522100Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044416.xml
2022-05-18T04:44:18.1196395Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:18.1212258Z 
2022-05-18T04:44:18.1212462Z Running tests...
2022-05-18T04:44:18.1213106Z ----------------------------------------------------------------------
2022-05-18T04:44:18.1235330Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:18.1235840Z 
2022-05-18T04:44:18.1236565Z ----------------------------------------------------------------------
2022-05-18T04:44:18.1237228Z Ran 1 test in 0.002s
2022-05-18T04:44:18.1237544Z 
2022-05-18T04:44:18.1237769Z OK (skipped=1)
2022-05-18T04:44:18.1238064Z 
2022-05-18T04:44:18.1238298Z Generating XML reports...
2022-05-18T04:44:18.1282101Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044418.xml
2022-05-18T04:44:19.3735523Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:19.3750280Z 
2022-05-18T04:44:19.3750427Z Running tests...
2022-05-18T04:44:19.3751145Z ----------------------------------------------------------------------
2022-05-18T04:44:19.3770877Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:19.3771187Z 
2022-05-18T04:44:19.3771482Z ----------------------------------------------------------------------
2022-05-18T04:44:19.3771812Z Ran 1 test in 0.002s
2022-05-18T04:44:19.3771963Z 
2022-05-18T04:44:19.3772071Z OK (skipped=1)
2022-05-18T04:44:19.3772226Z 
2022-05-18T04:44:19.3772351Z Generating XML reports...
2022-05-18T04:44:19.3813519Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044419.xml
2022-05-18T04:44:20.6205446Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:20.6220070Z 
2022-05-18T04:44:20.6220455Z Running tests...
2022-05-18T04:44:20.6220995Z ----------------------------------------------------------------------
2022-05-18T04:44:20.6241277Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:20.6241609Z 
2022-05-18T04:44:20.6241900Z ----------------------------------------------------------------------
2022-05-18T04:44:20.6242209Z Ran 1 test in 0.002s
2022-05-18T04:44:20.6242370Z 
2022-05-18T04:44:20.6242478Z OK (skipped=1)
2022-05-18T04:44:20.6242630Z 
2022-05-18T04:44:20.6242759Z Generating XML reports...
2022-05-18T04:44:20.6284069Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044420.xml
2022-05-18T04:44:21.8987925Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:21.9003310Z 
2022-05-18T04:44:21.9003623Z Running tests...
2022-05-18T04:44:21.9004342Z ----------------------------------------------------------------------
2022-05-18T04:44:21.9027991Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-05-18T04:44:21.9028439Z 
2022-05-18T04:44:21.9028910Z ----------------------------------------------------------------------
2022-05-18T04:44:21.9029227Z Ran 1 test in 0.002s
2022-05-18T04:44:21.9029391Z 
2022-05-18T04:44:21.9029506Z OK (skipped=1)
2022-05-18T04:44:21.9029662Z 
2022-05-18T04:44:21.9029785Z Generating XML reports...
2022-05-18T04:44:21.9071960Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044421.xml
2022-05-18T04:44:23.1900544Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:23.1915506Z 
2022-05-18T04:44:23.1915941Z Running tests...
2022-05-18T04:44:23.1916511Z ----------------------------------------------------------------------
2022-05-18T04:44:23.1941977Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-05-18T04:44:23.1942414Z 
2022-05-18T04:44:23.1942959Z ----------------------------------------------------------------------
2022-05-18T04:44:23.1943958Z Ran 1 test in 0.003s
2022-05-18T04:44:23.1944133Z 
2022-05-18T04:44:23.1944241Z OK (skipped=1)
2022-05-18T04:44:23.1944395Z 
2022-05-18T04:44:23.1944519Z Generating XML reports...
2022-05-18T04:44:23.1986899Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044423.xml
2022-05-18T04:44:24.4584384Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:24.4598440Z 
2022-05-18T04:44:24.4598865Z Running tests...
2022-05-18T04:44:24.4599801Z ----------------------------------------------------------------------
2022-05-18T04:44:24.4621585Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:24.4622272Z 
2022-05-18T04:44:24.4622875Z ----------------------------------------------------------------------
2022-05-18T04:44:24.4623299Z Ran 1 test in 0.002s
2022-05-18T04:44:24.4623477Z 
2022-05-18T04:44:24.4623585Z OK (skipped=1)
2022-05-18T04:44:24.4623723Z 
2022-05-18T04:44:24.4623846Z Generating XML reports...
2022-05-18T04:44:24.4665235Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044424.xml
2022-05-18T04:44:25.7347521Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:25.7362072Z 
2022-05-18T04:44:25.7362558Z Running tests...
2022-05-18T04:44:25.7363031Z ----------------------------------------------------------------------
2022-05-18T04:44:27.4016189Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:27.4373237Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25829
2022-05-18T04:44:27.4479710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25830
2022-05-18T04:44:28.6262556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:28.6427499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:28.6428341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:28.6465609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:28.6472419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:28.7442881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:31.4573753Z ok (5.721s)
2022-05-18T04:44:31.4574024Z 
2022-05-18T04:44:31.4574673Z ----------------------------------------------------------------------
2022-05-18T04:44:31.4575039Z Ran 1 test in 5.721s
2022-05-18T04:44:31.4575202Z 
2022-05-18T04:44:31.4575293Z OK
2022-05-18T04:44:31.4575437Z 
2022-05-18T04:44:31.4575552Z Generating XML reports...
2022-05-18T04:44:31.4632566Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044425.xml
2022-05-18T04:44:32.8967103Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:32.8982212Z 
2022-05-18T04:44:32.8982704Z Running tests...
2022-05-18T04:44:32.8983197Z ----------------------------------------------------------------------
2022-05-18T04:44:32.9004600Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:32.9004913Z 
2022-05-18T04:44:32.9005199Z ----------------------------------------------------------------------
2022-05-18T04:44:32.9005543Z Ran 1 test in 0.002s
2022-05-18T04:44:32.9005688Z 
2022-05-18T04:44:32.9005805Z OK (skipped=1)
2022-05-18T04:44:32.9005958Z 
2022-05-18T04:44:32.9006082Z Generating XML reports...
2022-05-18T04:44:32.9048301Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044432.xml
2022-05-18T04:44:34.1781051Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:34.1795627Z 
2022-05-18T04:44:34.1795981Z Running tests...
2022-05-18T04:44:34.1796418Z ----------------------------------------------------------------------
2022-05-18T04:44:34.1818152Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:34.1819107Z 
2022-05-18T04:44:34.1819402Z ----------------------------------------------------------------------
2022-05-18T04:44:34.1819729Z Ran 1 test in 0.002s
2022-05-18T04:44:34.1819892Z 
2022-05-18T04:44:34.1820020Z OK (skipped=1)
2022-05-18T04:44:34.1820177Z 
2022-05-18T04:44:34.1820310Z Generating XML reports...
2022-05-18T04:44:34.1862886Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044434.xml
2022-05-18T04:44:35.4616637Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:35.4631732Z 
2022-05-18T04:44:35.4631875Z Running tests...
2022-05-18T04:44:35.4632583Z ----------------------------------------------------------------------
2022-05-18T04:44:35.4655870Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:44:35.4656212Z 
2022-05-18T04:44:35.4656505Z ----------------------------------------------------------------------
2022-05-18T04:44:35.4656831Z Ran 1 test in 0.002s
2022-05-18T04:44:35.4656994Z 
2022-05-18T04:44:35.4657104Z OK (skipped=1)
2022-05-18T04:44:35.4657241Z 
2022-05-18T04:44:35.4657378Z Generating XML reports...
2022-05-18T04:44:35.4699410Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044435.xml
2022-05-18T04:44:36.7221508Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:36.7237187Z 
2022-05-18T04:44:36.7237457Z Running tests...
2022-05-18T04:44:36.7237889Z ----------------------------------------------------------------------
2022-05-18T04:44:38.3891310Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:38.4249365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26057
2022-05-18T04:44:38.4355214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26058
2022-05-18T04:44:39.5692036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:39.5846472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:39.5847313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:39.5894758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:39.5901175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:39.6861411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:41.7439636Z ok (5.020s)
2022-05-18T04:44:41.7439834Z 
2022-05-18T04:44:41.7440488Z ----------------------------------------------------------------------
2022-05-18T04:44:41.7440847Z Ran 1 test in 5.020s
2022-05-18T04:44:41.7441011Z 
2022-05-18T04:44:41.7441105Z OK
2022-05-18T04:44:41.7441239Z 
2022-05-18T04:44:41.7441354Z Generating XML reports...
2022-05-18T04:44:41.7498299Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044436.xml
2022-05-18T04:44:43.1604762Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:43.1619974Z 
2022-05-18T04:44:43.1620411Z Running tests...
2022-05-18T04:44:43.1620884Z ----------------------------------------------------------------------
2022-05-18T04:44:44.7787892Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:44.8146449Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26183
2022-05-18T04:44:44.8255658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26184
2022-05-18T04:44:46.0241612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:46.0352341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:46.0353168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:46.0444139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:46.0450568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:46.1368445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:48.2340358Z ok (5.072s)
2022-05-18T04:44:48.2340575Z 
2022-05-18T04:44:48.2340943Z ----------------------------------------------------------------------
2022-05-18T04:44:48.2341267Z Ran 1 test in 5.072s
2022-05-18T04:44:48.2341433Z 
2022-05-18T04:44:48.2341532Z OK
2022-05-18T04:44:48.2341667Z 
2022-05-18T04:44:48.2341797Z Generating XML reports...
2022-05-18T04:44:48.2397632Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044443.xml
2022-05-18T04:44:49.6727384Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:49.6742899Z 
2022-05-18T04:44:49.6743136Z Running tests...
2022-05-18T04:44:49.6743588Z ----------------------------------------------------------------------
2022-05-18T04:44:51.3191279Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:51.3559969Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26309
2022-05-18T04:44:51.3666855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26310
2022-05-18T04:44:52.5401621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:52.5706145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:52.5706971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:52.5707653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:52.5713991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:52.5714741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:54.7753723Z ok (5.101s)
2022-05-18T04:44:54.7753957Z 
2022-05-18T04:44:54.7754339Z ----------------------------------------------------------------------
2022-05-18T04:44:54.7754659Z Ran 1 test in 5.101s
2022-05-18T04:44:54.7754825Z 
2022-05-18T04:44:54.7754920Z OK
2022-05-18T04:44:54.7755053Z 
2022-05-18T04:44:54.7755186Z Generating XML reports...
2022-05-18T04:44:54.7813696Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044449.xml
2022-05-18T04:44:56.2220312Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:56.2236057Z 
2022-05-18T04:44:56.2236313Z Running tests...
2022-05-18T04:44:56.2236751Z ----------------------------------------------------------------------
2022-05-18T04:44:56.2257067Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:44:56.2257792Z 
2022-05-18T04:44:56.2258094Z ----------------------------------------------------------------------
2022-05-18T04:44:56.2258422Z Ran 1 test in 0.002s
2022-05-18T04:44:56.2258588Z 
2022-05-18T04:44:56.2258697Z OK (skipped=1)
2022-05-18T04:44:56.2258851Z 
2022-05-18T04:44:56.2258977Z Generating XML reports...
2022-05-18T04:44:56.2301622Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044456.xml
2022-05-18T04:44:57.5015545Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:57.5031446Z 
2022-05-18T04:44:57.5031849Z Running tests...
2022-05-18T04:44:57.5032343Z ----------------------------------------------------------------------
2022-05-18T04:44:57.5053525Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:44:57.5053856Z 
2022-05-18T04:44:57.5054145Z ----------------------------------------------------------------------
2022-05-18T04:44:57.5054454Z Ran 1 test in 0.002s
2022-05-18T04:44:57.5054616Z 
2022-05-18T04:44:57.5054731Z OK (skipped=1)
2022-05-18T04:44:57.5054884Z 
2022-05-18T04:44:57.5055008Z Generating XML reports...
2022-05-18T04:44:57.5098082Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044457.xml
2022-05-18T04:44:58.7818848Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:44:58.7834734Z 
2022-05-18T04:44:58.7835132Z Running tests...
2022-05-18T04:44:58.7835640Z ----------------------------------------------------------------------
2022-05-18T04:45:00.4489122Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:00.4858788Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26505
2022-05-18T04:45:00.4966535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26506
2022-05-18T04:45:01.6331291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:01.6579488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:01.6580321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:01.6635066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:01.6642931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:01.7594710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:04.4061850Z ok (5.622s)
2022-05-18T04:45:04.4062071Z 
2022-05-18T04:45:04.4062462Z ----------------------------------------------------------------------
2022-05-18T04:45:04.4062806Z Ran 1 test in 5.623s
2022-05-18T04:45:04.4062968Z 
2022-05-18T04:45:04.4063042Z OK
2022-05-18T04:45:04.4065496Z 
2022-05-18T04:45:04.4065936Z Generating XML reports...
2022-05-18T04:45:04.4120719Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044458.xml
2022-05-18T04:45:05.8475848Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:45:05.8491574Z 
2022-05-18T04:45:05.8491818Z Running tests...
2022-05-18T04:45:05.8492256Z ----------------------------------------------------------------------
2022-05-18T04:45:07.4995535Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:07.5362779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26628
2022-05-18T04:45:07.5470816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26629
2022-05-18T04:45:08.6863247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:08.6965947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:08.6966965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:08.7065663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:08.7072671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:08.7981259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:11.3563902Z ok (5.507s)
2022-05-18T04:45:11.3564129Z 
2022-05-18T04:45:11.3564520Z ----------------------------------------------------------------------
2022-05-18T04:45:11.3564853Z Ran 1 test in 5.507s
2022-05-18T04:45:11.3564997Z 
2022-05-18T04:45:11.3565090Z OK
2022-05-18T04:45:11.3565227Z 
2022-05-18T04:45:11.3565363Z Generating XML reports...
2022-05-18T04:45:11.3622732Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044505.xml
2022-05-18T04:45:12.7997235Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:45:12.8013016Z 
2022-05-18T04:45:12.8013280Z Running tests...
2022-05-18T04:45:12.8013715Z ----------------------------------------------------------------------
2022-05-18T04:45:12.8037997Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:45:12.8038302Z 
2022-05-18T04:45:12.8038583Z ----------------------------------------------------------------------
2022-05-18T04:45:12.8038903Z Ran 1 test in 0.003s
2022-05-18T04:45:12.8039064Z 
2022-05-18T04:45:12.8039181Z OK (skipped=1)
2022-05-18T04:45:12.8039340Z 
2022-05-18T04:45:12.8039467Z Generating XML reports...
2022-05-18T04:45:12.8090216Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044512.xml
2022-05-18T04:45:14.0851467Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:45:14.0867385Z 
2022-05-18T04:45:14.0867649Z Running tests...
2022-05-18T04:45:14.0868086Z ----------------------------------------------------------------------
2022-05-18T04:45:15.7465688Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:15.7834199Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26786
2022-05-18T04:45:15.7944866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26787
2022-05-18T04:45:16.9465179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:16.9720345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:16.9721113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:16.9769551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:16.9777816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:16.9781296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:45:17.0731861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:17.0736894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:45:17.0737813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:45:17.0800495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:45:19.7038831Z ok (5.617s)
2022-05-18T04:45:19.7039048Z 
2022-05-18T04:45:19.7039455Z ----------------------------------------------------------------------
2022-05-18T04:45:19.7039776Z Ran 1 test in 5.617s
2022-05-18T04:45:19.7039942Z 
2022-05-18T04:45:19.7040042Z OK
2022-05-18T04:45:19.7040183Z 
2022-05-18T04:45:19.7040316Z Generating XML reports...
2022-05-18T04:45:19.7096905Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044514.xml
2022-05-18T04:45:21.1324002Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:45:21.1339194Z 
2022-05-18T04:45:21.1339443Z Running tests...
2022-05-18T04:45:21.1339905Z ----------------------------------------------------------------------
2022-05-18T04:45:21.1359209Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:45:21.1359597Z 
2022-05-18T04:45:21.1359883Z ----------------------------------------------------------------------
2022-05-18T04:45:21.1360227Z Ran 1 test in 0.002s
2022-05-18T04:45:21.1360392Z 
2022-05-18T04:45:21.1360506Z OK (skipped=1)
2022-05-18T04:45:21.1360643Z 
2022-05-18T04:45:21.1360772Z Generating XML reports...
2022-05-18T04:45:21.1402714Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044521.xml
2022-05-18T04:45:22.3913415Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:45:22.3928193Z 
2022-05-18T04:45:22.3928600Z Running tests...
2022-05-18T04:45:22.3929094Z ----------------------------------------------------------------------
2022-05-18T04:45:24.0078940Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:24.0438936Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26948
2022-05-18T04:45:24.0549328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26949
2022-05-18T04:45:25.1642994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:25.1955492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:25.1956282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:25.2049134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:25.2056138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:25.2969976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:25.4596926Z skip: Skipped due to small world size. (3.067s)
2022-05-18T04:45:25.4597176Z 
2022-05-18T04:45:25.4597562Z ----------------------------------------------------------------------
2022-05-18T04:45:25.4597878Z Ran 1 test in 3.067s
2022-05-18T04:45:25.4598043Z 
2022-05-18T04:45:25.4598152Z OK (skipped=1)
2022-05-18T04:45:25.4598306Z 
2022-05-18T04:45:25.4598432Z Generating XML reports...
2022-05-18T04:45:25.4655628Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044522.xml
2022-05-18T04:45:26.8840831Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:45:26.8856603Z 
2022-05-18T04:45:26.8856918Z Running tests...
2022-05-18T04:45:26.8857354Z ----------------------------------------------------------------------
2022-05-18T04:45:26.8876837Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:45:26.8877175Z 
2022-05-18T04:45:26.8877459Z ----------------------------------------------------------------------
2022-05-18T04:45:26.8877788Z Ran 1 test in 0.002s
2022-05-18T04:45:26.8877952Z 
2022-05-18T04:45:26.8878067Z OK (skipped=1)
2022-05-18T04:45:26.8878222Z 
2022-05-18T04:45:26.8878327Z Generating XML reports...
2022-05-18T04:45:26.8920293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044526.xml
2022-05-18T04:45:28.1516525Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:45:28.1530842Z 
2022-05-18T04:45:28.1531396Z Running tests...
2022-05-18T04:45:28.1531904Z ----------------------------------------------------------------------
2022-05-18T04:45:28.1551467Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:45:28.1551862Z 
2022-05-18T04:45:28.1552311Z ----------------------------------------------------------------------
2022-05-18T04:45:28.1552665Z Ran 1 test in 0.002s
2022-05-18T04:45:28.1552829Z 
2022-05-18T04:45:28.1552940Z OK (skipped=1)
2022-05-18T04:45:28.1553099Z 
2022-05-18T04:45:28.1553223Z Generating XML reports...
2022-05-18T04:45:28.1593418Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044528.xml
2022-05-18T04:45:29.4282478Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:45:29.4299526Z 
2022-05-18T04:45:29.4300044Z Running tests...
2022-05-18T04:45:29.4351585Z ----------------------------------------------------------------------
2022-05-18T04:45:31.0759293Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:31.1130984Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27131
2022-05-18T04:45:31.1238458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27132
2022-05-18T04:45:32.2789457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:32.2934985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:32.2935788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:32.2992160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:32.2998859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:32.3950874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:36.4358412Z ok (7.006s)
2022-05-18T04:45:36.4358644Z 
2022-05-18T04:45:36.4359041Z ----------------------------------------------------------------------
2022-05-18T04:45:36.4359380Z Ran 1 test in 7.006s
2022-05-18T04:45:36.4359533Z 
2022-05-18T04:45:36.4359626Z OK
2022-05-18T04:45:36.4359760Z 
2022-05-18T04:45:36.4359894Z Generating XML reports...
2022-05-18T04:45:36.4416778Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044529.xml
2022-05-18T04:45:37.8758530Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:45:37.8774337Z 
2022-05-18T04:45:37.8774749Z Running tests...
2022-05-18T04:45:37.8775246Z ----------------------------------------------------------------------
2022-05-18T04:45:39.5257871Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:39.5628881Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27259
2022-05-18T04:45:39.5737195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27260
2022-05-18T04:45:40.7508649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:40.7684400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:40.7685914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:40.7710422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:40.7717217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:40.8700876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:44.7872987Z ok (6.910s)
2022-05-18T04:45:44.7873211Z 
2022-05-18T04:45:44.7873615Z ----------------------------------------------------------------------
2022-05-18T04:45:44.7873949Z Ran 1 test in 6.910s
2022-05-18T04:45:44.7874095Z 
2022-05-18T04:45:44.7874188Z OK
2022-05-18T04:45:44.7874320Z 
2022-05-18T04:45:44.7874457Z Generating XML reports...
2022-05-18T04:45:44.7930542Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044537.xml
2022-05-18T04:45:46.2132270Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:45:46.2146842Z 
2022-05-18T04:45:46.2147155Z Running tests...
2022-05-18T04:45:46.2147591Z ----------------------------------------------------------------------
2022-05-18T04:45:46.2166892Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:45:46.2167275Z 
2022-05-18T04:45:46.2167613Z ----------------------------------------------------------------------
2022-05-18T04:45:46.2167964Z Ran 1 test in 0.002s
2022-05-18T04:45:46.2168127Z 
2022-05-18T04:45:46.2168233Z OK (skipped=1)
2022-05-18T04:45:46.2168389Z 
2022-05-18T04:45:46.2168496Z Generating XML reports...
2022-05-18T04:45:46.2209440Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044546.xml
2022-05-18T04:45:47.4850999Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:45:47.4866125Z 
2022-05-18T04:45:47.4866394Z Running tests...
2022-05-18T04:45:47.4866840Z ----------------------------------------------------------------------
2022-05-18T04:45:49.1311994Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:49.1681732Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27422
2022-05-18T04:45:49.1790464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27423
2022-05-18T04:45:50.3118922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:50.3194128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:50.3194925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:50.3220125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:50.3227103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:50.3230297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:45:50.4205617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:50.4209098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:45:50.4210393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:45:50.4243530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:45:54.4909268Z ok (7.004s)
2022-05-18T04:45:54.4909477Z 
2022-05-18T04:45:54.4909871Z ----------------------------------------------------------------------
2022-05-18T04:45:54.4910211Z Ran 1 test in 7.004s
2022-05-18T04:45:54.4910356Z 
2022-05-18T04:45:54.4910450Z OK
2022-05-18T04:45:54.4910586Z 
2022-05-18T04:45:54.4910720Z Generating XML reports...
2022-05-18T04:45:54.4966426Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044547.xml
2022-05-18T04:45:55.9346793Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:45:55.9361743Z 
2022-05-18T04:45:55.9362048Z Running tests...
2022-05-18T04:45:55.9362486Z ----------------------------------------------------------------------
2022-05-18T04:45:55.9382683Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:45:55.9383020Z 
2022-05-18T04:45:55.9383299Z ----------------------------------------------------------------------
2022-05-18T04:45:55.9383629Z Ran 1 test in 0.002s
2022-05-18T04:45:55.9383801Z 
2022-05-18T04:45:55.9383910Z OK (skipped=1)
2022-05-18T04:45:55.9384049Z 
2022-05-18T04:45:55.9384175Z Generating XML reports...
2022-05-18T04:45:55.9426360Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044555.xml
2022-05-18T04:45:57.2128898Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:45:57.2144893Z 
2022-05-18T04:45:57.2145129Z Running tests...
2022-05-18T04:45:57.2145590Z ----------------------------------------------------------------------
2022-05-18T04:45:58.8639729Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:58.9002388Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27589
2022-05-18T04:45:58.9109439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27590
2022-05-18T04:46:00.0541951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:00.1207705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:00.1208755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:00.1251155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:00.1260022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:00.2222036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:00.4158708Z skip: Skipped due to small world size. (3.201s)
2022-05-18T04:46:00.4158974Z 
2022-05-18T04:46:00.4159338Z ----------------------------------------------------------------------
2022-05-18T04:46:00.4159662Z Ran 1 test in 3.201s
2022-05-18T04:46:00.4159829Z 
2022-05-18T04:46:00.4159944Z OK (skipped=1)
2022-05-18T04:46:00.4160100Z 
2022-05-18T04:46:00.4160227Z Generating XML reports...
2022-05-18T04:46:00.4218245Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044557.xml
2022-05-18T04:46:01.8124389Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:01.8140191Z 
2022-05-18T04:46:01.8140672Z Running tests...
2022-05-18T04:46:01.8141105Z ----------------------------------------------------------------------
2022-05-18T04:46:01.8160300Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:46:01.8160626Z 
2022-05-18T04:46:01.8160906Z ----------------------------------------------------------------------
2022-05-18T04:46:01.8161249Z Ran 1 test in 0.002s
2022-05-18T04:46:01.8161414Z 
2022-05-18T04:46:01.8161505Z OK (skipped=1)
2022-05-18T04:46:01.8161660Z 
2022-05-18T04:46:01.8161784Z Generating XML reports...
2022-05-18T04:46:01.8205002Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044601.xml
2022-05-18T04:46:03.0749151Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:03.0769837Z 
2022-05-18T04:46:03.0770338Z Running tests...
2022-05-18T04:46:03.0770850Z ----------------------------------------------------------------------
2022-05-18T04:46:03.0789773Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:46:03.0790126Z 
2022-05-18T04:46:03.0790389Z ----------------------------------------------------------------------
2022-05-18T04:46:03.0790724Z Ran 1 test in 0.002s
2022-05-18T04:46:03.0790888Z 
2022-05-18T04:46:03.0790997Z OK (skipped=1)
2022-05-18T04:46:03.0791154Z 
2022-05-18T04:46:03.0792400Z Generating XML reports...
2022-05-18T04:46:03.0831225Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044603.xml
2022-05-18T04:46:04.3420687Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:04.3435736Z 
2022-05-18T04:46:04.3435961Z Running tests...
2022-05-18T04:46:04.3436402Z ----------------------------------------------------------------------
2022-05-18T04:46:05.9981616Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:06.0348862Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27772
2022-05-18T04:46:06.0456789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27773
2022-05-18T04:46:07.1839582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:07.2037643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:07.2038456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:07.2042006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:07.2048450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:07.3053540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:09.8549769Z ok (5.511s)
2022-05-18T04:46:09.8550202Z 
2022-05-18T04:46:09.8550853Z ----------------------------------------------------------------------
2022-05-18T04:46:09.8551499Z Ran 1 test in 5.511s
2022-05-18T04:46:09.8551797Z 
2022-05-18T04:46:09.8551955Z OK
2022-05-18T04:46:09.8552170Z 
2022-05-18T04:46:09.8552432Z Generating XML reports...
2022-05-18T04:46:09.8609305Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044604.xml
2022-05-18T04:46:11.2730249Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:11.2745858Z 
2022-05-18T04:46:11.2746121Z Running tests...
2022-05-18T04:46:11.2746547Z ----------------------------------------------------------------------
2022-05-18T04:46:12.9227305Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:12.9600007Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27895
2022-05-18T04:46:12.9709217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27896
2022-05-18T04:46:14.1449411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:14.1523273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:14.1524085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:14.1550558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:14.1557718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:14.2539224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:16.8819463Z ok (5.607s)
2022-05-18T04:46:16.8819730Z 
2022-05-18T04:46:16.8820108Z ----------------------------------------------------------------------
2022-05-18T04:46:16.8820449Z Ran 1 test in 5.607s
2022-05-18T04:46:16.8820614Z 
2022-05-18T04:46:16.8820708Z OK
2022-05-18T04:46:16.8820830Z 
2022-05-18T04:46:16.8820965Z Generating XML reports...
2022-05-18T04:46:16.8879620Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044611.xml
2022-05-18T04:46:18.3113180Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:18.3127446Z 
2022-05-18T04:46:18.3127692Z Running tests...
2022-05-18T04:46:18.3128146Z ----------------------------------------------------------------------
2022-05-18T04:46:18.3147776Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:46:18.3148131Z 
2022-05-18T04:46:18.3148413Z ----------------------------------------------------------------------
2022-05-18T04:46:18.3148721Z Ran 1 test in 0.002s
2022-05-18T04:46:18.3148884Z 
2022-05-18T04:46:18.3148993Z OK (skipped=1)
2022-05-18T04:46:18.3149146Z 
2022-05-18T04:46:18.3149270Z Generating XML reports...
2022-05-18T04:46:18.3190533Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044618.xml
2022-05-18T04:46:19.5574263Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:19.5589449Z 
2022-05-18T04:46:19.5589700Z Running tests...
2022-05-18T04:46:19.5590439Z ----------------------------------------------------------------------
2022-05-18T04:46:21.2243180Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:21.2619001Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28053
2022-05-18T04:46:21.2727706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28054
2022-05-18T04:46:22.4545816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:22.4546655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:22.4547205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:22.4547863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:22.4553278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:22.4554315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:22.4556365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:46:22.4558722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:46:22.4559504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:46:22.4659627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:46:25.1823969Z ok (5.623s)
2022-05-18T04:46:25.1824325Z 
2022-05-18T04:46:25.1824787Z ----------------------------------------------------------------------
2022-05-18T04:46:25.1825132Z Ran 1 test in 5.623s
2022-05-18T04:46:25.1825294Z 
2022-05-18T04:46:25.1825384Z OK
2022-05-18T04:46:25.1825522Z 
2022-05-18T04:46:25.1825655Z Generating XML reports...
2022-05-18T04:46:25.1882580Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044619.xml
2022-05-18T04:46:26.6067526Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:26.6082263Z 
2022-05-18T04:46:26.6082561Z Running tests...
2022-05-18T04:46:26.6083006Z ----------------------------------------------------------------------
2022-05-18T04:46:26.6103088Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:46:26.6103643Z 
2022-05-18T04:46:26.6103939Z ----------------------------------------------------------------------
2022-05-18T04:46:26.6104271Z Ran 1 test in 0.002s
2022-05-18T04:46:26.6104433Z 
2022-05-18T04:46:26.6104543Z OK (skipped=1)
2022-05-18T04:46:26.6104702Z 
2022-05-18T04:46:26.6104825Z Generating XML reports...
2022-05-18T04:46:26.6146371Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044626.xml
2022-05-18T04:46:27.8898101Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:27.8913166Z 
2022-05-18T04:46:27.8913570Z Running tests...
2022-05-18T04:46:27.8914045Z ----------------------------------------------------------------------
2022-05-18T04:46:29.5779963Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:29.6149583Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28215
2022-05-18T04:46:29.6258992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28216
2022-05-18T04:46:30.7563536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:30.7801758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:30.7802573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:30.7867458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:30.7874253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:30.8816856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:31.0308603Z skip: Skipped due to small world size. (3.139s)
2022-05-18T04:46:31.0308957Z 
2022-05-18T04:46:31.0309519Z ----------------------------------------------------------------------
2022-05-18T04:46:31.0309862Z Ran 1 test in 3.140s
2022-05-18T04:46:31.0310029Z 
2022-05-18T04:46:31.0310138Z OK (skipped=1)
2022-05-18T04:46:31.0310272Z 
2022-05-18T04:46:31.0310418Z Generating XML reports...
2022-05-18T04:46:31.0366315Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044627.xml
2022-05-18T04:46:32.4561099Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:32.4577534Z 
2022-05-18T04:46:32.4577776Z Running tests...
2022-05-18T04:46:32.4578216Z ----------------------------------------------------------------------
2022-05-18T04:46:34.1144795Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:34.1514431Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28328
2022-05-18T04:46:34.1622748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28329
2022-05-18T04:46:35.3487879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:35.3731339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:35.3732343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:35.3791405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:35.3797785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:35.4746748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:37.8202096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:46:37.8202972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:46:37.8203522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:46:37.8204195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:46:38.1717674Z ok (5.714s)
2022-05-18T04:46:38.1717956Z 
2022-05-18T04:46:38.1718347Z ----------------------------------------------------------------------
2022-05-18T04:46:38.1718688Z Ran 1 test in 5.714s
2022-05-18T04:46:38.1718859Z 
2022-05-18T04:46:38.1718953Z OK
2022-05-18T04:46:38.1719073Z 
2022-05-18T04:46:38.1719207Z Generating XML reports...
2022-05-18T04:46:38.1776510Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044632.xml
2022-05-18T04:46:39.6227234Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:39.6243186Z 
2022-05-18T04:46:39.6243444Z Running tests...
2022-05-18T04:46:39.6243883Z ----------------------------------------------------------------------
2022-05-18T04:46:41.2680971Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:41.3054576Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28462
2022-05-18T04:46:41.3162566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28463
2022-05-18T04:46:42.4678566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:42.5008287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:42.5009069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:42.5085022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:42.5092643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:42.6023300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:42.8213162Z skip: Need at least 3 CUDA devices (3.197s)
2022-05-18T04:46:42.8213963Z 
2022-05-18T04:46:42.8214542Z ----------------------------------------------------------------------
2022-05-18T04:46:42.8214891Z Ran 1 test in 3.197s
2022-05-18T04:46:42.8215058Z 
2022-05-18T04:46:42.8215148Z OK (skipped=1)
2022-05-18T04:46:42.8215300Z 
2022-05-18T04:46:42.8215426Z Generating XML reports...
2022-05-18T04:46:42.8273398Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044639.xml
2022-05-18T04:46:44.2426683Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:44.2442979Z 
2022-05-18T04:46:44.2443540Z Running tests...
2022-05-18T04:46:44.2444051Z ----------------------------------------------------------------------
2022-05-18T04:46:44.2464659Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-05-18T04:46:44.2465358Z 
2022-05-18T04:46:44.2465656Z ----------------------------------------------------------------------
2022-05-18T04:46:44.2465998Z Ran 1 test in 0.002s
2022-05-18T04:46:44.2466162Z 
2022-05-18T04:46:44.2466272Z OK (skipped=1)
2022-05-18T04:46:44.2466429Z 
2022-05-18T04:46:44.2466559Z Generating XML reports...
2022-05-18T04:46:44.2510383Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044644.xml
2022-05-18T04:46:45.5227811Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:45.5243710Z 
2022-05-18T04:46:45.5243959Z Running tests...
2022-05-18T04:46:45.5244405Z ----------------------------------------------------------------------
2022-05-18T04:46:45.5264943Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-05-18T04:46:45.5266993Z 
2022-05-18T04:46:45.5267607Z ----------------------------------------------------------------------
2022-05-18T04:46:45.5267985Z Ran 1 test in 0.002s
2022-05-18T04:46:45.5268160Z 
2022-05-18T04:46:45.5268272Z OK (skipped=1)
2022-05-18T04:46:45.5268440Z 
2022-05-18T04:46:45.5268547Z Generating XML reports...
2022-05-18T04:46:45.5309885Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044645.xml
2022-05-18T04:46:46.7790822Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:46.7806368Z 
2022-05-18T04:46:46.7806692Z Running tests...
2022-05-18T04:46:46.7807147Z ----------------------------------------------------------------------
2022-05-18T04:46:48.4274560Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:48.4647594Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28645
2022-05-18T04:46:48.4756653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28646
2022-05-18T04:46:49.6472463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:49.6837561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:49.6838370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:49.6878483Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:49.6885282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:49.7854226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:53.2866628Z ok (6.506s)
2022-05-18T04:46:53.2866834Z 
2022-05-18T04:46:53.2867235Z ----------------------------------------------------------------------
2022-05-18T04:46:53.2867575Z Ran 1 test in 6.506s
2022-05-18T04:46:53.2867745Z 
2022-05-18T04:46:53.2867839Z OK
2022-05-18T04:46:53.2867976Z 
2022-05-18T04:46:53.2868091Z Generating XML reports...
2022-05-18T04:46:53.2925314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044646.xml
2022-05-18T04:46:54.7278398Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:54.7294228Z 
2022-05-18T04:46:54.7294369Z Running tests...
2022-05-18T04:46:54.7295138Z ----------------------------------------------------------------------
2022-05-18T04:46:54.7315352Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-05-18T04:46:54.7315837Z 
2022-05-18T04:46:54.7316120Z ----------------------------------------------------------------------
2022-05-18T04:46:54.7316462Z Ran 1 test in 0.002s
2022-05-18T04:46:54.7316625Z 
2022-05-18T04:46:54.7316752Z OK (skipped=1)
2022-05-18T04:46:54.7316890Z 
2022-05-18T04:46:54.7317015Z Generating XML reports...
2022-05-18T04:46:54.7360460Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044654.xml
2022-05-18T04:46:56.0082780Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:46:56.0099070Z 
2022-05-18T04:46:56.0099474Z Running tests...
2022-05-18T04:46:56.0099999Z ----------------------------------------------------------------------
2022-05-18T04:46:57.6451094Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:57.6822615Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28803
2022-05-18T04:46:57.6930535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28804
2022-05-18T04:46:58.7838419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:58.8343535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:58.8344335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:58.8345035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:58.8353536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:58.8354313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:58.9979055Z skip: Skipped due to small world size. (2.988s)
2022-05-18T04:46:58.9979499Z 
2022-05-18T04:46:58.9979933Z ----------------------------------------------------------------------
2022-05-18T04:46:58.9980282Z Ran 1 test in 2.988s
2022-05-18T04:46:58.9980445Z 
2022-05-18T04:46:58.9980823Z OK (skipped=1)
2022-05-18T04:46:58.9981005Z 
2022-05-18T04:46:58.9981132Z Generating XML reports...
2022-05-18T04:46:59.0038147Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044656.xml
2022-05-18T04:47:00.4303837Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:00.4319501Z 
2022-05-18T04:47:00.4319735Z Running tests...
2022-05-18T04:47:00.4320173Z ----------------------------------------------------------------------
2022-05-18T04:47:00.4341327Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-05-18T04:47:00.4341776Z 
2022-05-18T04:47:00.4342057Z ----------------------------------------------------------------------
2022-05-18T04:47:00.4342401Z Ran 1 test in 0.002s
2022-05-18T04:47:00.4342564Z 
2022-05-18T04:47:00.4342673Z OK (skipped=1)
2022-05-18T04:47:00.4342831Z 
2022-05-18T04:47:00.4342955Z Generating XML reports...
2022-05-18T04:47:00.4386260Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044700.xml
2022-05-18T04:47:01.7159551Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:01.7176026Z 
2022-05-18T04:47:01.7176635Z Running tests...
2022-05-18T04:47:01.7177291Z ----------------------------------------------------------------------
2022-05-18T04:47:03.3724818Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:03.4095255Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28951
2022-05-18T04:47:03.4205004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28952
2022-05-18T04:47:04.5066780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:04.5694146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:04.5695052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:04.5775280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:04.5782251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:04.6709083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:04.8256885Z skip: Skipped due to small world size. (3.108s)
2022-05-18T04:47:04.8257337Z 
2022-05-18T04:47:04.8257722Z ----------------------------------------------------------------------
2022-05-18T04:47:04.8258045Z Ran 1 test in 3.108s
2022-05-18T04:47:04.8258298Z 
2022-05-18T04:47:04.8258496Z OK (skipped=1)
2022-05-18T04:47:04.8258736Z 
2022-05-18T04:47:04.8258866Z Generating XML reports...
2022-05-18T04:47:04.8314775Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044701.xml
2022-05-18T04:47:06.2331765Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:06.2346566Z 
2022-05-18T04:47:06.2346878Z Running tests...
2022-05-18T04:47:06.2347317Z ----------------------------------------------------------------------
2022-05-18T04:47:06.2367158Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-05-18T04:47:06.2367485Z 
2022-05-18T04:47:06.2367768Z ----------------------------------------------------------------------
2022-05-18T04:47:06.2368094Z Ran 1 test in 0.002s
2022-05-18T04:47:06.2368253Z 
2022-05-18T04:47:06.2368362Z OK (skipped=1)
2022-05-18T04:47:06.2368499Z 
2022-05-18T04:47:06.2368623Z Generating XML reports...
2022-05-18T04:47:06.2410618Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044706.xml
2022-05-18T04:47:07.4676717Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:07.4694003Z 
2022-05-18T04:47:07.4694544Z Running tests...
2022-05-18T04:47:07.4695199Z ----------------------------------------------------------------------
2022-05-18T04:47:07.4719672Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.003s)
2022-05-18T04:47:07.4720284Z 
2022-05-18T04:47:07.4720850Z ----------------------------------------------------------------------
2022-05-18T04:47:07.4721529Z Ran 1 test in 0.003s
2022-05-18T04:47:07.4721861Z 
2022-05-18T04:47:07.4722043Z OK (skipped=1)
2022-05-18T04:47:07.4722219Z 
2022-05-18T04:47:07.4722348Z Generating XML reports...
2022-05-18T04:47:07.4765923Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044707.xml
2022-05-18T04:47:08.7491625Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:08.7507057Z 
2022-05-18T04:47:08.7507376Z Running tests...
2022-05-18T04:47:08.7507805Z ----------------------------------------------------------------------
2022-05-18T04:47:08.7529204Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-05-18T04:47:08.7529670Z 
2022-05-18T04:47:08.7530359Z ----------------------------------------------------------------------
2022-05-18T04:47:08.7530754Z Ran 1 test in 0.002s
2022-05-18T04:47:08.7530924Z 
2022-05-18T04:47:08.7531049Z OK (skipped=1)
2022-05-18T04:47:08.7531203Z 
2022-05-18T04:47:08.7531328Z Generating XML reports...
2022-05-18T04:47:08.7574092Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044708.xml
2022-05-18T04:47:10.0338675Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:10.0353918Z 
2022-05-18T04:47:10.0354270Z Running tests...
2022-05-18T04:47:10.0354736Z ----------------------------------------------------------------------
2022-05-18T04:47:10.0381502Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.003s)
2022-05-18T04:47:10.0381816Z 
2022-05-18T04:47:10.0382104Z ----------------------------------------------------------------------
2022-05-18T04:47:10.0382434Z Ran 1 test in 0.003s
2022-05-18T04:47:10.0382599Z 
2022-05-18T04:47:10.0382692Z OK (skipped=1)
2022-05-18T04:47:10.0382853Z 
2022-05-18T04:47:10.0382981Z Generating XML reports...
2022-05-18T04:47:10.0425249Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044710.xml
2022-05-18T04:47:11.3119693Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:11.3134880Z 
2022-05-18T04:47:11.3135062Z Running tests...
2022-05-18T04:47:11.3135537Z ----------------------------------------------------------------------
2022-05-18T04:47:11.3161879Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.003s)
2022-05-18T04:47:11.3162338Z 
2022-05-18T04:47:11.3162764Z ----------------------------------------------------------------------
2022-05-18T04:47:11.3163172Z Ran 1 test in 0.003s
2022-05-18T04:47:11.3163491Z 
2022-05-18T04:47:11.3163671Z OK (skipped=1)
2022-05-18T04:47:11.3163834Z 
2022-05-18T04:47:11.3163942Z Generating XML reports...
2022-05-18T04:47:11.3205973Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044711.xml
2022-05-18T04:47:12.5850854Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:12.5866678Z 
2022-05-18T04:47:12.5867084Z Running tests...
2022-05-18T04:47:12.5867607Z ----------------------------------------------------------------------
2022-05-18T04:47:14.2207841Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:14.2574191Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29239
2022-05-18T04:47:14.2676718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29240
2022-05-18T04:47:15.4053127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:15.4134960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:15.4135750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:15.4154809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:15.4161129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:15.5147952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:15.5369742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:47:15.5370805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:47:15.5371492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:47:15.5372163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:47:15.5374617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:47:15.5474355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:47:15.5475616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:47:15.5476894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:47:15.8726732Z ok (3.286s)
2022-05-18T04:47:15.8726954Z 
2022-05-18T04:47:15.8727337Z ----------------------------------------------------------------------
2022-05-18T04:47:15.8727676Z Ran 1 test in 3.286s
2022-05-18T04:47:15.8727822Z 
2022-05-18T04:47:15.8727922Z OK
2022-05-18T04:47:15.8728058Z 
2022-05-18T04:47:15.8728192Z Generating XML reports...
2022-05-18T04:47:15.8784861Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044712.xml
2022-05-18T04:47:17.3069673Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:17.3085152Z 
2022-05-18T04:47:17.3085403Z Running tests...
2022-05-18T04:47:17.3085850Z ----------------------------------------------------------------------
2022-05-18T04:47:18.9476501Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:18.9841016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29362
2022-05-18T04:47:18.9947148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29363
2022-05-18T04:47:20.1316199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:20.1469703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:20.1470517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:20.1519257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:20.1527280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:20.2480841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:21.9023796Z ok (4.594s)
2022-05-18T04:47:21.9023998Z 
2022-05-18T04:47:21.9024705Z ----------------------------------------------------------------------
2022-05-18T04:47:21.9025074Z Ran 1 test in 4.594s
2022-05-18T04:47:21.9025247Z 
2022-05-18T04:47:21.9025355Z OK
2022-05-18T04:47:21.9025494Z 
2022-05-18T04:47:21.9025636Z Generating XML reports...
2022-05-18T04:47:21.9081580Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044717.xml
2022-05-18T04:47:23.3628731Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:23.3643541Z 
2022-05-18T04:47:23.3643792Z Running tests...
2022-05-18T04:47:23.3644230Z ----------------------------------------------------------------------
2022-05-18T04:47:25.0142923Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:25.0504385Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29484
2022-05-18T04:47:25.0611278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29485
2022-05-18T04:47:26.2617539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:26.2765702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:26.2766492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:26.2820404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:26.2828488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:26.3780871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:26.5660741Z skip: Skipped due to small world size. (3.201s)
2022-05-18T04:47:26.5661010Z 
2022-05-18T04:47:26.5661643Z ----------------------------------------------------------------------
2022-05-18T04:47:26.5661975Z Ran 1 test in 3.202s
2022-05-18T04:47:26.5662144Z 
2022-05-18T04:47:26.5662253Z OK (skipped=1)
2022-05-18T04:47:26.5662406Z 
2022-05-18T04:47:26.5662531Z Generating XML reports...
2022-05-18T04:47:26.5718695Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044723.xml
2022-05-18T04:47:28.0006711Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:28.0022403Z 
2022-05-18T04:47:28.0022671Z Running tests...
2022-05-18T04:47:28.0023165Z ----------------------------------------------------------------------
2022-05-18T04:47:29.6750812Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:29.7122686Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29597
2022-05-18T04:47:29.7232408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29598
2022-05-18T04:47:30.8848736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:30.9115874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:30.9116685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:30.9152309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:30.9159196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:31.0126879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:32.5306213Z ok (4.528s)
2022-05-18T04:47:32.5306436Z 
2022-05-18T04:47:32.5306802Z ----------------------------------------------------------------------
2022-05-18T04:47:32.5307159Z Ran 1 test in 4.528s
2022-05-18T04:47:32.5307326Z 
2022-05-18T04:47:32.5307429Z OK
2022-05-18T04:47:32.5307566Z 
2022-05-18T04:47:32.5307704Z Generating XML reports...
2022-05-18T04:47:32.5364218Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044727.xml
2022-05-18T04:47:33.9514578Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:33.9529119Z 
2022-05-18T04:47:33.9529379Z Running tests...
2022-05-18T04:47:33.9530105Z ----------------------------------------------------------------------
2022-05-18T04:47:35.5771281Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:35.6136447Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29711
2022-05-18T04:47:35.6242920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29712
2022-05-18T04:47:36.7779636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:36.7994903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:36.7995695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:36.8084097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:36.8090793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:36.9006966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:37.1291807Z ok (3.176s)
2022-05-18T04:47:37.1292013Z 
2022-05-18T04:47:37.1292374Z ----------------------------------------------------------------------
2022-05-18T04:47:37.1292736Z Ran 1 test in 3.176s
2022-05-18T04:47:37.1292902Z 
2022-05-18T04:47:37.1292995Z OK
2022-05-18T04:47:37.1293129Z 
2022-05-18T04:47:37.1293246Z Generating XML reports...
2022-05-18T04:47:37.1350026Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044733.xml
2022-05-18T04:47:38.5745251Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:38.5760260Z 
2022-05-18T04:47:38.5760750Z Running tests...
2022-05-18T04:47:38.5761243Z ----------------------------------------------------------------------
2022-05-18T04:47:40.2189147Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:40.2561392Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29824
2022-05-18T04:47:40.2669470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29825
2022-05-18T04:47:41.3902634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:41.4041713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:41.4042524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:41.4105799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:41.4112308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:41.5055014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:43.1747843Z ok (4.598s)
2022-05-18T04:47:43.1748107Z 
2022-05-18T04:47:43.1748502Z ----------------------------------------------------------------------
2022-05-18T04:47:43.1748827Z Ran 1 test in 4.599s
2022-05-18T04:47:43.1749002Z 
2022-05-18T04:47:43.1749095Z OK
2022-05-18T04:47:43.1749230Z 
2022-05-18T04:47:43.1749364Z Generating XML reports...
2022-05-18T04:47:43.1805653Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044738.xml
2022-05-18T04:47:44.6118517Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:44.6133897Z 
2022-05-18T04:47:44.6134158Z Running tests...
2022-05-18T04:47:44.6134602Z ----------------------------------------------------------------------
2022-05-18T04:47:46.2662663Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:46.3034138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29946
2022-05-18T04:47:46.3141941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29947
2022-05-18T04:47:47.4842570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:47.4928632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:47.4929442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:47.4944029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:47.4950562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:47.5943011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:49.4222759Z ok (4.809s)
2022-05-18T04:47:49.4222983Z 
2022-05-18T04:47:49.4223398Z ----------------------------------------------------------------------
2022-05-18T04:47:49.4223720Z Ran 1 test in 4.809s
2022-05-18T04:47:49.4223887Z 
2022-05-18T04:47:49.4223981Z OK
2022-05-18T04:47:49.4224131Z 
2022-05-18T04:47:49.4224269Z Generating XML reports...
2022-05-18T04:47:49.4281631Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044744.xml
2022-05-18T04:47:50.8357549Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:50.8374072Z 
2022-05-18T04:47:50.8374609Z Running tests...
2022-05-18T04:47:50.8375109Z ----------------------------------------------------------------------
2022-05-18T04:47:52.4890642Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:52.5262433Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30068
2022-05-18T04:47:52.5370746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30069
2022-05-18T04:47:53.6720704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:53.6960653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:53.6961477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:53.7024305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:53.7030958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:53.7972969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:54.0421493Z ok (3.204s)
2022-05-18T04:47:54.0421868Z 
2022-05-18T04:47:54.0422927Z ----------------------------------------------------------------------
2022-05-18T04:47:54.0423544Z Ran 1 test in 3.205s
2022-05-18T04:47:54.0423876Z 
2022-05-18T04:47:54.0424036Z OK
2022-05-18T04:47:54.0424295Z 
2022-05-18T04:47:54.0424533Z Generating XML reports...
2022-05-18T04:47:54.0480592Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044750.xml
2022-05-18T04:47:55.4869897Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:55.4885088Z 
2022-05-18T04:47:55.4885333Z Running tests...
2022-05-18T04:47:55.4885767Z ----------------------------------------------------------------------
2022-05-18T04:47:55.4906057Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:47:55.4906360Z 
2022-05-18T04:47:55.4906638Z ----------------------------------------------------------------------
2022-05-18T04:47:55.4906972Z Ran 1 test in 0.002s
2022-05-18T04:47:55.4907142Z 
2022-05-18T04:47:55.4907272Z OK (skipped=1)
2022-05-18T04:47:55.4907431Z 
2022-05-18T04:47:55.4907537Z Generating XML reports...
2022-05-18T04:47:55.4950185Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044755.xml
2022-05-18T04:47:56.7166180Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:47:56.7181046Z 
2022-05-18T04:47:56.7181482Z Running tests...
2022-05-18T04:47:56.7181967Z ----------------------------------------------------------------------
2022-05-18T04:47:58.3831900Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:58.4202801Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30216
2022-05-18T04:47:58.4310501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30217
2022-05-18T04:47:59.5953770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:59.6299350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:59.6300188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:59.6358802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:59.6365636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:59.7313714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:01.9396123Z ok (5.221s)
2022-05-18T04:48:01.9396349Z 
2022-05-18T04:48:01.9396745Z ----------------------------------------------------------------------
2022-05-18T04:48:01.9397067Z Ran 1 test in 5.221s
2022-05-18T04:48:01.9397236Z 
2022-05-18T04:48:01.9397331Z OK
2022-05-18T04:48:01.9397472Z 
2022-05-18T04:48:01.9397622Z Generating XML reports...
2022-05-18T04:48:01.9454369Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044756.xml
2022-05-18T04:48:03.3956703Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:48:03.3978127Z 
2022-05-18T04:48:03.3978773Z Running tests...
2022-05-18T04:48:03.3979273Z ----------------------------------------------------------------------
2022-05-18T04:48:03.3998357Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:48:03.3998986Z 
2022-05-18T04:48:03.3999380Z ----------------------------------------------------------------------
2022-05-18T04:48:03.3999714Z Ran 1 test in 0.002s
2022-05-18T04:48:03.3999878Z 
2022-05-18T04:48:03.3999970Z OK (skipped=1)
2022-05-18T04:48:03.4000130Z 
2022-05-18T04:48:03.4000262Z Generating XML reports...
2022-05-18T04:48:03.4042934Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044803.xml
2022-05-18T04:48:04.6807337Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:48:04.6822614Z 
2022-05-18T04:48:04.6822846Z Running tests...
2022-05-18T04:48:04.6823282Z ----------------------------------------------------------------------
2022-05-18T04:48:04.6843356Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:48:04.6843660Z 
2022-05-18T04:48:04.6843943Z ----------------------------------------------------------------------
2022-05-18T04:48:04.6844253Z Ran 1 test in 0.002s
2022-05-18T04:48:04.6844418Z 
2022-05-18T04:48:04.6844534Z OK (skipped=1)
2022-05-18T04:48:04.6844691Z 
2022-05-18T04:48:04.6844815Z Generating XML reports...
2022-05-18T04:48:04.6887261Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044804.xml
2022-05-18T04:48:05.9549331Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:48:05.9564107Z 
2022-05-18T04:48:05.9564356Z Running tests...
2022-05-18T04:48:05.9565150Z ----------------------------------------------------------------------
2022-05-18T04:48:05.9586008Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-05-18T04:48:05.9586470Z 
2022-05-18T04:48:05.9586929Z ----------------------------------------------------------------------
2022-05-18T04:48:05.9587267Z Ran 1 test in 0.002s
2022-05-18T04:48:05.9587433Z 
2022-05-18T04:48:05.9587544Z OK (skipped=1)
2022-05-18T04:48:05.9587700Z 
2022-05-18T04:48:05.9587827Z Generating XML reports...
2022-05-18T04:48:05.9630108Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044805.xml
2022-05-18T04:48:07.2156681Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:48:07.2170818Z 
2022-05-18T04:48:07.2171066Z Running tests...
2022-05-18T04:48:07.2171522Z ----------------------------------------------------------------------
2022-05-18T04:48:08.8352537Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:08.8714650Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30447
2022-05-18T04:48:08.8823970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30448
2022-05-18T04:48:10.0561423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:10.0720516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:10.0721311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:10.0764250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:10.0771190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:10.1736037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:12.7915540Z ok (5.574s)
2022-05-18T04:48:12.7915867Z 
2022-05-18T04:48:12.7916251Z ----------------------------------------------------------------------
2022-05-18T04:48:12.7916595Z Ran 1 test in 5.574s
2022-05-18T04:48:12.7916739Z 
2022-05-18T04:48:12.7916833Z OK
2022-05-18T04:48:12.7916967Z 
2022-05-18T04:48:12.7917100Z Generating XML reports...
2022-05-18T04:48:12.7974094Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044807.xml
2022-05-18T04:48:14.2337020Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:48:14.2352393Z 
2022-05-18T04:48:14.2352744Z Running tests...
2022-05-18T04:48:14.2353259Z ----------------------------------------------------------------------
2022-05-18T04:48:15.8912549Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:15.9282744Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30570
2022-05-18T04:48:15.9391230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30571
2022-05-18T04:48:17.0809247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:17.0922833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:17.0923638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:17.1011827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:17.1018517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:17.1935099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:17.2137706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:48:17.2138506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:48:17.2139189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:48:17.2139882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:48:17.2140883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:48:17.2141389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:48:17.2142034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:48:17.2142893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:48:18.5163856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_64qe5x
2022-05-18T04:48:18.5164478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_64qe5x/_remote_module_non_scriptable.py
2022-05-18T04:48:18.5534100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zmgw9vx
2022-05-18T04:48:18.5536595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zmgw9vx/_remote_module_non_scriptable.py
2022-05-18T04:48:18.9470281Z ok (4.711s)
2022-05-18T04:48:18.9472041Z 
2022-05-18T04:48:18.9472588Z ----------------------------------------------------------------------
2022-05-18T04:48:18.9472931Z Ran 1 test in 4.712s
2022-05-18T04:48:18.9473084Z 
2022-05-18T04:48:18.9473184Z OK
2022-05-18T04:48:18.9473331Z 
2022-05-18T04:48:18.9473460Z Generating XML reports...
2022-05-18T04:48:18.9528057Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044814.xml
2022-05-18T04:48:20.3891608Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:48:20.3907235Z 
2022-05-18T04:48:20.3907805Z Running tests...
2022-05-18T04:48:20.3908283Z ----------------------------------------------------------------------
2022-05-18T04:48:22.0402202Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:22.0775897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30700
2022-05-18T04:48:22.0886440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30701
2022-05-18T04:48:23.2605322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:23.2646888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:23.2647683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:23.2706602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:23.2713057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:23.3660269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:23.3871035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:48:23.3871945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:48:23.3872677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:48:23.3874086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:48:23.3875068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:48:23.3875578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:48:23.3876238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:48:23.3876908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:48:24.6981054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4myybhxu
2022-05-18T04:48:24.6981899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4myybhxu/_remote_module_non_scriptable.py
2022-05-18T04:48:24.7044883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf30r_qlk
2022-05-18T04:48:24.7047669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf30r_qlk/_remote_module_non_scriptable.py
2022-05-18T04:48:25.0963326Z ok (4.705s)
2022-05-18T04:48:25.0963606Z 
2022-05-18T04:48:25.0964165Z ----------------------------------------------------------------------
2022-05-18T04:48:25.0964606Z Ran 1 test in 4.706s
2022-05-18T04:48:25.0964886Z 
2022-05-18T04:48:25.0965034Z OK
2022-05-18T04:48:25.0965274Z 
2022-05-18T04:48:25.0965425Z Generating XML reports...
2022-05-18T04:48:25.1023338Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044820.xml
2022-05-18T04:48:26.5301585Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:48:26.5315238Z 
2022-05-18T04:48:26.5315463Z Running tests...
2022-05-18T04:48:26.5316390Z ----------------------------------------------------------------------
2022-05-18T04:48:28.1531262Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:28.1893728Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30830
2022-05-18T04:48:28.2003513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30831
2022-05-18T04:48:29.3687922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:29.3709795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:29.3710833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:29.3789425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:29.3796163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:29.4724761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:30.6528938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzaodrx0g
2022-05-18T04:48:30.6530072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzaodrx0g/_remote_module_non_scriptable.py
2022-05-18T04:48:30.7664258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9vksr2c9
2022-05-18T04:48:30.7665283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9vksr2c9/_remote_module_non_scriptable.py
2022-05-18T04:48:31.1167214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:31.1178998Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:31.4083462Z ok (4.876s)
2022-05-18T04:48:31.4083704Z 
2022-05-18T04:48:31.4084591Z ----------------------------------------------------------------------
2022-05-18T04:48:31.4084940Z Ran 1 test in 4.877s
2022-05-18T04:48:31.4085112Z 
2022-05-18T04:48:31.4085204Z OK
2022-05-18T04:48:31.4085319Z 
2022-05-18T04:48:31.4085528Z Generating XML reports...
2022-05-18T04:48:31.4141748Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044826.xml
2022-05-18T04:48:32.8247538Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:48:32.8261835Z 
2022-05-18T04:48:32.8262071Z Running tests...
2022-05-18T04:48:32.8262493Z ----------------------------------------------------------------------
2022-05-18T04:48:34.4534069Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:34.4899091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30956
2022-05-18T04:48:34.5002894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30957
2022-05-18T04:48:35.6238583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:35.6483431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:35.6484498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:35.6542160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:35.6548578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:35.7498956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:36.9397289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_xnctk3
2022-05-18T04:48:36.9397928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_xnctk3/_remote_module_non_scriptable.py
2022-05-18T04:48:37.0226273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgq5x69qq
2022-05-18T04:48:37.0227155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgq5x69qq/_remote_module_non_scriptable.py
2022-05-18T04:48:37.3807594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:37.3808134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:37.3814226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:37.3817290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:37.7095754Z ok (4.883s)
2022-05-18T04:48:37.7095979Z 
2022-05-18T04:48:37.7096348Z ----------------------------------------------------------------------
2022-05-18T04:48:37.7096713Z Ran 1 test in 4.883s
2022-05-18T04:48:37.7096878Z 
2022-05-18T04:48:37.7096981Z OK
2022-05-18T04:48:37.7097117Z 
2022-05-18T04:48:37.7097253Z Generating XML reports...
2022-05-18T04:48:37.7153814Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044832.xml
2022-05-18T04:48:39.1506669Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:48:39.1521797Z 
2022-05-18T04:48:39.1522092Z Running tests...
2022-05-18T04:48:39.1522534Z ----------------------------------------------------------------------
2022-05-18T04:48:40.8174510Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:40.8540474Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31082
2022-05-18T04:48:40.8648508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31083
2022-05-18T04:48:42.0337726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:42.0461570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:42.0462351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:42.0540996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:42.0547739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:42.1476787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:43.3641866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_imu5rr
2022-05-18T04:48:43.3642487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_imu5rr/_remote_module_non_scriptable.py
2022-05-18T04:48:43.4554494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5z719klp
2022-05-18T04:48:43.4555779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5z719klp/_remote_module_non_scriptable.py
2022-05-18T04:48:43.8196049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:43.8196827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:43.8205753Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:43.8208161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:43.8337812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:43.8339385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:43.8346493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:43.8349227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:44.1731370Z ok (5.021s)
2022-05-18T04:48:44.1731716Z 
2022-05-18T04:48:44.1732505Z ----------------------------------------------------------------------
2022-05-18T04:48:44.1732994Z Ran 1 test in 5.021s
2022-05-18T04:48:44.1733160Z 
2022-05-18T04:48:44.1733234Z OK
2022-05-18T04:48:44.1733371Z 
2022-05-18T04:48:44.1733510Z Generating XML reports...
2022-05-18T04:48:44.1790232Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044839.xml
2022-05-18T04:48:45.5970571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:48:45.5984894Z 
2022-05-18T04:48:45.5985654Z Running tests...
2022-05-18T04:48:45.5986466Z ----------------------------------------------------------------------
2022-05-18T04:48:47.2287746Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:47.2403936Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2022-05-18T04:48:47.2404516Z 
2022-05-18T04:48:47.2404805Z ----------------------------------------------------------------------
2022-05-18T04:48:47.2405322Z Ran 1 test in 1.642s
2022-05-18T04:48:47.2405614Z 
2022-05-18T04:48:47.2405772Z OK (skipped=1)
2022-05-18T04:48:47.2405931Z 
2022-05-18T04:48:47.2406123Z Generating XML reports...
2022-05-18T04:48:47.2442570Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044845.xml
2022-05-18T04:48:48.6392373Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:48:48.6407302Z 
2022-05-18T04:48:48.6407613Z Running tests...
2022-05-18T04:48:48.6408544Z ----------------------------------------------------------------------
2022-05-18T04:48:50.2931597Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:50.3300920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31244
2022-05-18T04:48:50.3408087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31245
2022-05-18T04:48:51.4731821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:51.5031473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:51.5032260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:51.5034895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:51.5042408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:51.6046135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:52.7973555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1j6q0j4e
2022-05-18T04:48:52.7974559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1j6q0j4e/_remote_module_non_scriptable.py
2022-05-18T04:48:52.8904044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7lddbn0
2022-05-18T04:48:52.8904647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7lddbn0/_remote_module_non_scriptable.py
2022-05-18T04:48:53.8366372Z <class 'torch.nn.parameter.Parameter'>
2022-05-18T04:48:54.1499325Z ok (5.509s)
2022-05-18T04:48:54.1500197Z 
2022-05-18T04:48:54.1500759Z ----------------------------------------------------------------------
2022-05-18T04:48:54.1501414Z Ran 1 test in 5.509s
2022-05-18T04:48:54.1501582Z 
2022-05-18T04:48:54.1501656Z OK
2022-05-18T04:48:54.1501793Z 
2022-05-18T04:48:54.1501928Z Generating XML reports...
2022-05-18T04:48:54.1563814Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044848.xml
2022-05-18T04:48:55.5889201Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:48:55.5904792Z 
2022-05-18T04:48:55.5905050Z Running tests...
2022-05-18T04:48:55.5905487Z ----------------------------------------------------------------------
2022-05-18T04:48:57.2549141Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:57.2910647Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31367
2022-05-18T04:48:57.3019139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31368
2022-05-18T04:48:58.4664856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:58.5073783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:58.5074584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:58.5171859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:58.5179253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:58.6088550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:59.8372284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi98h1esb
2022-05-18T04:48:59.8373191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi98h1esb/_remote_module_non_scriptable.py
2022-05-18T04:48:59.9165472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppttyd8pj
2022-05-18T04:48:59.9166258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppttyd8pj/_remote_module_non_scriptable.py
2022-05-18T04:49:01.2115540Z ok (5.621s)
2022-05-18T04:49:01.2115747Z 
2022-05-18T04:49:01.2116135Z ----------------------------------------------------------------------
2022-05-18T04:49:01.2116454Z Ran 1 test in 5.621s
2022-05-18T04:49:01.2116619Z 
2022-05-18T04:49:01.2116716Z OK
2022-05-18T04:49:01.2116854Z 
2022-05-18T04:49:01.2116982Z Generating XML reports...
2022-05-18T04:49:01.2173833Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044855.xml
2022-05-18T04:49:02.6587877Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:49:02.6602749Z 
2022-05-18T04:49:02.6603004Z Running tests...
2022-05-18T04:49:02.6603419Z ----------------------------------------------------------------------
2022-05-18T04:49:04.3244098Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:04.3614958Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31490
2022-05-18T04:49:04.3722526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31491
2022-05-18T04:49:05.5117577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:05.5343390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:05.5344190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:05.5421016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:05.5427746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:05.6358605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:06.8584097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpto5jere5
2022-05-18T04:49:06.8584703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpto5jere5/_remote_module_non_scriptable.py
2022-05-18T04:49:06.9576939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0q_72k6
2022-05-18T04:49:06.9577740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0q_72k6/_remote_module_non_scriptable.py
2022-05-18T04:49:08.1914229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:08.1914827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:08.5822402Z ok (5.922s)
2022-05-18T04:49:08.5822647Z 
2022-05-18T04:49:08.5823032Z ----------------------------------------------------------------------
2022-05-18T04:49:08.5823377Z Ran 1 test in 5.922s
2022-05-18T04:49:08.5823541Z 
2022-05-18T04:49:08.5823633Z OK
2022-05-18T04:49:08.5823749Z 
2022-05-18T04:49:08.5823882Z Generating XML reports...
2022-05-18T04:49:08.5881342Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044902.xml
2022-05-18T04:49:10.0343669Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:49:10.0359316Z 
2022-05-18T04:49:10.0359941Z Running tests...
2022-05-18T04:49:10.0360454Z ----------------------------------------------------------------------
2022-05-18T04:49:11.6986910Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:11.7350796Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31617
2022-05-18T04:49:11.7458233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31618
2022-05-18T04:49:12.8875719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:12.9456610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:12.9457399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:12.9483778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:12.9490148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:13.0472182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:14.2439875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfag6gwmi
2022-05-18T04:49:14.2440496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfag6gwmi/_remote_module_non_scriptable.py
2022-05-18T04:49:14.3486651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxuby2xn
2022-05-18T04:49:14.3487239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxuby2xn/_remote_module_non_scriptable.py
2022-05-18T04:49:14.7138068Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:49:15.0541850Z ok (5.018s)
2022-05-18T04:49:15.0542078Z 
2022-05-18T04:49:15.0542475Z ----------------------------------------------------------------------
2022-05-18T04:49:15.0542816Z Ran 1 test in 5.018s
2022-05-18T04:49:15.0542983Z 
2022-05-18T04:49:15.0543057Z OK
2022-05-18T04:49:15.0545341Z 
2022-05-18T04:49:15.0545616Z Generating XML reports...
2022-05-18T04:49:15.0600016Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044910.xml
2022-05-18T04:49:16.5039876Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:49:16.5055983Z 
2022-05-18T04:49:16.5056241Z Running tests...
2022-05-18T04:49:16.5056893Z ----------------------------------------------------------------------
2022-05-18T04:49:18.1658899Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:18.2032565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31743
2022-05-18T04:49:18.2141197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31744
2022-05-18T04:49:19.3491196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:19.3635806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:19.3636609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:19.3694363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:19.3701756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:19.4651939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:20.6694386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2po1hrul
2022-05-18T04:49:20.6695498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2po1hrul/_remote_module_non_scriptable.py
2022-05-18T04:49:20.7886524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprldje3ns
2022-05-18T04:49:20.7887818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprldje3ns/_remote_module_non_scriptable.py
2022-05-18T04:49:21.1454119Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:49:21.1455738Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:49:21.5224659Z ok (5.016s)
2022-05-18T04:49:21.5224906Z 
2022-05-18T04:49:21.5225303Z ----------------------------------------------------------------------
2022-05-18T04:49:21.5225642Z Ran 1 test in 5.017s
2022-05-18T04:49:21.5225808Z 
2022-05-18T04:49:21.5225909Z OK
2022-05-18T04:49:21.5226058Z 
2022-05-18T04:49:21.5226177Z Generating XML reports...
2022-05-18T04:49:21.5283130Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044916.xml
2022-05-18T04:49:22.9332051Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:49:22.9348818Z 
2022-05-18T04:49:22.9349292Z Running tests...
2022-05-18T04:49:22.9349797Z ----------------------------------------------------------------------
2022-05-18T04:49:22.9376017Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.003s)
2022-05-18T04:49:22.9376514Z 
2022-05-18T04:49:22.9376809Z ----------------------------------------------------------------------
2022-05-18T04:49:22.9377149Z Ran 1 test in 0.003s
2022-05-18T04:49:22.9377319Z 
2022-05-18T04:49:22.9377434Z OK (skipped=1)
2022-05-18T04:49:22.9377595Z 
2022-05-18T04:49:22.9377704Z Generating XML reports...
2022-05-18T04:49:22.9421949Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044922.xml
2022-05-18T04:49:24.1967386Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:49:24.1982368Z 
2022-05-18T04:49:24.1982619Z Running tests...
2022-05-18T04:49:24.1983058Z ----------------------------------------------------------------------
2022-05-18T04:49:25.8241979Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:25.8604628Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31904
2022-05-18T04:49:25.8714596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31905
2022-05-18T04:49:27.0067647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:27.0409043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:27.0410123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:27.0472391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:27.0478763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:27.1424916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:28.3308893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_krvibz
2022-05-18T04:49:28.3309490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_krvibz/_remote_module_non_scriptable.py
2022-05-18T04:49:28.4490709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0nat56w
2022-05-18T04:49:28.4491637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0nat56w/_remote_module_non_scriptable.py
2022-05-18T04:49:29.7922160Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:29.7922718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:30.1814268Z ok (5.983s)
2022-05-18T04:49:30.1814503Z 
2022-05-18T04:49:30.1814898Z ----------------------------------------------------------------------
2022-05-18T04:49:30.1815243Z Ran 1 test in 5.983s
2022-05-18T04:49:30.1815393Z 
2022-05-18T04:49:30.1815488Z OK
2022-05-18T04:49:30.1815630Z 
2022-05-18T04:49:30.1815766Z Generating XML reports...
2022-05-18T04:49:30.1872746Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044924.xml
2022-05-18T04:49:31.6057951Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:49:31.6072585Z 
2022-05-18T04:49:31.6073025Z Running tests...
2022-05-18T04:49:31.6073563Z ----------------------------------------------------------------------
2022-05-18T04:49:33.2114040Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:33.2477386Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32031
2022-05-18T04:49:33.2583776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32032
2022-05-18T04:49:34.4270426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:34.4481379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:34.4482407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:34.4574712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:34.4581537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:34.5495783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:35.7758402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1868e1_
2022-05-18T04:49:35.7759013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1868e1_/_remote_module_non_scriptable.py
2022-05-18T04:49:35.8409092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg811v31c
2022-05-18T04:49:35.8410367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg811v31c/_remote_module_non_scriptable.py
2022-05-18T04:49:36.8764493Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T04:49:36.8767199Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T04:49:36.8776045Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T04:49:36.8777258Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T04:49:37.5683602Z ok (5.961s)
2022-05-18T04:49:37.5683832Z 
2022-05-18T04:49:37.5684198Z ----------------------------------------------------------------------
2022-05-18T04:49:37.5684533Z Ran 1 test in 5.961s
2022-05-18T04:49:37.5684700Z 
2022-05-18T04:49:37.5684795Z OK
2022-05-18T04:49:37.5684933Z 
2022-05-18T04:49:37.5685047Z Generating XML reports...
2022-05-18T04:49:37.5748719Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044931.xml
2022-05-18T04:49:38.9910263Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:49:38.9926087Z 
2022-05-18T04:49:38.9926407Z Running tests...
2022-05-18T04:49:38.9926838Z ----------------------------------------------------------------------
2022-05-18T04:49:40.6140156Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:40.6502118Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32188
2022-05-18T04:49:40.6613307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32189
2022-05-18T04:49:41.8138194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:41.8226633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:41.8227418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:41.8240077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:41.8246869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:41.9242079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:43.1485390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2u6m5w0a
2022-05-18T04:49:43.1486058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2u6m5w0a/_remote_module_non_scriptable.py
2022-05-18T04:49:43.2019294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplob6dmm7
2022-05-18T04:49:43.2021588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplob6dmm7/_remote_module_non_scriptable.py
2022-05-18T04:49:44.5176477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:44.5177085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:44.5360823Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmplob6dmm7'>
2022-05-18T04:49:44.5361309Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:49:44.5362168Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp2u6m5w0a'>
2022-05-18T04:49:44.5362626Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:49:44.8712143Z ok (5.878s)
2022-05-18T04:49:44.8712394Z 
2022-05-18T04:49:44.8712953Z ----------------------------------------------------------------------
2022-05-18T04:49:44.8713278Z Ran 1 test in 5.879s
2022-05-18T04:49:44.8713447Z 
2022-05-18T04:49:44.8713558Z OK
2022-05-18T04:49:44.8713693Z 
2022-05-18T04:49:44.8713823Z Generating XML reports...
2022-05-18T04:49:44.8770795Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044938.xml
2022-05-18T04:49:46.3135652Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:49:46.3150959Z 
2022-05-18T04:49:46.3151413Z Running tests...
2022-05-18T04:49:46.3151905Z ----------------------------------------------------------------------
2022-05-18T04:49:47.9793167Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:47.9914108Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.676s)
2022-05-18T04:49:47.9914782Z 
2022-05-18T04:49:47.9915059Z ----------------------------------------------------------------------
2022-05-18T04:49:47.9915372Z Ran 1 test in 1.676s
2022-05-18T04:49:47.9915546Z 
2022-05-18T04:49:47.9915655Z OK (skipped=1)
2022-05-18T04:49:47.9915808Z 
2022-05-18T04:49:47.9915934Z Generating XML reports...
2022-05-18T04:49:47.9954398Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044946.xml
2022-05-18T04:49:49.3832126Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:49:49.3847354Z 
2022-05-18T04:49:49.3847513Z Running tests...
2022-05-18T04:49:49.3847961Z ----------------------------------------------------------------------
2022-05-18T04:49:51.0331463Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:51.0704811Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32351
2022-05-18T04:49:51.0813685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32352
2022-05-18T04:49:52.2349168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:52.2712256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:52.2713054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:52.2754517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:52.2761654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:52.2765004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:49:52.3726158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:52.3729866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:49:52.3730992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:49:52.3783924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:49:53.6929831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpil9518z8
2022-05-18T04:49:53.6930468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpil9518z8/_remote_module_non_scriptable.py
2022-05-18T04:49:53.7135281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv167lzyv
2022-05-18T04:49:53.7138062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv167lzyv/_remote_module_non_scriptable.py
2022-05-18T04:49:55.0119732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:55.0120298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:55.0129488Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:55.0130608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:55.5918703Z ok (6.207s)
2022-05-18T04:49:55.5919073Z 
2022-05-18T04:49:55.5919488Z ----------------------------------------------------------------------
2022-05-18T04:49:55.5919825Z Ran 1 test in 6.207s
2022-05-18T04:49:55.5919986Z 
2022-05-18T04:49:55.5920078Z OK
2022-05-18T04:49:55.5920214Z 
2022-05-18T04:49:55.5920327Z Generating XML reports...
2022-05-18T04:49:55.5977496Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044949.xml
2022-05-18T04:49:57.0291974Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:49:57.0307551Z 
2022-05-18T04:49:57.0307998Z Running tests...
2022-05-18T04:49:57.0308520Z ----------------------------------------------------------------------
2022-05-18T04:49:58.6973188Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:58.7342480Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32482
2022-05-18T04:49:58.7450865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32483
2022-05-18T04:49:59.8975241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:59.9069321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:59.9070115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:59.9076484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:59.9083080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:59.9085365Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:50:00.0084613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:00.0085462Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:50:01.1821180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuf7qvdgd
2022-05-18T04:50:01.1821780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuf7qvdgd/_remote_module_non_scriptable.py
2022-05-18T04:50:01.3216392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgchpqmn6
2022-05-18T04:50:01.3217624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgchpqmn6/_remote_module_non_scriptable.py
2022-05-18T04:50:02.5156675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:02.5157225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:02.5166868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:02.5167361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:02.5377400Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:50:02.5380614Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:50:02.7550638Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:50:02.7551279Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:50:02.7614250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:02.7615201Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:02.7626184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:02.7626675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:02.7836005Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:50:02.7839507Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:50:02.9220478Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T04:50:02.9228971Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T04:50:02.9290440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:02.9290926Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:02.9302353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:02.9302836Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:03.5560629Z ok (6.525s)
2022-05-18T04:50:03.5564601Z 
2022-05-18T04:50:03.5565289Z ----------------------------------------------------------------------
2022-05-18T04:50:03.5565828Z Ran 1 test in 6.525s
2022-05-18T04:50:03.5565994Z 
2022-05-18T04:50:03.5566089Z OK
2022-05-18T04:50:03.5566229Z 
2022-05-18T04:50:03.5567862Z Generating XML reports...
2022-05-18T04:50:03.5626921Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044957.xml
2022-05-18T04:50:04.9754576Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:04.9768470Z 
2022-05-18T04:50:04.9768772Z Running tests...
2022-05-18T04:50:04.9769489Z ----------------------------------------------------------------------
2022-05-18T04:50:06.5727236Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:06.6089354Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32609
2022-05-18T04:50:06.6197015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32610
2022-05-18T04:50:07.7465460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:50:07.7685249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:50:07.7772066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:07.7772780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:07.7778921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:07.7781750Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:50:07.8700620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:07.8702408Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:50:09.0765243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcurg4u7
2022-05-18T04:50:09.0766423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcurg4u7/_remote_module_non_scriptable.py
2022-05-18T04:50:09.1659030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe57lq5_5
2022-05-18T04:50:09.1660431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe57lq5_5/_remote_module_non_scriptable.py
2022-05-18T04:50:10.5067668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:10.5068490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:10.5077840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:10.5080641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:10.5084534Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:50:10.5087239Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:50:10.5111382Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:50:10.5112906Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:50:10.5114454Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T04:50:10.5115101Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:50:10.5117869Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:50:10.5119046Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T04:50:10.8076980Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:50:10.8079338Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:50:10.8142566Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:10.8145112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:10.8155197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:10.8156592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:10.8162187Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:50:10.8162785Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:50:10.8188193Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:50:10.8189864Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:50:10.8190678Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:50:10.8191924Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:50:11.5305179Z ok (6.553s)
2022-05-18T04:50:11.5305503Z 
2022-05-18T04:50:11.5306275Z ----------------------------------------------------------------------
2022-05-18T04:50:11.5306908Z Ran 1 test in 6.554s
2022-05-18T04:50:11.5307078Z 
2022-05-18T04:50:11.5307151Z OK
2022-05-18T04:50:11.5307290Z 
2022-05-18T04:50:11.5307424Z Generating XML reports...
2022-05-18T04:50:11.5363111Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045004.xml
2022-05-18T04:50:12.9667282Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:12.9681509Z 
2022-05-18T04:50:12.9681835Z Running tests...
2022-05-18T04:50:12.9682767Z ----------------------------------------------------------------------
2022-05-18T04:50:12.9705312Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:50:12.9706281Z 
2022-05-18T04:50:12.9706673Z ----------------------------------------------------------------------
2022-05-18T04:50:12.9707001Z Ran 1 test in 0.002s
2022-05-18T04:50:12.9707143Z 
2022-05-18T04:50:12.9707257Z OK (skipped=1)
2022-05-18T04:50:12.9707412Z 
2022-05-18T04:50:12.9707535Z Generating XML reports...
2022-05-18T04:50:12.9749536Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045012.xml
2022-05-18T04:50:14.2463948Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:14.2479360Z 
2022-05-18T04:50:14.2479667Z Running tests...
2022-05-18T04:50:14.2480100Z ----------------------------------------------------------------------
2022-05-18T04:50:14.2502544Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:50:14.2502987Z 
2022-05-18T04:50:14.2503284Z ----------------------------------------------------------------------
2022-05-18T04:50:14.2503617Z Ran 1 test in 0.002s
2022-05-18T04:50:14.2503762Z 
2022-05-18T04:50:14.2503874Z OK (skipped=1)
2022-05-18T04:50:14.2504028Z 
2022-05-18T04:50:14.2504152Z Generating XML reports...
2022-05-18T04:50:14.2546574Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045014.xml
2022-05-18T04:50:15.5203194Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:15.5218315Z 
2022-05-18T04:50:15.5218564Z Running tests...
2022-05-18T04:50:15.5219002Z ----------------------------------------------------------------------
2022-05-18T04:50:15.5243611Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:50:15.5244111Z 
2022-05-18T04:50:15.5244395Z ----------------------------------------------------------------------
2022-05-18T04:50:15.5244704Z Ran 1 test in 0.003s
2022-05-18T04:50:15.5244868Z 
2022-05-18T04:50:15.5244983Z OK (skipped=1)
2022-05-18T04:50:15.5245140Z 
2022-05-18T04:50:15.5245278Z Generating XML reports...
2022-05-18T04:50:15.5286981Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045015.xml
2022-05-18T04:50:16.8044477Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:16.8060337Z 
2022-05-18T04:50:16.8060596Z Running tests...
2022-05-18T04:50:16.8061045Z ----------------------------------------------------------------------
2022-05-18T04:50:16.8085759Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:50:16.8086866Z 
2022-05-18T04:50:16.8087248Z ----------------------------------------------------------------------
2022-05-18T04:50:16.8087564Z Ran 1 test in 0.003s
2022-05-18T04:50:16.8087726Z 
2022-05-18T04:50:16.8087835Z OK (skipped=1)
2022-05-18T04:50:16.8088011Z 
2022-05-18T04:50:16.8088137Z Generating XML reports...
2022-05-18T04:50:16.8129876Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045016.xml
2022-05-18T04:50:18.0531257Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:18.0547038Z 
2022-05-18T04:50:18.0547456Z Running tests...
2022-05-18T04:50:18.0547973Z ----------------------------------------------------------------------
2022-05-18T04:50:18.0583769Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:50:18.0584286Z 
2022-05-18T04:50:18.0584577Z ----------------------------------------------------------------------
2022-05-18T04:50:18.0584907Z Ran 1 test in 0.003s
2022-05-18T04:50:18.0585068Z 
2022-05-18T04:50:18.0585193Z OK (skipped=1)
2022-05-18T04:50:18.0585352Z 
2022-05-18T04:50:18.0585457Z Generating XML reports...
2022-05-18T04:50:18.0615716Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045018.xml
2022-05-18T04:50:19.3117235Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:19.3132058Z 
2022-05-18T04:50:19.3132544Z Running tests...
2022-05-18T04:50:19.3133177Z ----------------------------------------------------------------------
2022-05-18T04:50:19.3156423Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:50:19.3156930Z 
2022-05-18T04:50:19.3157201Z ----------------------------------------------------------------------
2022-05-18T04:50:19.3157787Z Ran 1 test in 0.002s
2022-05-18T04:50:19.3157967Z 
2022-05-18T04:50:19.3158079Z OK (skipped=1)
2022-05-18T04:50:19.3158236Z 
2022-05-18T04:50:19.3158362Z Generating XML reports...
2022-05-18T04:50:19.3198748Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045019.xml
2022-05-18T04:50:20.5893593Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:20.5908410Z 
2022-05-18T04:50:20.5908664Z Running tests...
2022-05-18T04:50:20.5909367Z ----------------------------------------------------------------------
2022-05-18T04:50:20.5933786Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:50:20.5934431Z 
2022-05-18T04:50:20.5934857Z ----------------------------------------------------------------------
2022-05-18T04:50:20.5935190Z Ran 1 test in 0.003s
2022-05-18T04:50:20.5935351Z 
2022-05-18T04:50:20.5935459Z OK (skipped=1)
2022-05-18T04:50:20.5935611Z 
2022-05-18T04:50:20.5935716Z Generating XML reports...
2022-05-18T04:50:20.5977014Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045020.xml
2022-05-18T04:50:21.8698119Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:21.8712865Z 
2022-05-18T04:50:21.8713370Z Running tests...
2022-05-18T04:50:21.8713991Z ----------------------------------------------------------------------
2022-05-18T04:50:21.8738106Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:50:21.8738827Z 
2022-05-18T04:50:21.8739195Z ----------------------------------------------------------------------
2022-05-18T04:50:21.8739532Z Ran 1 test in 0.003s
2022-05-18T04:50:21.8739692Z 
2022-05-18T04:50:21.8739800Z OK (skipped=1)
2022-05-18T04:50:21.8739951Z 
2022-05-18T04:50:21.8740075Z Generating XML reports...
2022-05-18T04:50:21.8781675Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045021.xml
2022-05-18T04:50:23.1316111Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:23.1331137Z 
2022-05-18T04:50:23.1331671Z Running tests...
2022-05-18T04:50:23.1332238Z ----------------------------------------------------------------------
2022-05-18T04:50:23.1356234Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:50:23.1357012Z 
2022-05-18T04:50:23.1357305Z ----------------------------------------------------------------------
2022-05-18T04:50:23.1357637Z Ran 1 test in 0.003s
2022-05-18T04:50:23.1357798Z 
2022-05-18T04:50:23.1357919Z OK (skipped=1)
2022-05-18T04:50:23.1358072Z 
2022-05-18T04:50:23.1358183Z Generating XML reports...
2022-05-18T04:50:23.1399240Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045023.xml
2022-05-18T04:50:24.4170963Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:24.4186962Z 
2022-05-18T04:50:24.4187228Z Running tests...
2022-05-18T04:50:24.4187651Z ----------------------------------------------------------------------
2022-05-18T04:50:24.4213400Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:50:24.4213919Z 
2022-05-18T04:50:24.4214208Z ----------------------------------------------------------------------
2022-05-18T04:50:24.4214532Z Ran 1 test in 0.003s
2022-05-18T04:50:24.4214687Z 
2022-05-18T04:50:24.4214797Z OK (skipped=1)
2022-05-18T04:50:24.4214951Z 
2022-05-18T04:50:24.4215077Z Generating XML reports...
2022-05-18T04:50:24.4258060Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045024.xml
2022-05-18T04:50:25.6984552Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:25.7000392Z 
2022-05-18T04:50:25.7000734Z Running tests...
2022-05-18T04:50:25.7001173Z ----------------------------------------------------------------------
2022-05-18T04:50:25.7024759Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:50:25.7025182Z 
2022-05-18T04:50:25.7025465Z ----------------------------------------------------------------------
2022-05-18T04:50:25.7025777Z Ran 1 test in 0.002s
2022-05-18T04:50:25.7026224Z 
2022-05-18T04:50:25.7026342Z OK (skipped=1)
2022-05-18T04:50:25.7026498Z 
2022-05-18T04:50:25.7026621Z Generating XML reports...
2022-05-18T04:50:25.7069502Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045025.xml
2022-05-18T04:50:26.9745639Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:26.9761281Z 
2022-05-18T04:50:26.9761693Z Running tests...
2022-05-18T04:50:26.9762187Z ----------------------------------------------------------------------
2022-05-18T04:50:26.9785121Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:50:26.9785561Z 
2022-05-18T04:50:26.9785852Z ----------------------------------------------------------------------
2022-05-18T04:50:26.9786161Z Ran 1 test in 0.002s
2022-05-18T04:50:26.9786334Z 
2022-05-18T04:50:26.9786441Z OK (skipped=1)
2022-05-18T04:50:26.9786596Z 
2022-05-18T04:50:26.9786718Z Generating XML reports...
2022-05-18T04:50:26.9830043Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045026.xml
2022-05-18T04:50:28.2346059Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:28.2360402Z 
2022-05-18T04:50:28.2360835Z Running tests...
2022-05-18T04:50:28.2361355Z ----------------------------------------------------------------------
2022-05-18T04:50:29.8377192Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:29.8493301Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.613s)
2022-05-18T04:50:29.8493994Z 
2022-05-18T04:50:29.8494253Z ----------------------------------------------------------------------
2022-05-18T04:50:29.8494584Z Ran 1 test in 1.613s
2022-05-18T04:50:29.8494746Z 
2022-05-18T04:50:29.8494855Z OK (skipped=1)
2022-05-18T04:50:29.8495009Z 
2022-05-18T04:50:29.8495133Z Generating XML reports...
2022-05-18T04:50:29.8531801Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045028.xml
2022-05-18T04:50:31.2435822Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:31.2451904Z 
2022-05-18T04:50:31.2452441Z Running tests...
2022-05-18T04:50:31.2453255Z ----------------------------------------------------------------------
2022-05-18T04:50:32.9046622Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:32.9417770Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33192
2022-05-18T04:50:32.9524474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33193
2022-05-18T04:50:34.0983856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:50:34.1065008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:50:34.1065805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:34.1085087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:34.1091803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:34.2080666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:35.3870494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwkxusihc
2022-05-18T04:50:35.3871424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwkxusihc/_remote_module_non_scriptable.py
2022-05-18T04:50:35.5203638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01xf64fn
2022-05-18T04:50:35.5204475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01xf64fn/_remote_module_non_scriptable.py
2022-05-18T04:50:36.5613087Z ok (5.316s)
2022-05-18T04:50:36.5613327Z 
2022-05-18T04:50:36.5613732Z ----------------------------------------------------------------------
2022-05-18T04:50:36.5614094Z Ran 1 test in 5.316s
2022-05-18T04:50:36.5614259Z 
2022-05-18T04:50:36.5614354Z OK
2022-05-18T04:50:36.5614471Z 
2022-05-18T04:50:36.5614604Z Generating XML reports...
2022-05-18T04:50:36.5681174Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045031.xml
2022-05-18T04:50:37.9990263Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:38.0005948Z 
2022-05-18T04:50:38.0006218Z Running tests...
2022-05-18T04:50:38.0006883Z ----------------------------------------------------------------------
2022-05-18T04:50:39.6583627Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:39.6955330Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33314
2022-05-18T04:50:39.7065947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33315
2022-05-18T04:50:40.8578567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:50:40.8714292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:50:40.8715109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:40.8781295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:40.8788763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:40.9729344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:42.4617648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpar3mg9vw
2022-05-18T04:50:42.4618260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpar3mg9vw/_remote_module_non_scriptable.py
2022-05-18T04:50:42.5214253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_autkrc
2022-05-18T04:50:42.5215253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_autkrc/_remote_module_non_scriptable.py
2022-05-18T04:50:43.5632738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:43.5633294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:43.5723179Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpv_autkrc'>
2022-05-18T04:50:43.5723666Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:50:43.5724275Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpar3mg9vw'>
2022-05-18T04:50:43.5724710Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:50:43.9166303Z ok (5.916s)
2022-05-18T04:50:43.9166639Z 
2022-05-18T04:50:43.9167039Z ----------------------------------------------------------------------
2022-05-18T04:50:43.9167384Z Ran 1 test in 5.916s
2022-05-18T04:50:43.9167552Z 
2022-05-18T04:50:43.9167663Z OK
2022-05-18T04:50:43.9167800Z 
2022-05-18T04:50:43.9167929Z Generating XML reports...
2022-05-18T04:50:43.9224508Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045037.xml
2022-05-18T04:50:45.3528107Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:45.3544080Z 
2022-05-18T04:50:45.3544720Z Running tests...
2022-05-18T04:50:45.3545218Z ----------------------------------------------------------------------
2022-05-18T04:50:45.3618147Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.007s)
2022-05-18T04:50:45.3618731Z 
2022-05-18T04:50:45.3619030Z ----------------------------------------------------------------------
2022-05-18T04:50:45.3619379Z Ran 1 test in 0.007s
2022-05-18T04:50:45.3619525Z 
2022-05-18T04:50:45.3619641Z OK (skipped=1)
2022-05-18T04:50:45.3619797Z 
2022-05-18T04:50:45.3619945Z Generating XML reports...
2022-05-18T04:50:45.3662935Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045045.xml
2022-05-18T04:50:46.6177168Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:46.6192559Z 
2022-05-18T04:50:46.6193030Z Running tests...
2022-05-18T04:50:46.6193526Z ----------------------------------------------------------------------
2022-05-18T04:50:48.2298933Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:48.2661899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33476
2022-05-18T04:50:48.2768359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33477
2022-05-18T04:50:49.4079823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:50:49.4291839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:50:49.4292868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:49.4385266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:49.4392358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:49.5307678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:50.7223008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiv74w8af
2022-05-18T04:50:50.7223618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiv74w8af/_remote_module_non_scriptable.py
2022-05-18T04:50:50.8485512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtlaiabc
2022-05-18T04:50:50.8486586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtlaiabc/_remote_module_non_scriptable.py
2022-05-18T04:50:52.0467568Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:52.0468140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:52.4871747Z ok (5.868s)
2022-05-18T04:50:52.4871955Z 
2022-05-18T04:50:52.4872324Z ----------------------------------------------------------------------
2022-05-18T04:50:52.4872661Z Ran 1 test in 5.868s
2022-05-18T04:50:52.4872824Z 
2022-05-18T04:50:52.4872924Z OK
2022-05-18T04:50:52.4873058Z 
2022-05-18T04:50:52.4873198Z Generating XML reports...
2022-05-18T04:50:52.4930775Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045046.xml
2022-05-18T04:50:53.9358422Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:50:53.9373835Z 
2022-05-18T04:50:53.9374401Z Running tests...
2022-05-18T04:50:53.9374911Z ----------------------------------------------------------------------
2022-05-18T04:50:55.5791675Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:55.6155684Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33603
2022-05-18T04:50:55.6263547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33604
2022-05-18T04:50:56.7672454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:50:56.7706546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:50:56.7707331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:56.7773926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:56.7780361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:56.8720166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:56.8931823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:50:56.8932558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:50:56.8933465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:50:56.8934520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:50:56.8936132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:50:56.8936705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:50:56.8937381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:50:56.8938071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:50:58.2358194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp453e0ulg
2022-05-18T04:50:58.2358801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp453e0ulg/_remote_module_non_scriptable.py
2022-05-18T04:50:58.2585440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__mzjq3w
2022-05-18T04:50:58.2588106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__mzjq3w/_remote_module_non_scriptable.py
2022-05-18T04:50:58.6353758Z ok (4.698s)
2022-05-18T04:50:58.6353987Z 
2022-05-18T04:50:58.6354656Z ----------------------------------------------------------------------
2022-05-18T04:50:58.6354999Z Ran 1 test in 4.698s
2022-05-18T04:50:58.6355163Z 
2022-05-18T04:50:58.6355256Z OK
2022-05-18T04:50:58.6355394Z 
2022-05-18T04:50:58.6355530Z Generating XML reports...
2022-05-18T04:50:58.6412028Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045053.xml
2022-05-18T04:51:00.0795795Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:51:00.0810585Z 
2022-05-18T04:51:00.0810830Z Running tests...
2022-05-18T04:51:00.0811560Z ----------------------------------------------------------------------
2022-05-18T04:51:01.7327260Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:01.7695726Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33733
2022-05-18T04:51:01.7804755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33734
2022-05-18T04:51:02.9504875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:02.9692298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:02.9693845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:02.9707722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:02.9714273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:03.0705237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:03.0833436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:51:03.0834089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:51:03.0834789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:51:03.0835481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:51:03.0837344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:51:03.0838108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:51:03.0838783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:51:03.0839467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:51:04.4165689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprema59h7
2022-05-18T04:51:04.4166554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprema59h7/_remote_module_non_scriptable.py
2022-05-18T04:51:04.4312186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdolacnnr
2022-05-18T04:51:04.4315357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdolacnnr/_remote_module_non_scriptable.py
2022-05-18T04:51:15.4070368Z ok (15.326s)
2022-05-18T04:51:15.4070715Z 
2022-05-18T04:51:15.4071119Z ----------------------------------------------------------------------
2022-05-18T04:51:15.4071459Z Ran 1 test in 15.326s
2022-05-18T04:51:15.4071623Z 
2022-05-18T04:51:15.4071717Z OK
2022-05-18T04:51:15.4071855Z 
2022-05-18T04:51:15.4071971Z Generating XML reports...
2022-05-18T04:51:15.4128313Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045100.xml
2022-05-18T04:51:16.8493668Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:51:16.8509359Z 
2022-05-18T04:51:16.8509801Z Running tests...
2022-05-18T04:51:16.8510278Z ----------------------------------------------------------------------
2022-05-18T04:51:18.4921940Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:18.5288316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33863
2022-05-18T04:51:18.5396884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33864
2022-05-18T04:51:19.7230924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:19.7371414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:19.7372232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:19.7433648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:19.7440486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:19.8386516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:21.0463928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr8fbmu0x
2022-05-18T04:51:21.0464548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr8fbmu0x/_remote_module_non_scriptable.py
2022-05-18T04:51:21.1455143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3wnx6q8l
2022-05-18T04:51:21.1456065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3wnx6q8l/_remote_module_non_scriptable.py
2022-05-18T04:51:23.1504768Z ok (6.299s)
2022-05-18T04:51:23.1505153Z 
2022-05-18T04:51:23.1506184Z ----------------------------------------------------------------------
2022-05-18T04:51:23.1506819Z Ran 1 test in 6.299s
2022-05-18T04:51:23.1507127Z 
2022-05-18T04:51:23.1507298Z OK
2022-05-18T04:51:23.1507540Z 
2022-05-18T04:51:23.1507790Z Generating XML reports...
2022-05-18T04:51:23.1564569Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045116.xml
2022-05-18T04:51:24.5930594Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:51:24.5945955Z 
2022-05-18T04:51:24.5946416Z Running tests...
2022-05-18T04:51:24.5946910Z ----------------------------------------------------------------------
2022-05-18T04:51:26.2657954Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:26.3028810Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33990
2022-05-18T04:51:26.3138330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33991
2022-05-18T04:51:27.4834534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:27.5292270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:27.5293131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:27.5340872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:27.5347872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:27.6308203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:28.8788806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97aoawp_
2022-05-18T04:51:28.8789678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97aoawp_/_remote_module_non_scriptable.py
2022-05-18T04:51:28.9441877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiyhlhe9l
2022-05-18T04:51:28.9442962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiyhlhe9l/_remote_module_non_scriptable.py
2022-05-18T04:51:30.9244267Z ok (6.329s)
2022-05-18T04:51:30.9244493Z 
2022-05-18T04:51:30.9244885Z ----------------------------------------------------------------------
2022-05-18T04:51:30.9245227Z Ran 1 test in 6.330s
2022-05-18T04:51:30.9245399Z 
2022-05-18T04:51:30.9245492Z OK
2022-05-18T04:51:30.9245610Z 
2022-05-18T04:51:30.9245748Z Generating XML reports...
2022-05-18T04:51:30.9302509Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045124.xml
2022-05-18T04:51:32.3498887Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:51:32.3513462Z 
2022-05-18T04:51:32.3513995Z Running tests...
2022-05-18T04:51:32.3514635Z ----------------------------------------------------------------------
2022-05-18T04:51:33.9810151Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:34.0176810Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34117
2022-05-18T04:51:34.0282453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34118
2022-05-18T04:51:35.1751530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:35.2125106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:35.2125936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:35.2157562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:35.2164909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:35.3140217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:36.4967795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzlb6sonj
2022-05-18T04:51:36.4968956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzlb6sonj/_remote_module_non_scriptable.py
2022-05-18T04:51:36.6225571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxopugqt5
2022-05-18T04:51:36.6226681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxopugqt5/_remote_module_non_scriptable.py
2022-05-18T04:51:38.2382233Z ok (5.887s)
2022-05-18T04:51:38.2382467Z 
2022-05-18T04:51:38.2382859Z ----------------------------------------------------------------------
2022-05-18T04:51:38.2383205Z Ran 1 test in 5.887s
2022-05-18T04:51:38.2383370Z 
2022-05-18T04:51:38.2383447Z OK
2022-05-18T04:51:38.2385317Z 
2022-05-18T04:51:38.2385865Z Generating XML reports...
2022-05-18T04:51:38.2440327Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045132.xml
2022-05-18T04:51:39.6525346Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:51:39.6540248Z 
2022-05-18T04:51:39.6540537Z Running tests...
2022-05-18T04:51:39.6540983Z ----------------------------------------------------------------------
2022-05-18T04:51:41.2697571Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:41.3061034Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34240
2022-05-18T04:51:41.3172124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34241
2022-05-18T04:51:42.4791047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:42.4791829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:42.4792631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:42.4793334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:42.4799986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:42.4800966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:43.8214127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwosr9w3o
2022-05-18T04:51:43.8215914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwosr9w3o/_remote_module_non_scriptable.py
2022-05-18T04:51:43.8513859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpce8zovwz
2022-05-18T04:51:43.8515049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpce8zovwz/_remote_module_non_scriptable.py
2022-05-18T04:51:45.1472806Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:51:45.1517513Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:51:45.1592917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:51:45.1593426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:51:45.5272558Z ok (5.873s)
2022-05-18T04:51:45.5272805Z 
2022-05-18T04:51:45.5273189Z ----------------------------------------------------------------------
2022-05-18T04:51:45.5273536Z Ran 1 test in 5.873s
2022-05-18T04:51:45.5273707Z 
2022-05-18T04:51:45.5273803Z OK
2022-05-18T04:51:45.5273940Z 
2022-05-18T04:51:45.5274055Z Generating XML reports...
2022-05-18T04:51:45.5330195Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045139.xml
2022-05-18T04:51:46.9426692Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:51:46.9441372Z 
2022-05-18T04:51:46.9441694Z Running tests...
2022-05-18T04:51:46.9442369Z ----------------------------------------------------------------------
2022-05-18T04:51:48.5625722Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:48.5991463Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34367
2022-05-18T04:51:48.6102160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34368
2022-05-18T04:51:49.7829386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:49.8060298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:49.8061339Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:49.8136190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:49.8142429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:49.9075390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:51.0983195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpol70kgh1
2022-05-18T04:51:51.0983895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpol70kgh1/_remote_module_non_scriptable.py
2022-05-18T04:51:51.2057857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc__wrguf
2022-05-18T04:51:51.2059257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc__wrguf/_remote_module_non_scriptable.py
2022-05-18T04:51:52.2668218Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:51:52.2669434Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T04:51:52.2670603Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:51:52.2671425Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T04:51:52.9203036Z ok (5.976s)
2022-05-18T04:51:52.9203399Z 
2022-05-18T04:51:52.9204044Z ----------------------------------------------------------------------
2022-05-18T04:51:52.9204393Z Ran 1 test in 5.976s
2022-05-18T04:51:52.9204559Z 
2022-05-18T04:51:52.9204653Z OK
2022-05-18T04:51:52.9208088Z 
2022-05-18T04:51:52.9208373Z Generating XML reports...
2022-05-18T04:51:52.9263172Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045146.xml
2022-05-18T04:51:54.3601684Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:51:54.3616419Z 
2022-05-18T04:51:54.3616677Z Running tests...
2022-05-18T04:51:54.3617115Z ----------------------------------------------------------------------
2022-05-18T04:51:56.0135176Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:56.0255592Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.664s)
2022-05-18T04:51:56.0256246Z 
2022-05-18T04:51:56.0256545Z ----------------------------------------------------------------------
2022-05-18T04:51:56.0256876Z Ran 1 test in 1.664s
2022-05-18T04:51:56.0257021Z 
2022-05-18T04:51:56.0257129Z OK (skipped=1)
2022-05-18T04:51:56.0257284Z 
2022-05-18T04:51:56.0257409Z Generating XML reports...
2022-05-18T04:51:56.0295545Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045154.xml
2022-05-18T04:51:57.4063581Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:51:57.4078148Z 
2022-05-18T04:51:57.4078532Z Running tests...
2022-05-18T04:51:57.4079020Z ----------------------------------------------------------------------
2022-05-18T04:51:59.0079064Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:59.0441592Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34530
2022-05-18T04:51:59.0549651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34531
2022-05-18T04:52:00.1452542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:00.1951144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:00.1951966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:00.1959412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:00.1966995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:00.2967615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:01.4866326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7jyx6j6
2022-05-18T04:52:01.4867784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7jyx6j6/_remote_module_non_scriptable.py
2022-05-18T04:52:01.6223407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsid2tgfv
2022-05-18T04:52:01.6224596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsid2tgfv/_remote_module_non_scriptable.py
2022-05-18T04:52:02.4020251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:02.4021079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:02.4360903Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:52:02.4362512Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:52:02.7649668Z ok (5.357s)
2022-05-18T04:52:02.7650410Z 
2022-05-18T04:52:02.7651070Z ----------------------------------------------------------------------
2022-05-18T04:52:02.7651520Z Ran 1 test in 5.357s
2022-05-18T04:52:02.7651685Z 
2022-05-18T04:52:02.7651759Z OK
2022-05-18T04:52:02.7651899Z 
2022-05-18T04:52:02.7652048Z Generating XML reports...
2022-05-18T04:52:02.7707411Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045157.xml
2022-05-18T04:52:04.1983717Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:52:04.2002513Z 
2022-05-18T04:52:04.2002820Z Running tests...
2022-05-18T04:52:04.2003304Z ----------------------------------------------------------------------
2022-05-18T04:52:05.8277246Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:05.8651702Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34660
2022-05-18T04:52:05.8759624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34661
2022-05-18T04:52:07.0602203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:07.0736606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:07.0737695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:07.0805150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:07.0812463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:07.1753135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:08.4000870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpslgdtzo2
2022-05-18T04:52:08.4001499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpslgdtzo2/_remote_module_non_scriptable.py
2022-05-18T04:52:08.4551331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdvsx2_g4
2022-05-18T04:52:08.4552780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdvsx2_g4/_remote_module_non_scriptable.py
2022-05-18T04:52:09.8852059Z ok (5.685s)
2022-05-18T04:52:09.8852268Z 
2022-05-18T04:52:09.8852648Z ----------------------------------------------------------------------
2022-05-18T04:52:09.8852988Z Ran 1 test in 5.685s
2022-05-18T04:52:09.8853153Z 
2022-05-18T04:52:09.8853247Z OK
2022-05-18T04:52:09.8853384Z 
2022-05-18T04:52:09.8853517Z Generating XML reports...
2022-05-18T04:52:09.8910470Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045204.xml
2022-05-18T04:52:11.3243777Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:52:11.3258664Z 
2022-05-18T04:52:11.3259176Z Running tests...
2022-05-18T04:52:11.3259674Z ----------------------------------------------------------------------
2022-05-18T04:52:12.9315144Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:12.9677909Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34783
2022-05-18T04:52:12.9784719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34784
2022-05-18T04:52:14.1209790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:14.1276608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:14.1277426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:14.1310534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:14.1317231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:14.2291571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:15.4110876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7cumaul
2022-05-18T04:52:15.4112242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7cumaul/_remote_module_non_scriptable.py
2022-05-18T04:52:15.5235884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_bt892u
2022-05-18T04:52:15.5237300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_bt892u/_remote_module_non_scriptable.py
2022-05-18T04:52:15.6015996Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:52:15.6016908Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T04:52:15.6018085Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:52:15.6018907Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T04:52:15.8896793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:15.8897310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:15.8954177Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:52:15.8956029Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:52:15.9066595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:15.9067098Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:15.9133606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:15.9134094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:16.1865550Z ok (4.860s)
2022-05-18T04:52:16.1865762Z 
2022-05-18T04:52:16.1866161Z ----------------------------------------------------------------------
2022-05-18T04:52:16.1866501Z Ran 1 test in 4.861s
2022-05-18T04:52:16.1866665Z 
2022-05-18T04:52:16.1866766Z OK
2022-05-18T04:52:16.1866904Z 
2022-05-18T04:52:16.1867021Z Generating XML reports...
2022-05-18T04:52:16.1922573Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045211.xml
2022-05-18T04:52:17.5965878Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:52:17.5980855Z 
2022-05-18T04:52:17.5981188Z Running tests...
2022-05-18T04:52:17.5981632Z ----------------------------------------------------------------------
2022-05-18T04:52:19.2207448Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:19.2578137Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34909
2022-05-18T04:52:19.2687006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34910
2022-05-18T04:52:20.4143730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:20.4482603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:20.4483625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:20.4548939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:20.4555705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:20.5497353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:21.7748904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8g1af08e
2022-05-18T04:52:21.7750035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8g1af08e/_remote_module_non_scriptable.py
2022-05-18T04:52:21.8485594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2a1cnz_r
2022-05-18T04:52:21.8487056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2a1cnz_r/_remote_module_non_scriptable.py
2022-05-18T04:52:21.9186641Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:52:21.9187836Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T04:52:21.9189001Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:52:21.9189829Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T04:52:22.2068348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:22.2068913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:22.5769588Z ok (4.979s)
2022-05-18T04:52:22.5769831Z 
2022-05-18T04:52:22.5770480Z ----------------------------------------------------------------------
2022-05-18T04:52:22.5770821Z Ran 1 test in 4.979s
2022-05-18T04:52:22.5770986Z 
2022-05-18T04:52:22.5771078Z OK
2022-05-18T04:52:22.5771194Z 
2022-05-18T04:52:22.5771326Z Generating XML reports...
2022-05-18T04:52:22.5829987Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045217.xml
2022-05-18T04:52:24.0118798Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:52:24.0133205Z 
2022-05-18T04:52:24.0133367Z Running tests...
2022-05-18T04:52:24.0133818Z ----------------------------------------------------------------------
2022-05-18T04:52:25.6448532Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:25.6564519Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.643s)
2022-05-18T04:52:25.6565188Z 
2022-05-18T04:52:25.6565468Z ----------------------------------------------------------------------
2022-05-18T04:52:25.6565780Z Ran 1 test in 1.643s
2022-05-18T04:52:25.6565944Z 
2022-05-18T04:52:25.6566051Z OK (skipped=1)
2022-05-18T04:52:25.6568026Z 
2022-05-18T04:52:25.6568536Z Generating XML reports...
2022-05-18T04:52:25.6604057Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045224.xml
2022-05-18T04:52:27.0600746Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:52:27.0615689Z 
2022-05-18T04:52:27.0615826Z Running tests...
2022-05-18T04:52:27.0616523Z ----------------------------------------------------------------------
2022-05-18T04:52:28.7202716Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:28.7575162Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35071
2022-05-18T04:52:28.7684254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35072
2022-05-18T04:52:29.8877930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:29.9084187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:29.9084993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:29.9182053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:29.9188410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:30.0100150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:31.1996271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ou_dd0i
2022-05-18T04:52:31.1996858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ou_dd0i/_remote_module_non_scriptable.py
2022-05-18T04:52:31.3185060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvob051bt
2022-05-18T04:52:31.3186064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvob051bt/_remote_module_non_scriptable.py
2022-05-18T04:52:31.7650158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:31.7650735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:32.8781430Z ok (5.816s)
2022-05-18T04:52:32.8781646Z 
2022-05-18T04:52:32.8782012Z ----------------------------------------------------------------------
2022-05-18T04:52:32.8782370Z Ran 1 test in 5.817s
2022-05-18T04:52:32.8782540Z 
2022-05-18T04:52:32.8782634Z OK
2022-05-18T04:52:32.8782767Z 
2022-05-18T04:52:32.8782902Z Generating XML reports...
2022-05-18T04:52:32.8840939Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045227.xml
2022-05-18T04:52:34.3070103Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:52:34.3086451Z 
2022-05-18T04:52:34.3086713Z Running tests...
2022-05-18T04:52:34.3087156Z ----------------------------------------------------------------------
2022-05-18T04:52:35.9723470Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:36.0089902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35201
2022-05-18T04:52:36.0199435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35202
2022-05-18T04:52:37.1565969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:37.1751558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:37.1753069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:37.1767583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:37.1774547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:37.2768585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:38.4579501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1vlyr7r
2022-05-18T04:52:38.4580732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1vlyr7r/_remote_module_non_scriptable.py
2022-05-18T04:52:38.5913578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_exvh0ho
2022-05-18T04:52:38.5914796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_exvh0ho/_remote_module_non_scriptable.py
2022-05-18T04:52:40.0295601Z ok (5.721s)
2022-05-18T04:52:40.0295825Z 
2022-05-18T04:52:40.0296223Z ----------------------------------------------------------------------
2022-05-18T04:52:40.0296579Z Ran 1 test in 5.721s
2022-05-18T04:52:40.0296725Z 
2022-05-18T04:52:40.0296820Z OK
2022-05-18T04:52:40.0296955Z 
2022-05-18T04:52:40.0297088Z Generating XML reports...
2022-05-18T04:52:40.0355232Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045234.xml
2022-05-18T04:52:41.4727167Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:52:41.4742595Z 
2022-05-18T04:52:41.4743069Z Running tests...
2022-05-18T04:52:41.4743513Z ----------------------------------------------------------------------
2022-05-18T04:52:43.1250369Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:43.1623904Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35324
2022-05-18T04:52:43.1732969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35325
2022-05-18T04:52:44.3227998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:44.3581885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:44.3582703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:44.3634797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:44.3641810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:44.4596588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:45.6812960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobwmsj7n
2022-05-18T04:52:45.6813837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobwmsj7n/_remote_module_non_scriptable.py
2022-05-18T04:52:45.7504145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7d4jjer8
2022-05-18T04:52:45.7505460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7d4jjer8/_remote_module_non_scriptable.py
2022-05-18T04:52:47.1829726Z ok (5.708s)
2022-05-18T04:52:47.1830156Z 
2022-05-18T04:52:47.1830815Z ----------------------------------------------------------------------
2022-05-18T04:52:47.1831425Z Ran 1 test in 5.709s
2022-05-18T04:52:47.1831728Z 
2022-05-18T04:52:47.1831908Z OK
2022-05-18T04:52:47.1832156Z 
2022-05-18T04:52:47.1832377Z Generating XML reports...
2022-05-18T04:52:47.1891689Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045241.xml
2022-05-18T04:52:48.6276925Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:52:48.6292049Z 
2022-05-18T04:52:48.6292358Z Running tests...
2022-05-18T04:52:48.6292783Z ----------------------------------------------------------------------
2022-05-18T04:52:50.2890508Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:50.3265346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35447
2022-05-18T04:52:50.3377278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35448
2022-05-18T04:52:51.4846652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:51.5031682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:51.5032462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:51.5051028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:51.5059448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:51.6046570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:52.8073295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsubtlsgw
2022-05-18T04:52:52.8073895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsubtlsgw/_remote_module_non_scriptable.py
2022-05-18T04:52:52.9062893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqv8dfq0b
2022-05-18T04:52:52.9064173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqv8dfq0b/_remote_module_non_scriptable.py
2022-05-18T04:52:54.2472484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:54.2473033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:52:54.6477787Z ok (6.018s)
2022-05-18T04:52:54.6478110Z 
2022-05-18T04:52:54.6478652Z ----------------------------------------------------------------------
2022-05-18T04:52:54.6479117Z Ran 1 test in 6.018s
2022-05-18T04:52:54.6479281Z 
2022-05-18T04:52:54.6479357Z OK
2022-05-18T04:52:54.6479490Z 
2022-05-18T04:52:54.6479627Z Generating XML reports...
2022-05-18T04:52:54.6537504Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045248.xml
2022-05-18T04:52:56.0879645Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:52:56.0895819Z 
2022-05-18T04:52:56.0896265Z Running tests...
2022-05-18T04:52:56.0896767Z ----------------------------------------------------------------------
2022-05-18T04:52:57.7513446Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:57.7645310Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.675s)
2022-05-18T04:52:57.7645882Z 
2022-05-18T04:52:57.7646159Z ----------------------------------------------------------------------
2022-05-18T04:52:57.7646506Z Ran 1 test in 1.675s
2022-05-18T04:52:57.7646668Z 
2022-05-18T04:52:57.7646778Z OK (skipped=1)
2022-05-18T04:52:57.7646916Z 
2022-05-18T04:52:57.7647040Z Generating XML reports...
2022-05-18T04:52:57.7691402Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045256.xml
2022-05-18T04:52:59.1336776Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:52:59.1351492Z 
2022-05-18T04:52:59.1351749Z Running tests...
2022-05-18T04:52:59.1352175Z ----------------------------------------------------------------------
2022-05-18T04:53:00.7465443Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:00.7828248Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35610
2022-05-18T04:53:00.7936001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35611
2022-05-18T04:53:01.9263062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:01.9654261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:01.9655093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:01.9667247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:01.9674706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:02.0669620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:03.2610162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05lum4_a
2022-05-18T04:53:03.2610798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05lum4_a/_remote_module_non_scriptable.py
2022-05-18T04:53:03.3476181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgp2caubd
2022-05-18T04:53:03.3478211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgp2caubd/_remote_module_non_scriptable.py
2022-05-18T04:53:03.7115759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:53:03.7116335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:53:03.7342745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:53:03.7343433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:53:03.7443822Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp05lum4_a'>
2022-05-18T04:53:03.7444285Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:53:04.1019672Z ok (4.966s)
2022-05-18T04:53:04.1020083Z 
2022-05-18T04:53:04.1020759Z ----------------------------------------------------------------------
2022-05-18T04:53:04.1021487Z Ran 1 test in 4.967s
2022-05-18T04:53:04.1021757Z 
2022-05-18T04:53:04.1021893Z OK
2022-05-18T04:53:04.1022032Z 
2022-05-18T04:53:04.1022164Z Generating XML reports...
2022-05-18T04:53:04.1078606Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045259.xml
2022-05-18T04:53:05.5174639Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:53:05.5189693Z 
2022-05-18T04:53:05.5189950Z Running tests...
2022-05-18T04:53:05.5190390Z ----------------------------------------------------------------------
2022-05-18T04:53:07.1255170Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:07.1618719Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35736
2022-05-18T04:53:07.1727533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35737
2022-05-18T04:53:08.3022752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:08.3210900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:08.3211692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:08.3224966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:08.3232194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:08.4226952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:09.6256273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82ldgqtj
2022-05-18T04:53:09.6256903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82ldgqtj/_remote_module_non_scriptable.py
2022-05-18T04:53:09.7313807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0lmqzgh
2022-05-18T04:53:09.7314868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0lmqzgh/_remote_module_non_scriptable.py
2022-05-18T04:53:11.2824659Z ok (5.763s)
2022-05-18T04:53:11.2824885Z 
2022-05-18T04:53:11.2825270Z ----------------------------------------------------------------------
2022-05-18T04:53:11.2825589Z Ran 1 test in 5.763s
2022-05-18T04:53:11.2825758Z 
2022-05-18T04:53:11.2825858Z OK
2022-05-18T04:53:11.2825994Z 
2022-05-18T04:53:11.2826125Z Generating XML reports...
2022-05-18T04:53:11.2882292Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045305.xml
2022-05-18T04:53:12.7257421Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:53:12.7273303Z 
2022-05-18T04:53:12.7273738Z Running tests...
2022-05-18T04:53:12.7274218Z ----------------------------------------------------------------------
2022-05-18T04:53:14.3945865Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:14.4318370Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35863
2022-05-18T04:53:14.4427628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35864
2022-05-18T04:53:15.5684279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:15.5843804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:15.5844587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:15.5886821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:15.5894459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:15.5897863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:53:15.6855678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:15.6859823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:53:15.6860528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:53:15.6916935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:53:15.9478796Z ok (3.220s)
2022-05-18T04:53:15.9479021Z 
2022-05-18T04:53:15.9479427Z ----------------------------------------------------------------------
2022-05-18T04:53:15.9479745Z Ran 1 test in 3.221s
2022-05-18T04:53:15.9479915Z 
2022-05-18T04:53:15.9480014Z OK
2022-05-18T04:53:15.9480149Z 
2022-05-18T04:53:15.9480291Z Generating XML reports...
2022-05-18T04:53:15.9537676Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045312.xml
2022-05-18T04:53:17.3989327Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:53:17.4004927Z 
2022-05-18T04:53:17.4005313Z Running tests...
2022-05-18T04:53:17.4005801Z ----------------------------------------------------------------------
2022-05-18T04:53:19.0646296Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:19.1020409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35980
2022-05-18T04:53:19.1129326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35981
2022-05-18T04:53:20.2631711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:20.2915864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:20.2916689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:20.2935419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:20.2942290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:20.2945593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:53:20.3927146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:20.3930470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:53:20.3931168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:53:20.3963100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:53:20.7187359Z ok (3.318s)
2022-05-18T04:53:20.7187573Z 
2022-05-18T04:53:20.7188197Z ----------------------------------------------------------------------
2022-05-18T04:53:20.7188580Z Ran 1 test in 3.318s
2022-05-18T04:53:20.7188741Z 
2022-05-18T04:53:20.7188834Z OK
2022-05-18T04:53:20.7188970Z 
2022-05-18T04:53:20.7189107Z Generating XML reports...
2022-05-18T04:53:20.7247416Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045317.xml
2022-05-18T04:53:22.1661415Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:53:22.1676382Z 
2022-05-18T04:53:22.1676654Z Running tests...
2022-05-18T04:53:22.1677094Z ----------------------------------------------------------------------
2022-05-18T04:53:23.8243743Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:23.8616139Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36097
2022-05-18T04:53:23.8725098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36098
2022-05-18T04:53:25.0700454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:25.0750420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:25.0751259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:25.0803609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:25.0809871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:25.1766301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:26.3809937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1mm6ksr
2022-05-18T04:53:26.3810545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1mm6ksr/_remote_module_non_scriptable.py
2022-05-18T04:53:26.4815006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaze6mg_0
2022-05-18T04:53:26.4816092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaze6mg_0/_remote_module_non_scriptable.py
2022-05-18T04:53:26.8493068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:53:26.8493583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:53:26.8560512Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:53:26.8562113Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:53:27.1807671Z ok (5.013s)
2022-05-18T04:53:27.1807889Z 
2022-05-18T04:53:27.1808513Z ----------------------------------------------------------------------
2022-05-18T04:53:27.1809132Z Ran 1 test in 5.013s
2022-05-18T04:53:27.1809296Z 
2022-05-18T04:53:27.1809400Z OK
2022-05-18T04:53:27.1809754Z 
2022-05-18T04:53:27.1809896Z Generating XML reports...
2022-05-18T04:53:27.1866404Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045322.xml
2022-05-18T04:53:28.6018621Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:53:28.6032751Z 
2022-05-18T04:53:28.6033154Z Running tests...
2022-05-18T04:53:28.6033643Z ----------------------------------------------------------------------
2022-05-18T04:53:30.2327873Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:30.2694406Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36223
2022-05-18T04:53:30.2801924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36224
2022-05-18T04:53:31.3793321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:31.4295417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:31.4296238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:31.4300133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:31.4307359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:31.5311977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:32.7357630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbq8dwlgt
2022-05-18T04:53:32.7358240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbq8dwlgt/_remote_module_non_scriptable.py
2022-05-18T04:53:32.8423194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpucuamvx6
2022-05-18T04:53:32.8424693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpucuamvx6/_remote_module_non_scriptable.py
2022-05-18T04:53:33.1935463Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:53:33.2126537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:53:33.2127065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:53:33.4894332Z ok (4.886s)
2022-05-18T04:53:33.4894557Z 
2022-05-18T04:53:33.4894956Z ----------------------------------------------------------------------
2022-05-18T04:53:33.4895279Z Ran 1 test in 4.886s
2022-05-18T04:53:33.4895444Z 
2022-05-18T04:53:33.4895541Z OK
2022-05-18T04:53:33.4896754Z 
2022-05-18T04:53:33.4898798Z Generating XML reports...
2022-05-18T04:53:33.4952356Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045328.xml
2022-05-18T04:53:34.9699247Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:53:34.9714764Z 
2022-05-18T04:53:34.9715357Z Running tests...
2022-05-18T04:53:34.9715958Z ----------------------------------------------------------------------
2022-05-18T04:53:36.6230424Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:36.6600580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36349
2022-05-18T04:53:36.6711455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36350
2022-05-18T04:53:37.8278640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:37.8544925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:37.8545938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:37.8582574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:37.8589761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:37.9557606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:38.1761672Z ok (3.204s)
2022-05-18T04:53:38.1761898Z 
2022-05-18T04:53:38.1762285Z ----------------------------------------------------------------------
2022-05-18T04:53:38.1762629Z Ran 1 test in 3.205s
2022-05-18T04:53:38.1762775Z 
2022-05-18T04:53:38.1762873Z OK
2022-05-18T04:53:38.1763008Z 
2022-05-18T04:53:38.1763153Z Generating XML reports...
2022-05-18T04:53:38.1819974Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045334.xml
2022-05-18T04:53:39.5821632Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:53:39.5836079Z 
2022-05-18T04:53:39.5836536Z Running tests...
2022-05-18T04:53:39.5837071Z ----------------------------------------------------------------------
2022-05-18T04:53:39.5855942Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:53:39.5856822Z 
2022-05-18T04:53:39.5857330Z ----------------------------------------------------------------------
2022-05-18T04:53:39.5857696Z Ran 1 test in 0.002s
2022-05-18T04:53:39.5857861Z 
2022-05-18T04:53:39.5857971Z OK (skipped=1)
2022-05-18T04:53:39.5858130Z 
2022-05-18T04:53:39.5858261Z Generating XML reports...
2022-05-18T04:53:39.5899312Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045339.xml
2022-05-18T04:53:40.8629115Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:53:40.8644408Z 
2022-05-18T04:53:40.8644732Z Running tests...
2022-05-18T04:53:40.8645185Z ----------------------------------------------------------------------
2022-05-18T04:53:40.8673626Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.003s)
2022-05-18T04:53:40.8674224Z 
2022-05-18T04:53:40.8674530Z ----------------------------------------------------------------------
2022-05-18T04:53:40.8674881Z Ran 1 test in 0.003s
2022-05-18T04:53:40.8675046Z 
2022-05-18T04:53:40.8675156Z OK (skipped=1)
2022-05-18T04:53:40.8675307Z 
2022-05-18T04:53:40.8675431Z Generating XML reports...
2022-05-18T04:53:40.8717495Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045340.xml
2022-05-18T04:53:42.1388352Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:53:42.1403473Z 
2022-05-18T04:53:42.1403884Z Running tests...
2022-05-18T04:53:42.1404395Z ----------------------------------------------------------------------
2022-05-18T04:53:43.7892821Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:43.8259557Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36532
2022-05-18T04:53:43.8367189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36533
2022-05-18T04:53:45.0017398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:45.0059910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:45.0060727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:45.0118873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:45.0125464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:45.1075375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:49.0481917Z ok (6.908s)
2022-05-18T04:53:49.0482143Z 
2022-05-18T04:53:49.0483033Z ----------------------------------------------------------------------
2022-05-18T04:53:49.0483390Z Ran 1 test in 6.908s
2022-05-18T04:53:49.0483571Z 
2022-05-18T04:53:49.0483667Z OK
2022-05-18T04:53:49.0483813Z 
2022-05-18T04:53:49.0483944Z Generating XML reports...
2022-05-18T04:53:49.0540327Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045342.xml
2022-05-18T04:53:50.4903379Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:53:50.4918610Z 
2022-05-18T04:53:50.4919066Z Running tests...
2022-05-18T04:53:50.4919553Z ----------------------------------------------------------------------
2022-05-18T04:53:50.4939228Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:53:50.4939555Z 
2022-05-18T04:53:50.4939856Z ----------------------------------------------------------------------
2022-05-18T04:53:50.4940189Z Ran 1 test in 0.002s
2022-05-18T04:53:50.4940355Z 
2022-05-18T04:53:50.4940464Z OK (skipped=1)
2022-05-18T04:53:50.4940617Z 
2022-05-18T04:53:50.4940735Z Generating XML reports...
2022-05-18T04:53:50.4982690Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045350.xml
2022-05-18T04:53:51.7709221Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:53:51.7724263Z 
2022-05-18T04:53:51.7724739Z Running tests...
2022-05-18T04:53:51.7725226Z ----------------------------------------------------------------------
2022-05-18T04:53:51.7744914Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:53:51.7745233Z 
2022-05-18T04:53:51.7745537Z ----------------------------------------------------------------------
2022-05-18T04:53:51.7745850Z Ran 1 test in 0.002s
2022-05-18T04:53:51.7746294Z 
2022-05-18T04:53:51.7746422Z OK (skipped=1)
2022-05-18T04:53:51.7746583Z 
2022-05-18T04:53:51.7746710Z Generating XML reports...
2022-05-18T04:53:51.7788993Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045351.xml
2022-05-18T04:53:53.0499651Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:53:53.0514721Z 
2022-05-18T04:53:53.0515063Z Running tests...
2022-05-18T04:53:53.0515516Z ----------------------------------------------------------------------
2022-05-18T04:53:54.7014667Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:54.7380850Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36730
2022-05-18T04:53:54.7488698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36731
2022-05-18T04:53:55.9219890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:55.9489347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:55.9490832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:55.9523899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:55.9530885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:56.0504617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:58.7585424Z ok (5.707s)
2022-05-18T04:53:58.7585652Z 
2022-05-18T04:53:58.7586293Z ----------------------------------------------------------------------
2022-05-18T04:53:58.7586699Z Ran 1 test in 5.707s
2022-05-18T04:53:58.7586867Z 
2022-05-18T04:53:58.7586971Z OK
2022-05-18T04:53:58.7587125Z 
2022-05-18T04:53:58.7587259Z Generating XML reports...
2022-05-18T04:53:58.7643135Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045353.xml
2022-05-18T04:54:00.1757471Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:00.1772040Z 
2022-05-18T04:54:00.1772185Z Running tests...
2022-05-18T04:54:00.1772788Z ----------------------------------------------------------------------
2022-05-18T04:54:01.8086032Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:01.8454994Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36854
2022-05-18T04:54:01.8562592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36855
2022-05-18T04:54:02.9827768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:03.0089099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:03.0090234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:03.0130548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:03.0137532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:03.1102190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:03.1313778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:54:03.1314298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:54:03.1315209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:03.1315932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:05.6047189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:54:05.6047747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:54:05.6048536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:54:05.6049994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:54:05.6638847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:54:05.6639411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:54:05.6640172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:54:05.6640868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:54:06.0661801Z ok (5.889s)
2022-05-18T04:54:06.0662018Z 
2022-05-18T04:54:06.0662394Z ----------------------------------------------------------------------
2022-05-18T04:54:06.0662748Z Ran 1 test in 5.889s
2022-05-18T04:54:06.0662916Z 
2022-05-18T04:54:06.0663010Z OK
2022-05-18T04:54:06.0663143Z 
2022-05-18T04:54:06.0663276Z Generating XML reports...
2022-05-18T04:54:06.0722079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045400.xml
2022-05-18T04:54:07.4906484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:07.4921147Z 
2022-05-18T04:54:07.4922070Z Running tests...
2022-05-18T04:54:07.4923082Z ----------------------------------------------------------------------
2022-05-18T04:54:09.1125786Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:09.1489231Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37017
2022-05-18T04:54:09.1600194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37018
2022-05-18T04:54:10.2860301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:10.2960796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:10.2962215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:10.2962925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:10.2970446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:10.2971531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:10.2974011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:54:10.2975115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:54:10.2976468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:10.2977177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:10.4644851Z ok (2.972s)
2022-05-18T04:54:10.4645210Z 
2022-05-18T04:54:10.4645834Z ----------------------------------------------------------------------
2022-05-18T04:54:10.4646179Z Ran 1 test in 2.972s
2022-05-18T04:54:10.4646328Z 
2022-05-18T04:54:10.4646741Z OK
2022-05-18T04:54:10.4647024Z 
2022-05-18T04:54:10.4647261Z Generating XML reports...
2022-05-18T04:54:10.4702961Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045407.xml
2022-05-18T04:54:11.8943039Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:11.8957810Z 
2022-05-18T04:54:11.8958144Z Running tests...
2022-05-18T04:54:11.8958585Z ----------------------------------------------------------------------
2022-05-18T04:54:13.5561967Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:13.5936684Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37134
2022-05-18T04:54:13.6046411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37135
2022-05-18T04:54:14.7282949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:14.7721512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:14.7722333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:14.7790085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:14.7796777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:14.8737631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:17.4140010Z ok (5.518s)
2022-05-18T04:54:17.4140204Z 
2022-05-18T04:54:17.4140618Z ----------------------------------------------------------------------
2022-05-18T04:54:17.4140953Z Ran 1 test in 5.518s
2022-05-18T04:54:17.4141117Z 
2022-05-18T04:54:17.4141209Z OK
2022-05-18T04:54:17.4141343Z 
2022-05-18T04:54:17.4141473Z Generating XML reports...
2022-05-18T04:54:17.4198881Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045411.xml
2022-05-18T04:54:18.8563382Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:18.8579517Z 
2022-05-18T04:54:18.8579988Z Running tests...
2022-05-18T04:54:18.8580454Z ----------------------------------------------------------------------
2022-05-18T04:54:20.5070850Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:20.5443672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37257
2022-05-18T04:54:20.5553038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37258
2022-05-18T04:54:21.7059356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:21.7149020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:21.7149842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:21.7160354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:21.7167529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:21.8161101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:22.1604311Z ok (3.302s)
2022-05-18T04:54:22.1604519Z 
2022-05-18T04:54:22.1604903Z ----------------------------------------------------------------------
2022-05-18T04:54:22.1605236Z Ran 1 test in 3.302s
2022-05-18T04:54:22.1605397Z 
2022-05-18T04:54:22.1605479Z OK
2022-05-18T04:54:22.1605612Z 
2022-05-18T04:54:22.1605744Z Generating XML reports...
2022-05-18T04:54:22.1664654Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045418.xml
2022-05-18T04:54:23.5921716Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:23.5936744Z 
2022-05-18T04:54:23.5936892Z Running tests...
2022-05-18T04:54:23.5937638Z ----------------------------------------------------------------------
2022-05-18T04:54:25.2244438Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:25.2611685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37370
2022-05-18T04:54:25.2719222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37371
2022-05-18T04:54:26.3957389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:26.4122788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:26.4123595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:26.4159535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:26.4166059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:26.4168941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:54:26.5134606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:26.5138942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:54:26.5139636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:26.5187313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:26.7769371Z ok (3.183s)
2022-05-18T04:54:26.7769862Z 
2022-05-18T04:54:26.7770255Z ----------------------------------------------------------------------
2022-05-18T04:54:26.7770613Z Ran 1 test in 3.183s
2022-05-18T04:54:26.7770780Z 
2022-05-18T04:54:26.7770870Z OK
2022-05-18T04:54:26.7770986Z 
2022-05-18T04:54:26.7771118Z Generating XML reports...
2022-05-18T04:54:26.7827268Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045423.xml
2022-05-18T04:54:28.1969275Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:28.1983764Z 
2022-05-18T04:54:28.1984120Z Running tests...
2022-05-18T04:54:28.1984858Z ----------------------------------------------------------------------
2022-05-18T04:54:29.7970487Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:29.8333682Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37487
2022-05-18T04:54:29.8443133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37488
2022-05-18T04:54:30.9890524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:31.0038624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:31.0039428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:31.0093173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:31.0099779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:31.0103223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:54:31.1049924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:31.1054411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:54:31.1055129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:31.1121604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:31.3491940Z ok (3.150s)
2022-05-18T04:54:31.3492295Z 
2022-05-18T04:54:31.3492798Z ----------------------------------------------------------------------
2022-05-18T04:54:31.3493157Z Ran 1 test in 3.151s
2022-05-18T04:54:31.3493338Z 
2022-05-18T04:54:31.3493433Z OK
2022-05-18T04:54:31.3493573Z 
2022-05-18T04:54:31.3493706Z Generating XML reports...
2022-05-18T04:54:31.3550913Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045428.xml
2022-05-18T04:54:32.7660370Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:32.7675152Z 
2022-05-18T04:54:32.7675303Z Running tests...
2022-05-18T04:54:32.7676094Z ----------------------------------------------------------------------
2022-05-18T04:54:34.3798650Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:34.4163661Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37604
2022-05-18T04:54:34.4272785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37605
2022-05-18T04:54:35.5520729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:35.5684395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:35.5685227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:35.5723934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:35.5730602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:35.6700558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:36.8465007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81rj2afi
2022-05-18T04:54:36.8465610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81rj2afi/_remote_module_non_scriptable.py
2022-05-18T04:54:36.9772497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhyu_o7h
2022-05-18T04:54:36.9773552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhyu_o7h/_remote_module_non_scriptable.py
2022-05-18T04:54:37.6354024Z ok (4.868s)
2022-05-18T04:54:37.6354488Z 
2022-05-18T04:54:37.6354962Z ----------------------------------------------------------------------
2022-05-18T04:54:37.6355483Z Ran 1 test in 4.868s
2022-05-18T04:54:37.6355765Z 
2022-05-18T04:54:37.6355872Z OK
2022-05-18T04:54:37.6356013Z 
2022-05-18T04:54:37.6356146Z Generating XML reports...
2022-05-18T04:54:37.6413296Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045432.xml
2022-05-18T04:54:39.0792651Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:39.0808726Z 
2022-05-18T04:54:39.0809049Z Running tests...
2022-05-18T04:54:39.0809493Z ----------------------------------------------------------------------
2022-05-18T04:54:39.0837388Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.003s)
2022-05-18T04:54:39.0838090Z 
2022-05-18T04:54:39.0838863Z ----------------------------------------------------------------------
2022-05-18T04:54:39.0839247Z Ran 1 test in 0.003s
2022-05-18T04:54:39.0839411Z 
2022-05-18T04:54:39.0839527Z OK (skipped=1)
2022-05-18T04:54:39.0839684Z 
2022-05-18T04:54:39.0839819Z Generating XML reports...
2022-05-18T04:54:39.0882401Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045439.xml
2022-05-18T04:54:40.3625014Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:40.3640672Z 
2022-05-18T04:54:40.3640979Z Running tests...
2022-05-18T04:54:40.3641411Z ----------------------------------------------------------------------
2022-05-18T04:54:40.3661413Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-05-18T04:54:40.3661959Z 
2022-05-18T04:54:40.3662448Z ----------------------------------------------------------------------
2022-05-18T04:54:40.3662788Z Ran 1 test in 0.002s
2022-05-18T04:54:40.3662954Z 
2022-05-18T04:54:40.3663073Z OK (skipped=1)
2022-05-18T04:54:40.3663232Z 
2022-05-18T04:54:40.3663360Z Generating XML reports...
2022-05-18T04:54:40.3706411Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045440.xml
2022-05-18T04:54:41.6403155Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:41.6419420Z 
2022-05-18T04:54:41.6419681Z Running tests...
2022-05-18T04:54:41.6420339Z ----------------------------------------------------------------------
2022-05-18T04:54:41.6440233Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-05-18T04:54:41.6440563Z 
2022-05-18T04:54:41.6440851Z ----------------------------------------------------------------------
2022-05-18T04:54:41.6441343Z Ran 1 test in 0.002s
2022-05-18T04:54:41.6441582Z 
2022-05-18T04:54:41.6441693Z OK (skipped=1)
2022-05-18T04:54:41.6441849Z 
2022-05-18T04:54:41.6441993Z Generating XML reports...
2022-05-18T04:54:41.6484429Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045441.xml
2022-05-18T04:54:42.9272498Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:42.9288415Z 
2022-05-18T04:54:42.9288861Z Running tests...
2022-05-18T04:54:42.9289330Z ----------------------------------------------------------------------
2022-05-18T04:54:42.9310342Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-05-18T04:54:42.9310949Z 
2022-05-18T04:54:42.9311334Z ----------------------------------------------------------------------
2022-05-18T04:54:42.9311848Z Ran 1 test in 0.002s
2022-05-18T04:54:42.9311996Z 
2022-05-18T04:54:42.9312107Z OK (skipped=1)
2022-05-18T04:54:42.9312263Z 
2022-05-18T04:54:42.9312389Z Generating XML reports...
2022-05-18T04:54:42.9355344Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045442.xml
2022-05-18T04:54:44.1641951Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:44.1657772Z 
2022-05-18T04:54:44.1658071Z Running tests...
2022-05-18T04:54:44.1658509Z ----------------------------------------------------------------------
2022-05-18T04:54:44.1680060Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T04:54:44.1680420Z 
2022-05-18T04:54:44.1680709Z ----------------------------------------------------------------------
2022-05-18T04:54:44.1681024Z Ran 1 test in 0.002s
2022-05-18T04:54:44.1681194Z 
2022-05-18T04:54:44.1681304Z OK (skipped=1)
2022-05-18T04:54:44.1681463Z 
2022-05-18T04:54:44.1681589Z Generating XML reports...
2022-05-18T04:54:44.1724561Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045444.xml
2022-05-18T04:54:45.4403410Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:45.4419707Z 
2022-05-18T04:54:45.4420213Z Running tests...
2022-05-18T04:54:45.4420699Z ----------------------------------------------------------------------
2022-05-18T04:54:45.4441763Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T04:54:45.4442153Z 
2022-05-18T04:54:45.4442417Z ----------------------------------------------------------------------
2022-05-18T04:54:45.4442749Z Ran 1 test in 0.002s
2022-05-18T04:54:45.4442913Z 
2022-05-18T04:54:45.4443024Z OK (skipped=1)
2022-05-18T04:54:45.4443183Z 
2022-05-18T04:54:45.4443315Z Generating XML reports...
2022-05-18T04:54:45.4485583Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045445.xml
2022-05-18T04:54:46.7000268Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:46.7017146Z 
2022-05-18T04:54:46.7017408Z Running tests...
2022-05-18T04:54:46.7018175Z ----------------------------------------------------------------------
2022-05-18T04:54:46.7043946Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:54:46.7044308Z 
2022-05-18T04:54:46.7044569Z ----------------------------------------------------------------------
2022-05-18T04:54:46.7044903Z Ran 1 test in 0.003s
2022-05-18T04:54:46.7045066Z 
2022-05-18T04:54:46.7045175Z OK (skipped=1)
2022-05-18T04:54:46.7045329Z 
2022-05-18T04:54:46.7045459Z Generating XML reports...
2022-05-18T04:54:46.7088513Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045446.xml
2022-05-18T04:54:47.9803503Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:47.9820193Z 
2022-05-18T04:54:47.9820893Z Running tests...
2022-05-18T04:54:47.9821806Z ----------------------------------------------------------------------
2022-05-18T04:54:47.9852299Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.003s)
2022-05-18T04:54:47.9852974Z 
2022-05-18T04:54:47.9853538Z ----------------------------------------------------------------------
2022-05-18T04:54:47.9854190Z Ran 1 test in 0.003s
2022-05-18T04:54:47.9854509Z 
2022-05-18T04:54:47.9854723Z OK (skipped=1)
2022-05-18T04:54:47.9855013Z 
2022-05-18T04:54:47.9855217Z Generating XML reports...
2022-05-18T04:54:47.9898915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045447.xml
2022-05-18T04:54:49.2411195Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:49.2426294Z 
2022-05-18T04:54:49.2426774Z Running tests...
2022-05-18T04:54:49.2427189Z ----------------------------------------------------------------------
2022-05-18T04:54:49.2449356Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:54:49.2450070Z 
2022-05-18T04:54:49.2450369Z ----------------------------------------------------------------------
2022-05-18T04:54:49.2450703Z Ran 1 test in 0.002s
2022-05-18T04:54:49.2450865Z 
2022-05-18T04:54:49.2450956Z OK (skipped=1)
2022-05-18T04:54:49.2451109Z 
2022-05-18T04:54:49.2451232Z Generating XML reports...
2022-05-18T04:54:49.2491016Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045449.xml
2022-05-18T04:54:50.5149157Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:50.5163994Z 
2022-05-18T04:54:50.5164621Z Running tests...
2022-05-18T04:54:50.5165251Z ----------------------------------------------------------------------
2022-05-18T04:54:50.5188177Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:54:50.5188881Z 
2022-05-18T04:54:50.5189438Z ----------------------------------------------------------------------
2022-05-18T04:54:50.5189907Z Ran 1 test in 0.002s
2022-05-18T04:54:50.5190070Z 
2022-05-18T04:54:50.5190179Z OK (skipped=1)
2022-05-18T04:54:50.5190334Z 
2022-05-18T04:54:50.5190441Z Generating XML reports...
2022-05-18T04:54:50.5232628Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045450.xml
2022-05-18T04:54:51.7944340Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:51.7960253Z 
2022-05-18T04:54:51.7960502Z Running tests...
2022-05-18T04:54:51.7960959Z ----------------------------------------------------------------------
2022-05-18T04:54:51.7984449Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:54:51.7985566Z 
2022-05-18T04:54:51.7986096Z ----------------------------------------------------------------------
2022-05-18T04:54:51.7986477Z Ran 1 test in 0.002s
2022-05-18T04:54:51.7986638Z 
2022-05-18T04:54:51.7986746Z OK (skipped=1)
2022-05-18T04:54:51.7986898Z 
2022-05-18T04:54:51.7987021Z Generating XML reports...
2022-05-18T04:54:51.8029358Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045451.xml
2022-05-18T04:54:53.0805524Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:53.0821531Z 
2022-05-18T04:54:53.0821960Z Running tests...
2022-05-18T04:54:53.0822461Z ----------------------------------------------------------------------
2022-05-18T04:54:54.7538898Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:54.7909735Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38115
2022-05-18T04:54:54.8019353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38116
2022-05-18T04:54:55.9323775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:55.9552957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:55.9553763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:55.9627359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:55.9633802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:56.0568276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:57.7098078Z ok (4.627s)
2022-05-18T04:54:57.7098300Z 
2022-05-18T04:54:57.7098703Z ----------------------------------------------------------------------
2022-05-18T04:54:57.7099040Z Ran 1 test in 4.628s
2022-05-18T04:54:57.7099204Z 
2022-05-18T04:54:57.7099303Z OK
2022-05-18T04:54:57.7099437Z 
2022-05-18T04:54:57.7099571Z Generating XML reports...
2022-05-18T04:54:57.7156922Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045453.xml
2022-05-18T04:54:59.1274130Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:54:59.1288648Z 
2022-05-18T04:54:59.1289095Z Running tests...
2022-05-18T04:54:59.1289942Z ----------------------------------------------------------------------
2022-05-18T04:55:00.7299401Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:00.7662801Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38237
2022-05-18T04:55:00.7770982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38238
2022-05-18T04:55:01.8905157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:01.9123901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:01.9124720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:01.9209131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:01.9216680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:02.0139402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:03.6860054Z ok (4.557s)
2022-05-18T04:55:03.6860275Z 
2022-05-18T04:55:03.6860651Z ----------------------------------------------------------------------
2022-05-18T04:55:03.6861281Z Ran 1 test in 4.557s
2022-05-18T04:55:03.6861452Z 
2022-05-18T04:55:03.6861555Z OK
2022-05-18T04:55:03.6861690Z 
2022-05-18T04:55:03.6861823Z Generating XML reports...
2022-05-18T04:55:03.6919774Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045459.xml
2022-05-18T04:55:05.1107325Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:55:05.1122049Z 
2022-05-18T04:55:05.1122300Z Running tests...
2022-05-18T04:55:05.1122725Z ----------------------------------------------------------------------
2022-05-18T04:55:06.7205913Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:06.7571123Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38359
2022-05-18T04:55:06.7679084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38360
2022-05-18T04:55:07.9447680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:07.9708434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:07.9709251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:07.9751450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:07.9758259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:08.0722963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:10.6771183Z ok (5.565s)
2022-05-18T04:55:10.6771405Z 
2022-05-18T04:55:10.6771791Z ----------------------------------------------------------------------
2022-05-18T04:55:10.6772111Z Ran 1 test in 5.565s
2022-05-18T04:55:10.6772304Z 
2022-05-18T04:55:10.6772398Z OK
2022-05-18T04:55:10.6772534Z 
2022-05-18T04:55:10.6772667Z Generating XML reports...
2022-05-18T04:55:10.6829913Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045505.xml
2022-05-18T04:55:12.0978286Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:55:12.0992819Z 
2022-05-18T04:55:12.0993101Z Running tests...
2022-05-18T04:55:12.0993542Z ----------------------------------------------------------------------
2022-05-18T04:55:13.7002766Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:13.7366742Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38482
2022-05-18T04:55:13.7475819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38483
2022-05-18T04:55:14.8764454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:14.9242471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:14.9243281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:14.9270318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:14.9277265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:15.0257789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:16.6551528Z ok (4.555s)
2022-05-18T04:55:16.6551748Z 
2022-05-18T04:55:16.6552154Z ----------------------------------------------------------------------
2022-05-18T04:55:16.6552497Z Ran 1 test in 4.556s
2022-05-18T04:55:16.6552662Z 
2022-05-18T04:55:16.6552767Z OK
2022-05-18T04:55:16.6553183Z 
2022-05-18T04:55:16.6553299Z Generating XML reports...
2022-05-18T04:55:16.6610688Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045512.xml
2022-05-18T04:55:18.1044814Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:55:18.1061117Z 
2022-05-18T04:55:18.1061266Z Running tests...
2022-05-18T04:55:18.1062204Z ----------------------------------------------------------------------
2022-05-18T04:55:19.7588795Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:19.7956057Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38604
2022-05-18T04:55:19.8066121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38605
2022-05-18T04:55:20.9810881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:20.9887160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:20.9887972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:20.9912077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:20.9919425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:21.0898048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:24.3167758Z ok (6.210s)
2022-05-18T04:55:24.3167994Z 
2022-05-18T04:55:24.3168402Z ----------------------------------------------------------------------
2022-05-18T04:55:24.3168732Z Ran 1 test in 6.211s
2022-05-18T04:55:24.3168897Z 
2022-05-18T04:55:24.3168992Z OK
2022-05-18T04:55:24.3169126Z 
2022-05-18T04:55:24.3169261Z Generating XML reports...
2022-05-18T04:55:24.3227116Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045518.xml
2022-05-18T04:55:25.7531775Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:55:25.7546783Z 
2022-05-18T04:55:25.7547168Z Running tests...
2022-05-18T04:55:25.7547617Z ----------------------------------------------------------------------
2022-05-18T04:55:25.7570474Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:55:25.7571008Z 
2022-05-18T04:55:25.7571341Z ----------------------------------------------------------------------
2022-05-18T04:55:25.7571672Z Ran 1 test in 0.002s
2022-05-18T04:55:25.7571837Z 
2022-05-18T04:55:25.7572240Z OK (skipped=1)
2022-05-18T04:55:25.7572400Z 
2022-05-18T04:55:25.7572527Z Generating XML reports...
2022-05-18T04:55:25.7612915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045525.xml
2022-05-18T04:55:27.0430453Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:55:27.0445999Z 
2022-05-18T04:55:27.0446499Z Running tests...
2022-05-18T04:55:27.0447001Z ----------------------------------------------------------------------
2022-05-18T04:55:27.0481449Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-05-18T04:55:27.0481772Z 
2022-05-18T04:55:27.0482064Z ----------------------------------------------------------------------
2022-05-18T04:55:27.0482375Z Ran 1 test in 0.004s
2022-05-18T04:55:27.0482538Z 
2022-05-18T04:55:27.0482928Z OK (skipped=1)
2022-05-18T04:55:27.0483098Z 
2022-05-18T04:55:27.0483246Z Generating XML reports...
2022-05-18T04:55:27.0532927Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045527.xml
2022-05-18T04:55:28.2993026Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:55:28.3013096Z 
2022-05-18T04:55:28.3013637Z Running tests...
2022-05-18T04:55:28.3014127Z ----------------------------------------------------------------------
2022-05-18T04:55:28.3044723Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-05-18T04:55:28.3045091Z 
2022-05-18T04:55:28.3045378Z ----------------------------------------------------------------------
2022-05-18T04:55:28.3045689Z Ran 1 test in 0.003s
2022-05-18T04:55:28.3045855Z 
2022-05-18T04:55:28.3045963Z OK (skipped=1)
2022-05-18T04:55:28.3046118Z 
2022-05-18T04:55:28.3046243Z Generating XML reports...
2022-05-18T04:55:28.3093218Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045528.xml
2022-05-18T04:55:29.5658548Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:55:29.5672891Z 
2022-05-18T04:55:29.5673314Z Running tests...
2022-05-18T04:55:29.5673750Z ----------------------------------------------------------------------
2022-05-18T04:55:31.1930516Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:31.2296744Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38842
2022-05-18T04:55:31.2409319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38843
2022-05-18T04:55:32.4220295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:32.4294091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:32.4294883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:32.4321490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:32.4328163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:32.5305797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:32.7459814Z ok (3.178s)
2022-05-18T04:55:32.7460074Z 
2022-05-18T04:55:32.7460487Z ----------------------------------------------------------------------
2022-05-18T04:55:32.7460825Z Ran 1 test in 3.179s
2022-05-18T04:55:32.7460995Z 
2022-05-18T04:55:32.7461092Z OK
2022-05-18T04:55:32.7461208Z 
2022-05-18T04:55:32.7461351Z Generating XML reports...
2022-05-18T04:55:32.7517719Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045529.xml
2022-05-18T04:55:34.1730172Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:55:34.1747029Z 
2022-05-18T04:55:34.1747497Z Running tests...
2022-05-18T04:55:34.1748000Z ----------------------------------------------------------------------
2022-05-18T04:55:35.8256523Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:35.8630394Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38955
2022-05-18T04:55:35.8740077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38956
2022-05-18T04:55:37.0205555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:37.0461049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:37.0461930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:37.0509696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:37.0516983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:37.1473401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:37.3790636Z ok (3.204s)
2022-05-18T04:55:37.3791058Z 
2022-05-18T04:55:37.3791466Z ----------------------------------------------------------------------
2022-05-18T04:55:37.3791806Z Ran 1 test in 3.204s
2022-05-18T04:55:37.3791953Z 
2022-05-18T04:55:37.3792048Z OK
2022-05-18T04:55:37.3843738Z 
2022-05-18T04:55:37.3843892Z Generating XML reports...
2022-05-18T04:55:37.3850440Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045534.xml
2022-05-18T04:55:38.8370578Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:55:38.8386448Z 
2022-05-18T04:55:38.8386613Z Running tests...
2022-05-18T04:55:38.8387294Z ----------------------------------------------------------------------
2022-05-18T04:55:38.8409657Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:55:38.8410207Z 
2022-05-18T04:55:38.8410732Z ----------------------------------------------------------------------
2022-05-18T04:55:38.8411084Z Ran 1 test in 0.002s
2022-05-18T04:55:38.8411251Z 
2022-05-18T04:55:38.8411343Z OK (skipped=1)
2022-05-18T04:55:38.8411504Z 
2022-05-18T04:55:38.8411633Z Generating XML reports...
2022-05-18T04:55:38.8455155Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045538.xml
2022-05-18T04:55:40.1211677Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:55:40.1228331Z 
2022-05-18T04:55:40.1228615Z Running tests...
2022-05-18T04:55:40.1229038Z ----------------------------------------------------------------------
2022-05-18T04:55:40.1250273Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:55:40.1250998Z 
2022-05-18T04:55:40.1251315Z ----------------------------------------------------------------------
2022-05-18T04:55:40.1251865Z Ran 1 test in 0.002s
2022-05-18T04:55:40.1252069Z 
2022-05-18T04:55:40.1252162Z OK (skipped=1)
2022-05-18T04:55:40.1252318Z 
2022-05-18T04:55:40.1252447Z Generating XML reports...
2022-05-18T04:55:40.1295778Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045540.xml
2022-05-18T04:55:41.4162849Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:55:41.4178783Z 
2022-05-18T04:55:41.4178979Z Running tests...
2022-05-18T04:55:41.4179499Z ----------------------------------------------------------------------
2022-05-18T04:55:43.0712466Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:43.1088163Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39138
2022-05-18T04:55:43.1201929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39139
2022-05-18T04:55:44.2727621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:44.3126432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:44.3127245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:44.3134038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:44.3141201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:44.4142673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:45.6309188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wso_wga
2022-05-18T04:55:45.6309805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wso_wga/_remote_module_non_scriptable.py
2022-05-18T04:55:45.7415743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcurncim
2022-05-18T04:55:45.7416637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcurncim/_remote_module_non_scriptable.py
2022-05-18T04:55:47.5303103Z ok (6.112s)
2022-05-18T04:55:47.5303333Z 
2022-05-18T04:55:47.5303723Z ----------------------------------------------------------------------
2022-05-18T04:55:47.5304074Z Ran 1 test in 6.112s
2022-05-18T04:55:47.5304237Z 
2022-05-18T04:55:47.5304340Z OK
2022-05-18T04:55:47.5304456Z 
2022-05-18T04:55:47.5304588Z Generating XML reports...
2022-05-18T04:55:47.5361131Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045541.xml
2022-05-18T04:55:48.9797342Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:55:48.9813368Z 
2022-05-18T04:55:48.9813653Z Running tests...
2022-05-18T04:55:48.9814090Z ----------------------------------------------------------------------
2022-05-18T04:55:50.6400908Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:50.6780810Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39265
2022-05-18T04:55:50.6895448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39266
2022-05-18T04:55:51.8323233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:51.8384976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:51.8385779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:51.8427344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:51.8434224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:51.9400222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:53.1823939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7u5ae1g
2022-05-18T04:55:53.1824765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7u5ae1g/_remote_module_non_scriptable.py
2022-05-18T04:55:53.2248136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_rqt0vcr
2022-05-18T04:55:53.2250059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_rqt0vcr/_remote_module_non_scriptable.py
2022-05-18T04:55:54.9996908Z ok (6.018s)
2022-05-18T04:55:54.9997153Z 
2022-05-18T04:55:54.9997526Z ----------------------------------------------------------------------
2022-05-18T04:55:54.9997865Z Ran 1 test in 6.018s
2022-05-18T04:55:54.9998035Z 
2022-05-18T04:55:54.9998128Z OK
2022-05-18T04:55:54.9998245Z 
2022-05-18T04:55:54.9998382Z Generating XML reports...
2022-05-18T04:55:55.0057580Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045548.xml
2022-05-18T04:55:56.4358418Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:55:56.4373083Z 
2022-05-18T04:55:56.4373576Z Running tests...
2022-05-18T04:55:56.4374056Z ----------------------------------------------------------------------
2022-05-18T04:55:58.0516497Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:58.0885514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39392
2022-05-18T04:55:58.0997838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39393
2022-05-18T04:55:59.2643813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:59.2908277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:59.2909102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:59.2947556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:59.2954560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:59.3923635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:02.1092453Z ok (5.672s)
2022-05-18T04:56:02.1092716Z 
2022-05-18T04:56:02.1093264Z ----------------------------------------------------------------------
2022-05-18T04:56:02.1093610Z Ran 1 test in 5.672s
2022-05-18T04:56:02.1093758Z 
2022-05-18T04:56:02.1093849Z OK
2022-05-18T04:56:02.1093982Z 
2022-05-18T04:56:02.1094118Z Generating XML reports...
2022-05-18T04:56:02.1150786Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045556.xml
2022-05-18T04:56:03.5770805Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:03.5787342Z 
2022-05-18T04:56:03.5787632Z Running tests...
2022-05-18T04:56:03.5788072Z ----------------------------------------------------------------------
2022-05-18T04:56:05.2285877Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:05.2665382Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39515
2022-05-18T04:56:05.2775550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39516
2022-05-18T04:56:06.4635971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:06.4858419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:06.4859259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:06.4939396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:06.4946438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:06.5873715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:09.2882846Z ok (5.709s)
2022-05-18T04:56:09.2883089Z 
2022-05-18T04:56:09.2883464Z ----------------------------------------------------------------------
2022-05-18T04:56:09.2883802Z Ran 1 test in 5.710s
2022-05-18T04:56:09.2883965Z 
2022-05-18T04:56:09.2884062Z OK
2022-05-18T04:56:09.2884196Z 
2022-05-18T04:56:09.2884329Z Generating XML reports...
2022-05-18T04:56:09.2940375Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045603.xml
2022-05-18T04:56:10.7371068Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:10.7387476Z 
2022-05-18T04:56:10.7387921Z Running tests...
2022-05-18T04:56:10.7388361Z ----------------------------------------------------------------------
2022-05-18T04:56:12.4053291Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:12.4170980Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.678s)
2022-05-18T04:56:12.4171864Z 
2022-05-18T04:56:12.4172156Z ----------------------------------------------------------------------
2022-05-18T04:56:12.4172486Z Ran 1 test in 1.678s
2022-05-18T04:56:12.4172647Z 
2022-05-18T04:56:12.4172738Z OK (skipped=1)
2022-05-18T04:56:12.4172893Z 
2022-05-18T04:56:12.4173018Z Generating XML reports...
2022-05-18T04:56:12.4209404Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045610.xml
2022-05-18T04:56:13.8134906Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:13.8149950Z 
2022-05-18T04:56:13.8150336Z Running tests...
2022-05-18T04:56:13.8150800Z ----------------------------------------------------------------------
2022-05-18T04:56:15.4586678Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:15.4705003Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.655s)
2022-05-18T04:56:15.4705566Z 
2022-05-18T04:56:15.4705846Z ----------------------------------------------------------------------
2022-05-18T04:56:15.4706156Z Ran 1 test in 1.655s
2022-05-18T04:56:15.4706323Z 
2022-05-18T04:56:15.4706428Z OK (skipped=1)
2022-05-18T04:56:15.4706580Z 
2022-05-18T04:56:15.4706704Z Generating XML reports...
2022-05-18T04:56:15.4744058Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045613.xml
2022-05-18T04:56:16.8632828Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:16.8648468Z 
2022-05-18T04:56:16.8648725Z Running tests...
2022-05-18T04:56:16.8649130Z ----------------------------------------------------------------------
2022-05-18T04:56:18.5064437Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:18.5430783Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39710
2022-05-18T04:56:18.5538620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39711
2022-05-18T04:56:19.6788705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:19.7002235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:19.7003032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:19.7093499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:19.7099891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:19.8017716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:19.9587788Z skip: Need at least 4 CUDA devices (3.094s)
2022-05-18T04:56:19.9588020Z 
2022-05-18T04:56:19.9588399Z ----------------------------------------------------------------------
2022-05-18T04:56:19.9588735Z Ran 1 test in 3.094s
2022-05-18T04:56:19.9588896Z 
2022-05-18T04:56:19.9589003Z OK (skipped=1)
2022-05-18T04:56:19.9589162Z 
2022-05-18T04:56:19.9589290Z Generating XML reports...
2022-05-18T04:56:19.9647332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045616.xml
2022-05-18T04:56:21.3864834Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:21.3880165Z 
2022-05-18T04:56:21.3880387Z Running tests...
2022-05-18T04:56:21.3880802Z ----------------------------------------------------------------------
2022-05-18T04:56:23.0324114Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:23.0702802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39823
2022-05-18T04:56:23.0812300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39824
2022-05-18T04:56:24.2246580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:24.2399728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:24.2400750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:24.2450297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:24.2457666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:24.3414697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:24.4860984Z skip: Need at least 4 CUDA devices (3.098s)
2022-05-18T04:56:24.4861214Z 
2022-05-18T04:56:24.4861580Z ----------------------------------------------------------------------
2022-05-18T04:56:24.4861913Z Ran 1 test in 3.098s
2022-05-18T04:56:24.4862080Z 
2022-05-18T04:56:24.4862192Z OK (skipped=1)
2022-05-18T04:56:24.4862348Z 
2022-05-18T04:56:24.4862472Z Generating XML reports...
2022-05-18T04:56:24.4919334Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045621.xml
2022-05-18T04:56:25.9078553Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:25.9094091Z 
2022-05-18T04:56:25.9094241Z Running tests...
2022-05-18T04:56:25.9095178Z ----------------------------------------------------------------------
2022-05-18T04:56:25.9117217Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:56:25.9117535Z 
2022-05-18T04:56:25.9117798Z ----------------------------------------------------------------------
2022-05-18T04:56:25.9118121Z Ran 1 test in 0.002s
2022-05-18T04:56:25.9118283Z 
2022-05-18T04:56:25.9118396Z OK (skipped=1)
2022-05-18T04:56:25.9118550Z 
2022-05-18T04:56:25.9118672Z Generating XML reports...
2022-05-18T04:56:25.9162450Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045625.xml
2022-05-18T04:56:27.1885107Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:27.1900860Z 
2022-05-18T04:56:27.1901276Z Running tests...
2022-05-18T04:56:27.1901758Z ----------------------------------------------------------------------
2022-05-18T04:56:27.1923571Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:56:27.1923899Z 
2022-05-18T04:56:27.1924188Z ----------------------------------------------------------------------
2022-05-18T04:56:27.1924512Z Ran 1 test in 0.002s
2022-05-18T04:56:27.1924672Z 
2022-05-18T04:56:27.1924763Z OK (skipped=1)
2022-05-18T04:56:27.1924921Z 
2022-05-18T04:56:27.1925045Z Generating XML reports...
2022-05-18T04:56:27.1968726Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045627.xml
2022-05-18T04:56:28.4634703Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:28.4650354Z 
2022-05-18T04:56:28.4650650Z Running tests...
2022-05-18T04:56:28.4651434Z ----------------------------------------------------------------------
2022-05-18T04:56:28.4675017Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:56:28.4675341Z 
2022-05-18T04:56:28.4675628Z ----------------------------------------------------------------------
2022-05-18T04:56:28.4675937Z Ran 1 test in 0.003s
2022-05-18T04:56:28.4676098Z 
2022-05-18T04:56:28.4676210Z OK (skipped=1)
2022-05-18T04:56:28.4676363Z 
2022-05-18T04:56:28.4676488Z Generating XML reports...
2022-05-18T04:56:28.4719694Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045628.xml
2022-05-18T04:56:29.7484965Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:29.7500522Z 
2022-05-18T04:56:29.7500944Z Running tests...
2022-05-18T04:56:29.7501425Z ----------------------------------------------------------------------
2022-05-18T04:56:29.7524590Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:56:29.7525218Z 
2022-05-18T04:56:29.7525826Z ----------------------------------------------------------------------
2022-05-18T04:56:29.7526256Z Ran 1 test in 0.002s
2022-05-18T04:56:29.7526421Z 
2022-05-18T04:56:29.7526528Z OK (skipped=1)
2022-05-18T04:56:29.7526680Z 
2022-05-18T04:56:29.7526803Z Generating XML reports...
2022-05-18T04:56:29.7570278Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045629.xml
2022-05-18T04:56:31.0308905Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:31.0324469Z 
2022-05-18T04:56:31.0324915Z Running tests...
2022-05-18T04:56:31.0325462Z ----------------------------------------------------------------------
2022-05-18T04:56:31.0348146Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:56:31.0348723Z 
2022-05-18T04:56:31.0349307Z ----------------------------------------------------------------------
2022-05-18T04:56:31.0349941Z Ran 1 test in 0.002s
2022-05-18T04:56:31.0350103Z 
2022-05-18T04:56:31.0350210Z OK (skipped=1)
2022-05-18T04:56:31.0350364Z 
2022-05-18T04:56:31.0350486Z Generating XML reports...
2022-05-18T04:56:31.0393228Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045631.xml
2022-05-18T04:56:32.2731114Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:32.2746598Z 
2022-05-18T04:56:32.2746931Z Running tests...
2022-05-18T04:56:32.2748199Z ----------------------------------------------------------------------
2022-05-18T04:56:32.2769595Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:56:32.2770522Z 
2022-05-18T04:56:32.2771144Z ----------------------------------------------------------------------
2022-05-18T04:56:32.2771632Z Ran 1 test in 0.002s
2022-05-18T04:56:32.2771794Z 
2022-05-18T04:56:32.2771884Z OK (skipped=1)
2022-05-18T04:56:32.2772039Z 
2022-05-18T04:56:32.2772162Z Generating XML reports...
2022-05-18T04:56:32.2814755Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045632.xml
2022-05-18T04:56:33.5575525Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:33.5590786Z 
2022-05-18T04:56:33.5591178Z Running tests...
2022-05-18T04:56:33.5591598Z ----------------------------------------------------------------------
2022-05-18T04:56:33.5615275Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:56:33.5615590Z 
2022-05-18T04:56:33.5615873Z ----------------------------------------------------------------------
2022-05-18T04:56:33.5616498Z Ran 1 test in 0.003s
2022-05-18T04:56:33.5616642Z 
2022-05-18T04:56:33.5617004Z OK (skipped=1)
2022-05-18T04:56:33.5617164Z 
2022-05-18T04:56:33.5617508Z Generating XML reports...
2022-05-18T04:56:33.5659805Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045633.xml
2022-05-18T04:56:34.8464446Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:34.8479650Z 
2022-05-18T04:56:34.8479980Z Running tests...
2022-05-18T04:56:34.8480415Z ----------------------------------------------------------------------
2022-05-18T04:56:34.8503695Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:56:34.8504004Z 
2022-05-18T04:56:34.8504281Z ----------------------------------------------------------------------
2022-05-18T04:56:34.8504606Z Ran 1 test in 0.002s
2022-05-18T04:56:34.8504775Z 
2022-05-18T04:56:34.8504890Z OK (skipped=1)
2022-05-18T04:56:34.8505026Z 
2022-05-18T04:56:34.8505153Z Generating XML reports...
2022-05-18T04:56:34.8548403Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045634.xml
2022-05-18T04:56:36.1251931Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:36.1267245Z 
2022-05-18T04:56:36.1267537Z Running tests...
2022-05-18T04:56:36.1267973Z ----------------------------------------------------------------------
2022-05-18T04:56:36.1289883Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:56:36.1290372Z 
2022-05-18T04:56:36.1290805Z ----------------------------------------------------------------------
2022-05-18T04:56:36.1291149Z Ran 1 test in 0.002s
2022-05-18T04:56:36.1291315Z 
2022-05-18T04:56:36.1291440Z OK (skipped=1)
2022-05-18T04:56:36.1291596Z 
2022-05-18T04:56:36.1291729Z Generating XML reports...
2022-05-18T04:56:36.1334239Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045636.xml
2022-05-18T04:56:37.4158744Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:37.4174756Z 
2022-05-18T04:56:37.4175088Z Running tests...
2022-05-18T04:56:37.4175526Z ----------------------------------------------------------------------
2022-05-18T04:56:37.4197346Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:56:37.4197646Z 
2022-05-18T04:56:37.4197930Z ----------------------------------------------------------------------
2022-05-18T04:56:37.4198241Z Ran 1 test in 0.002s
2022-05-18T04:56:37.4198726Z 
2022-05-18T04:56:37.4198855Z OK (skipped=1)
2022-05-18T04:56:37.4199014Z 
2022-05-18T04:56:37.4199137Z Generating XML reports...
2022-05-18T04:56:37.4243229Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045637.xml
2022-05-18T04:56:38.6778399Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:38.6792735Z 
2022-05-18T04:56:38.6793044Z Running tests...
2022-05-18T04:56:38.6793482Z ----------------------------------------------------------------------
2022-05-18T04:56:40.3088944Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:40.3456183Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40286
2022-05-18T04:56:40.3562161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40287
2022-05-18T04:56:41.5609689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:41.5682211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:41.5683262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:41.5711461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:41.5718731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:41.6698437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:43.8649544Z ok (5.185s)
2022-05-18T04:56:43.8649778Z 
2022-05-18T04:56:43.8650430Z ----------------------------------------------------------------------
2022-05-18T04:56:43.8650753Z Ran 1 test in 5.186s
2022-05-18T04:56:43.8650917Z 
2022-05-18T04:56:43.8651019Z OK
2022-05-18T04:56:43.8651174Z 
2022-05-18T04:56:43.8651310Z Generating XML reports...
2022-05-18T04:56:43.8708626Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045638.xml
2022-05-18T04:56:45.3143987Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:45.3159099Z 
2022-05-18T04:56:45.3159578Z Running tests...
2022-05-18T04:56:45.3160066Z ----------------------------------------------------------------------
2022-05-18T04:56:45.3183501Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:56:45.3183814Z 
2022-05-18T04:56:45.3184115Z ----------------------------------------------------------------------
2022-05-18T04:56:45.3184428Z Ran 1 test in 0.002s
2022-05-18T04:56:45.3184599Z 
2022-05-18T04:56:45.3184707Z OK (skipped=1)
2022-05-18T04:56:45.3184863Z 
2022-05-18T04:56:45.3184987Z Generating XML reports...
2022-05-18T04:56:45.3227660Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045645.xml
2022-05-18T04:56:46.6010664Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:46.6026513Z 
2022-05-18T04:56:46.6026842Z Running tests...
2022-05-18T04:56:46.6027265Z ----------------------------------------------------------------------
2022-05-18T04:56:46.6049481Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:56:46.6049786Z 
2022-05-18T04:56:46.6050406Z ----------------------------------------------------------------------
2022-05-18T04:56:46.6050738Z Ran 1 test in 0.002s
2022-05-18T04:56:46.6050908Z 
2022-05-18T04:56:46.6050999Z OK (skipped=1)
2022-05-18T04:56:46.6051158Z 
2022-05-18T04:56:46.6051281Z Generating XML reports...
2022-05-18T04:56:46.6094025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045646.xml
2022-05-18T04:56:47.8588309Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:47.8607627Z 
2022-05-18T04:56:47.8607980Z Running tests...
2022-05-18T04:56:47.8608445Z ----------------------------------------------------------------------
2022-05-18T04:56:49.5068297Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:49.5441658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40482
2022-05-18T04:56:49.5550727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40483
2022-05-18T04:56:50.6860107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:50.7055397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:50.7056482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:50.7062007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:50.7068427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:50.8071216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:53.0637685Z ok (5.203s)
2022-05-18T04:56:53.0637902Z 
2022-05-18T04:56:53.0638304Z ----------------------------------------------------------------------
2022-05-18T04:56:53.0638634Z Ran 1 test in 5.203s
2022-05-18T04:56:53.0638781Z 
2022-05-18T04:56:53.0638876Z OK
2022-05-18T04:56:53.0639012Z 
2022-05-18T04:56:53.0639147Z Generating XML reports...
2022-05-18T04:56:53.0696678Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045647.xml
2022-05-18T04:56:54.4860710Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:56:54.4875114Z 
2022-05-18T04:56:54.4875483Z Running tests...
2022-05-18T04:56:54.4875971Z ----------------------------------------------------------------------
2022-05-18T04:56:56.0947269Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:56.1312647Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40608
2022-05-18T04:56:56.1422069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40609
2022-05-18T04:56:57.2918785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:57.3173796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:57.3174563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:57.3222306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:57.3229852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:57.4189169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:59.4505816Z ok (4.963s)
2022-05-18T04:56:59.4506022Z 
2022-05-18T04:56:59.4506395Z ----------------------------------------------------------------------
2022-05-18T04:56:59.4506709Z Ran 1 test in 4.963s
2022-05-18T04:56:59.4506910Z 
2022-05-18T04:56:59.4507007Z OK
2022-05-18T04:56:59.4507143Z 
2022-05-18T04:56:59.4507277Z Generating XML reports...
2022-05-18T04:56:59.4563445Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045654.xml
2022-05-18T04:57:00.8741889Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:00.8756419Z 
2022-05-18T04:57:00.8756737Z Running tests...
2022-05-18T04:57:00.8757170Z ----------------------------------------------------------------------
2022-05-18T04:57:00.8779563Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:57:00.8780079Z 
2022-05-18T04:57:00.8780572Z ----------------------------------------------------------------------
2022-05-18T04:57:00.8781153Z Ran 1 test in 0.002s
2022-05-18T04:57:00.8781323Z 
2022-05-18T04:57:00.8781430Z OK (skipped=1)
2022-05-18T04:57:00.8781587Z 
2022-05-18T04:57:00.8781713Z Generating XML reports...
2022-05-18T04:57:00.8822420Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045700.xml
2022-05-18T04:57:02.1601005Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:02.1616456Z 
2022-05-18T04:57:02.1616709Z Running tests...
2022-05-18T04:57:02.1617449Z ----------------------------------------------------------------------
2022-05-18T04:57:02.1636573Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:57:02.1637665Z 
2022-05-18T04:57:02.1638325Z ----------------------------------------------------------------------
2022-05-18T04:57:02.1638688Z Ran 1 test in 0.002s
2022-05-18T04:57:02.1638854Z 
2022-05-18T04:57:02.1638978Z OK (skipped=1)
2022-05-18T04:57:02.1639134Z 
2022-05-18T04:57:02.1639263Z Generating XML reports...
2022-05-18T04:57:02.1681154Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045702.xml
2022-05-18T04:57:03.4376019Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:03.4390786Z 
2022-05-18T04:57:03.4391217Z Running tests...
2022-05-18T04:57:03.4391704Z ----------------------------------------------------------------------
2022-05-18T04:57:03.4419469Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.003s)
2022-05-18T04:57:03.4419795Z 
2022-05-18T04:57:03.4420085Z ----------------------------------------------------------------------
2022-05-18T04:57:03.4420405Z Ran 1 test in 0.003s
2022-05-18T04:57:03.4420565Z 
2022-05-18T04:57:03.4420672Z OK (skipped=1)
2022-05-18T04:57:03.4420828Z 
2022-05-18T04:57:03.4420951Z Generating XML reports...
2022-05-18T04:57:03.4463486Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045703.xml
2022-05-18T04:57:04.7308275Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:04.7323505Z 
2022-05-18T04:57:04.7323934Z Running tests...
2022-05-18T04:57:04.7324447Z ----------------------------------------------------------------------
2022-05-18T04:57:04.7344488Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:57:04.7344960Z 
2022-05-18T04:57:04.7345403Z ----------------------------------------------------------------------
2022-05-18T04:57:04.7345950Z Ran 1 test in 0.002s
2022-05-18T04:57:04.7346205Z 
2022-05-18T04:57:04.7346315Z OK (skipped=1)
2022-05-18T04:57:04.7346472Z 
2022-05-18T04:57:04.7347106Z Generating XML reports...
2022-05-18T04:57:04.7388800Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045704.xml
2022-05-18T04:57:06.0081768Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:06.0097054Z 
2022-05-18T04:57:06.0097178Z Running tests...
2022-05-18T04:57:06.0097959Z ----------------------------------------------------------------------
2022-05-18T04:57:07.6653046Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:07.7024944Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40874
2022-05-18T04:57:07.7136127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40875
2022-05-18T04:57:08.8468258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:57:08.8478408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:57:08.8479706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:08.8569735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:08.8576651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:57:08.9493392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:57:12.9253294Z ok (6.915s)
2022-05-18T04:57:12.9253521Z 
2022-05-18T04:57:12.9253907Z ----------------------------------------------------------------------
2022-05-18T04:57:12.9254255Z Ran 1 test in 6.916s
2022-05-18T04:57:12.9254420Z 
2022-05-18T04:57:12.9254742Z OK
2022-05-18T04:57:12.9254880Z 
2022-05-18T04:57:12.9255012Z Generating XML reports...
2022-05-18T04:57:12.9312794Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045706.xml
2022-05-18T04:57:14.3793132Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:14.3807984Z 
2022-05-18T04:57:14.3808325Z Running tests...
2022-05-18T04:57:14.3808756Z ----------------------------------------------------------------------
2022-05-18T04:57:16.0406280Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:16.0774723Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41002
2022-05-18T04:57:16.0883076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41003
2022-05-18T04:57:17.1900905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:57:17.2432352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:57:17.2433148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:17.2508612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:17.2516096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:57:17.3446915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:57:21.3020448Z ok (6.921s)
2022-05-18T04:57:21.3020683Z 
2022-05-18T04:57:21.3021092Z ----------------------------------------------------------------------
2022-05-18T04:57:21.3021442Z Ran 1 test in 6.921s
2022-05-18T04:57:21.3021604Z 
2022-05-18T04:57:21.3021699Z OK
2022-05-18T04:57:21.3021832Z 
2022-05-18T04:57:21.3021946Z Generating XML reports...
2022-05-18T04:57:21.3078461Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045714.xml
2022-05-18T04:57:22.7486861Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:22.7502150Z 
2022-05-18T04:57:22.7502585Z Running tests...
2022-05-18T04:57:22.7503085Z ----------------------------------------------------------------------
2022-05-18T04:57:22.7522715Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:57:22.7523037Z 
2022-05-18T04:57:22.7523304Z ----------------------------------------------------------------------
2022-05-18T04:57:22.7523629Z Ran 1 test in 0.002s
2022-05-18T04:57:22.7524063Z 
2022-05-18T04:57:22.7524188Z OK (skipped=1)
2022-05-18T04:57:22.7524344Z 
2022-05-18T04:57:22.7524469Z Generating XML reports...
2022-05-18T04:57:22.7567021Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045722.xml
2022-05-18T04:57:23.9888600Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:23.9904155Z 
2022-05-18T04:57:23.9904561Z Running tests...
2022-05-18T04:57:23.9905058Z ----------------------------------------------------------------------
2022-05-18T04:57:23.9925053Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:57:23.9925347Z 
2022-05-18T04:57:23.9925798Z ----------------------------------------------------------------------
2022-05-18T04:57:23.9926293Z Ran 1 test in 0.002s
2022-05-18T04:57:23.9926458Z 
2022-05-18T04:57:23.9926567Z OK (skipped=1)
2022-05-18T04:57:23.9926720Z 
2022-05-18T04:57:23.9926843Z Generating XML reports...
2022-05-18T04:57:23.9968902Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045723.xml
2022-05-18T04:57:25.2630287Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:25.2647016Z 
2022-05-18T04:57:25.2647456Z Running tests...
2022-05-18T04:57:25.2648317Z ----------------------------------------------------------------------
2022-05-18T04:57:25.2675883Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.003s)
2022-05-18T04:57:25.2676532Z 
2022-05-18T04:57:25.2677089Z ----------------------------------------------------------------------
2022-05-18T04:57:25.2677740Z Ran 1 test in 0.003s
2022-05-18T04:57:25.2678041Z 
2022-05-18T04:57:25.2678252Z OK (skipped=1)
2022-05-18T04:57:25.2678526Z 
2022-05-18T04:57:25.2678763Z Generating XML reports...
2022-05-18T04:57:25.2722013Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045725.xml
2022-05-18T04:57:26.5484165Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:26.5499938Z 
2022-05-18T04:57:26.5500091Z Running tests...
2022-05-18T04:57:26.5500954Z ----------------------------------------------------------------------
2022-05-18T04:57:26.5520242Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:57:26.5520554Z 
2022-05-18T04:57:26.5520840Z ----------------------------------------------------------------------
2022-05-18T04:57:26.5521150Z Ran 1 test in 0.002s
2022-05-18T04:57:26.5521311Z 
2022-05-18T04:57:26.5521425Z OK (skipped=1)
2022-05-18T04:57:26.5521579Z 
2022-05-18T04:57:26.5521701Z Generating XML reports...
2022-05-18T04:57:26.5564008Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045726.xml
2022-05-18T04:57:27.8067839Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:27.8082225Z 
2022-05-18T04:57:27.8082406Z Running tests...
2022-05-18T04:57:27.8082852Z ----------------------------------------------------------------------
2022-05-18T04:57:27.8103071Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-05-18T04:57:27.8103396Z 
2022-05-18T04:57:27.8103657Z ----------------------------------------------------------------------
2022-05-18T04:57:27.8103981Z Ran 1 test in 0.002s
2022-05-18T04:57:27.8104141Z 
2022-05-18T04:57:27.8104467Z OK (skipped=1)
2022-05-18T04:57:27.8104627Z 
2022-05-18T04:57:27.8104755Z Generating XML reports...
2022-05-18T04:57:27.8146052Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045727.xml
2022-05-18T04:57:29.0615135Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:29.0630838Z 
2022-05-18T04:57:29.0631005Z Running tests...
2022-05-18T04:57:29.0631695Z ----------------------------------------------------------------------
2022-05-18T04:57:29.0651626Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-05-18T04:57:29.0652110Z 
2022-05-18T04:57:29.0652429Z ----------------------------------------------------------------------
2022-05-18T04:57:29.0652788Z Ran 1 test in 0.002s
2022-05-18T04:57:29.0652949Z 
2022-05-18T04:57:29.0653065Z OK (skipped=1)
2022-05-18T04:57:29.0653219Z 
2022-05-18T04:57:29.0653343Z Generating XML reports...
2022-05-18T04:57:29.0695937Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045729.xml
2022-05-18T04:57:30.3394912Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:30.3410399Z 
2022-05-18T04:57:30.3410842Z Running tests...
2022-05-18T04:57:30.3411328Z ----------------------------------------------------------------------
2022-05-18T04:57:30.3433018Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-05-18T04:57:30.3433382Z 
2022-05-18T04:57:30.3433667Z ----------------------------------------------------------------------
2022-05-18T04:57:30.3433995Z Ran 1 test in 0.002s
2022-05-18T04:57:30.3434155Z 
2022-05-18T04:57:30.3434265Z OK (skipped=1)
2022-05-18T04:57:30.3434419Z 
2022-05-18T04:57:30.3434525Z Generating XML reports...
2022-05-18T04:57:30.3477825Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045730.xml
2022-05-18T04:57:31.6252939Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:31.6268593Z 
2022-05-18T04:57:31.6268881Z Running tests...
2022-05-18T04:57:31.6269327Z ----------------------------------------------------------------------
2022-05-18T04:57:31.6289129Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:57:31.6290065Z 
2022-05-18T04:57:31.6290563Z ----------------------------------------------------------------------
2022-05-18T04:57:31.6291139Z Ran 1 test in 0.002s
2022-05-18T04:57:31.6291415Z 
2022-05-18T04:57:31.6291588Z OK (skipped=1)
2022-05-18T04:57:31.6291856Z 
2022-05-18T04:57:31.6291994Z Generating XML reports...
2022-05-18T04:57:31.6333706Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045731.xml
2022-05-18T04:57:32.8787436Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:32.8802090Z 
2022-05-18T04:57:32.8802385Z Running tests...
2022-05-18T04:57:32.8802827Z ----------------------------------------------------------------------
2022-05-18T04:57:34.5126139Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:34.5492301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41410
2022-05-18T04:57:34.5599609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41411
2022-05-18T04:57:35.7106734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:57:35.7107566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:57:35.7108556Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:35.7208290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:35.7216155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:57:35.8123058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:57:37.4673288Z ok (4.587s)
2022-05-18T04:57:37.4673534Z 
2022-05-18T04:57:37.4673943Z ----------------------------------------------------------------------
2022-05-18T04:57:37.4674279Z Ran 1 test in 4.587s
2022-05-18T04:57:37.4674445Z 
2022-05-18T04:57:37.4674539Z OK
2022-05-18T04:57:37.4674655Z 
2022-05-18T04:57:37.4674796Z Generating XML reports...
2022-05-18T04:57:37.4732978Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045732.xml
2022-05-18T04:57:38.8928818Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:38.8944357Z 
2022-05-18T04:57:38.8944512Z Running tests...
2022-05-18T04:57:38.8945674Z ----------------------------------------------------------------------
2022-05-18T04:57:40.5242918Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:40.5613768Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41532
2022-05-18T04:57:40.5720107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41533
2022-05-18T04:57:41.7271624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:57:41.7721981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:57:41.7722796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:41.7778106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:41.7785614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:57:41.8736851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:57:43.9805207Z ok (5.086s)
2022-05-18T04:57:43.9805479Z 
2022-05-18T04:57:43.9805870Z ----------------------------------------------------------------------
2022-05-18T04:57:43.9806209Z Ran 1 test in 5.086s
2022-05-18T04:57:43.9806373Z 
2022-05-18T04:57:43.9806465Z OK
2022-05-18T04:57:43.9806605Z 
2022-05-18T04:57:43.9806721Z Generating XML reports...
2022-05-18T04:57:43.9865008Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045738.xml
2022-05-18T04:57:45.4235032Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:45.4250515Z 
2022-05-18T04:57:45.4250805Z Running tests...
2022-05-18T04:57:45.4251245Z ----------------------------------------------------------------------
2022-05-18T04:57:47.0628305Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:47.1000278Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41658
2022-05-18T04:57:47.1109062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41659
2022-05-18T04:57:48.2369765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:57:48.2549451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:57:48.2550761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:48.2572603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:48.2579084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:57:48.3565384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:57:50.5195853Z ok (5.094s)
2022-05-18T04:57:50.5196103Z 
2022-05-18T04:57:50.5196497Z ----------------------------------------------------------------------
2022-05-18T04:57:50.5196854Z Ran 1 test in 5.094s
2022-05-18T04:57:50.5197023Z 
2022-05-18T04:57:50.5197116Z OK
2022-05-18T04:57:50.5197231Z 
2022-05-18T04:57:50.5197365Z Generating XML reports...
2022-05-18T04:57:50.5255651Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045745.xml
2022-05-18T04:57:51.9571366Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:51.9586052Z 
2022-05-18T04:57:51.9586341Z Running tests...
2022-05-18T04:57:51.9586773Z ----------------------------------------------------------------------
2022-05-18T04:57:51.9607537Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:57:51.9607860Z 
2022-05-18T04:57:51.9608135Z ----------------------------------------------------------------------
2022-05-18T04:57:51.9608438Z Ran 1 test in 0.002s
2022-05-18T04:57:51.9608899Z 
2022-05-18T04:57:51.9609014Z OK (skipped=1)
2022-05-18T04:57:51.9609167Z 
2022-05-18T04:57:51.9609293Z Generating XML reports...
2022-05-18T04:57:51.9650606Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045751.xml
2022-05-18T04:57:53.2333360Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:53.2348751Z 
2022-05-18T04:57:53.2349244Z Running tests...
2022-05-18T04:57:53.2349746Z ----------------------------------------------------------------------
2022-05-18T04:57:53.2369191Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:57:53.2369775Z 
2022-05-18T04:57:53.2370100Z ----------------------------------------------------------------------
2022-05-18T04:57:53.2370429Z Ran 1 test in 0.002s
2022-05-18T04:57:53.2370591Z 
2022-05-18T04:57:53.2370701Z OK (skipped=1)
2022-05-18T04:57:53.2370851Z 
2022-05-18T04:57:53.2370975Z Generating XML reports...
2022-05-18T04:57:53.2413012Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045753.xml
2022-05-18T04:57:54.4981364Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:54.4997064Z 
2022-05-18T04:57:54.4997220Z Running tests...
2022-05-18T04:57:54.4997968Z ----------------------------------------------------------------------
2022-05-18T04:57:54.5018115Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:57:54.5018893Z 
2022-05-18T04:57:54.5019228Z ----------------------------------------------------------------------
2022-05-18T04:57:54.5019582Z Ran 1 test in 0.002s
2022-05-18T04:57:54.5019745Z 
2022-05-18T04:57:54.5019854Z OK (skipped=1)
2022-05-18T04:57:54.5020008Z 
2022-05-18T04:57:54.5020126Z Generating XML reports...
2022-05-18T04:57:54.5062815Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045754.xml
2022-05-18T04:57:55.7450370Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:55.7466396Z 
2022-05-18T04:57:55.7466947Z Running tests...
2022-05-18T04:57:55.7467454Z ----------------------------------------------------------------------
2022-05-18T04:57:55.7488653Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:57:55.7488998Z 
2022-05-18T04:57:55.7489283Z ----------------------------------------------------------------------
2022-05-18T04:57:55.7490177Z Ran 1 test in 0.002s
2022-05-18T04:57:55.7490359Z 
2022-05-18T04:57:55.7490477Z OK (skipped=1)
2022-05-18T04:57:55.7490615Z 
2022-05-18T04:57:55.7490737Z Generating XML reports...
2022-05-18T04:57:55.7534520Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045755.xml
2022-05-18T04:57:57.0042084Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:57.0057392Z 
2022-05-18T04:57:57.0057719Z Running tests...
2022-05-18T04:57:57.0058156Z ----------------------------------------------------------------------
2022-05-18T04:57:57.0077054Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-05-18T04:57:57.0077383Z 
2022-05-18T04:57:57.0077644Z ----------------------------------------------------------------------
2022-05-18T04:57:57.0077971Z Ran 1 test in 0.002s
2022-05-18T04:57:57.0078134Z 
2022-05-18T04:57:57.0078261Z OK (skipped=1)
2022-05-18T04:57:57.0078415Z 
2022-05-18T04:57:57.0078524Z Generating XML reports...
2022-05-18T04:57:57.0120648Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045757.xml
2022-05-18T04:57:58.2921849Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:58.2937006Z 
2022-05-18T04:57:58.2937212Z Running tests...
2022-05-18T04:57:58.2937649Z ----------------------------------------------------------------------
2022-05-18T04:57:58.2958090Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-05-18T04:57:58.2958425Z 
2022-05-18T04:57:58.2958704Z ----------------------------------------------------------------------
2022-05-18T04:57:58.2959012Z Ran 1 test in 0.002s
2022-05-18T04:57:58.2959176Z 
2022-05-18T04:57:58.2959285Z OK (skipped=1)
2022-05-18T04:57:58.2960586Z 
2022-05-18T04:57:58.2961116Z Generating XML reports...
2022-05-18T04:57:58.3002587Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045758.xml
2022-05-18T04:57:59.5714321Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:57:59.5729432Z 
2022-05-18T04:57:59.5729799Z Running tests...
2022-05-18T04:57:59.5730714Z ----------------------------------------------------------------------
2022-05-18T04:58:01.2588514Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:01.2960242Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41994
2022-05-18T04:58:01.3069667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41995
2022-05-18T04:58:02.4867323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:02.5105960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:02.5106775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:02.5172416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:02.5179870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:02.6121162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:05.6172203Z ok (6.044s)
2022-05-18T04:58:05.6172423Z 
2022-05-18T04:58:05.6172814Z ----------------------------------------------------------------------
2022-05-18T04:58:05.6173135Z Ran 1 test in 6.044s
2022-05-18T04:58:05.6173299Z 
2022-05-18T04:58:05.6173399Z OK
2022-05-18T04:58:05.6173534Z 
2022-05-18T04:58:05.6173668Z Generating XML reports...
2022-05-18T04:58:05.6231046Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045759.xml
2022-05-18T04:58:07.0360484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:58:07.0383208Z 
2022-05-18T04:58:07.0383560Z Running tests...
2022-05-18T04:58:07.0384041Z ----------------------------------------------------------------------
2022-05-18T04:58:07.0401257Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-05-18T04:58:07.0402623Z 
2022-05-18T04:58:07.0402937Z ----------------------------------------------------------------------
2022-05-18T04:58:07.0403258Z Ran 1 test in 0.003s
2022-05-18T04:58:07.0403422Z 
2022-05-18T04:58:07.0403535Z OK (skipped=1)
2022-05-18T04:58:07.0403692Z 
2022-05-18T04:58:07.0403817Z Generating XML reports...
2022-05-18T04:58:07.0444452Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045807.xml
2022-05-18T04:58:08.3164899Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:58:08.3181058Z 
2022-05-18T04:58:08.3181501Z Running tests...
2022-05-18T04:58:08.3181955Z ----------------------------------------------------------------------
2022-05-18T04:58:09.9563470Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:09.9928880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42156
2022-05-18T04:58:10.0035910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42157
2022-05-18T04:58:11.1664874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:11.1912905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:11.1913729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:11.1968456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:11.1975557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:11.2928791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:12.5189321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpre7iz84r
2022-05-18T04:58:12.5189915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpre7iz84r/_remote_module_non_scriptable.py
2022-05-18T04:58:12.5655539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikxoegxp
2022-05-18T04:58:12.5656675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikxoegxp/_remote_module_non_scriptable.py
2022-05-18T04:58:13.9131069Z ok (5.595s)
2022-05-18T04:58:13.9131438Z 
2022-05-18T04:58:13.9132120Z ----------------------------------------------------------------------
2022-05-18T04:58:13.9132742Z Ran 1 test in 5.595s
2022-05-18T04:58:13.9133041Z 
2022-05-18T04:58:13.9133213Z OK
2022-05-18T04:58:13.9133453Z 
2022-05-18T04:58:13.9133675Z Generating XML reports...
2022-05-18T04:58:13.9190518Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045808.xml
2022-05-18T04:58:15.3605193Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:58:15.3620731Z 
2022-05-18T04:58:15.3620881Z Running tests...
2022-05-18T04:58:15.3621537Z ----------------------------------------------------------------------
2022-05-18T04:58:17.0013788Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:17.0383532Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42279
2022-05-18T04:58:17.0492732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42280
2022-05-18T04:58:18.2026769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:18.2204531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:18.2205337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:18.2229649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:18.2236577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:18.3219900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:19.5064480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvx53wh9a
2022-05-18T04:58:19.5065122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvx53wh9a/_remote_module_non_scriptable.py
2022-05-18T04:58:19.6256509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2uj50oh
2022-05-18T04:58:19.6257633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2uj50oh/_remote_module_non_scriptable.py
2022-05-18T04:58:20.9601476Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:58:20.9632040Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:58:21.3594362Z ok (5.997s)
2022-05-18T04:58:21.3594562Z 
2022-05-18T04:58:21.3594950Z ----------------------------------------------------------------------
2022-05-18T04:58:21.3595291Z Ran 1 test in 5.997s
2022-05-18T04:58:21.3595458Z 
2022-05-18T04:58:21.3595551Z OK
2022-05-18T04:58:21.3595688Z 
2022-05-18T04:58:21.3595803Z Generating XML reports...
2022-05-18T04:58:21.3653466Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045815.xml
2022-05-18T04:58:22.8031220Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:58:22.8046496Z 
2022-05-18T04:58:22.8046696Z Running tests...
2022-05-18T04:58:22.8047583Z ----------------------------------------------------------------------
2022-05-18T04:58:24.4511749Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:24.4885779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42406
2022-05-18T04:58:24.4995091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42407
2022-05-18T04:58:25.6671513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:25.6969863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:25.6971746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:25.6975384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:25.6981599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:25.7981499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:25.8090019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:58:25.8091344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:58:25.8092038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:58:25.8092707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:58:25.8094121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:58:25.8195136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:58:25.8196613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:58:25.8197280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:58:27.1520741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwccm7ysl
2022-05-18T04:58:27.1521842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwccm7ysl/_remote_module_non_scriptable.py
2022-05-18T04:58:27.1790144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0xtytei
2022-05-18T04:58:27.1792082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0xtytei/_remote_module_non_scriptable.py
2022-05-18T04:58:32.9164205Z ok (10.111s)
2022-05-18T04:58:32.9164583Z 
2022-05-18T04:58:32.9165335Z ----------------------------------------------------------------------
2022-05-18T04:58:32.9165850Z Ran 1 test in 10.112s
2022-05-18T04:58:32.9166024Z 
2022-05-18T04:58:32.9166119Z OK
2022-05-18T04:58:32.9166252Z 
2022-05-18T04:58:32.9166383Z Generating XML reports...
2022-05-18T04:58:32.9223113Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045822.xml
2022-05-18T04:58:34.3315890Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:58:34.3330023Z 
2022-05-18T04:58:34.3330447Z Running tests...
2022-05-18T04:58:34.3331369Z ----------------------------------------------------------------------
2022-05-18T04:58:35.9451350Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:35.9818773Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42536
2022-05-18T04:58:35.9930279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42537
2022-05-18T04:58:37.1252244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:37.1405588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:37.1406402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:37.1455839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:37.1463341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:37.2417509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:37.2580341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:58:37.2580891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:58:37.2581590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:58:37.2582285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:58:37.2583697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:58:37.2685438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:58:37.2686678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:58:37.2687878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:58:38.5729074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4gazyca
2022-05-18T04:58:38.5730563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4gazyca/_remote_module_non_scriptable.py
2022-05-18T04:58:38.6110096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8c6jukn
2022-05-18T04:58:38.6112438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8c6jukn/_remote_module_non_scriptable.py
2022-05-18T04:58:44.2091100Z ok (9.876s)
2022-05-18T04:58:44.2091333Z 
2022-05-18T04:58:44.2091720Z ----------------------------------------------------------------------
2022-05-18T04:58:44.2092063Z Ran 1 test in 9.876s
2022-05-18T04:58:44.2092212Z 
2022-05-18T04:58:44.2092311Z OK
2022-05-18T04:58:44.2092453Z 
2022-05-18T04:58:44.2092586Z Generating XML reports...
2022-05-18T04:58:44.2148941Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045834.xml
2022-05-18T04:58:44.6160877Z Running distributed tests for the gloo backend with env init_method
2022-05-18T04:58:44.6163798Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:58:44.615983]
2022-05-18T04:58:45.7661096Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:58:45.7708016Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-05-18T04:58:45.7742631Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7743122Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7743547Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7743960Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7744386Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7744967Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7745446Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7745922Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7746433Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7746973Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7747526Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7748035Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7748564Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7749080Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7749555Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7750016Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7750476Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7750903Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7751308Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7751749Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7752229Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7752699Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7753087Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7753484Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7753907Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7754307Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7754720Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7755141Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7755533Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7755912Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7756300Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7756792Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7757177Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7757563Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7757971Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7758371Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7758783Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7759204Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7759641Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7760061Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7760504Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7760928Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7761335Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7761763Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7762285Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7762678Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7763113Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7763549Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7763958Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7764348Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7764764Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7765191Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7765577Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7765988Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7766397Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7766797Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7767171Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7767566Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7767960Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7768320Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7768692Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7769074Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7769464Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7770514Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7770907Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7771292Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7771652Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7772040Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7772430Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7772803Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7773211Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7773592Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7773946Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7774402Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7774797Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7775192Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7775575Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7775968Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7776353Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7776740Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7777165Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7777594Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7778018Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7778466Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7778916Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7779359Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7779862Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7780291Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7780722Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7781144Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7781588Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7782040Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7782492Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7782930Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7783374Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7783800Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7784168Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7784540Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7784902Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7785261Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7785620Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7786013Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7786401Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7786762Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7787161Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7787563Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7787937Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7788332Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7788734Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7789153Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7789541Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7789949Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7790360Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7790747Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7791168Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7791637Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7792036Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7792421Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7792789Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7793172Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7793539Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7793921Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7794312Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7794745Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7795250Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7795702Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7796111Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7796505Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7796993Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7797436Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7797881Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7798313Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7798730Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7799149Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7799556Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7799929Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7800313Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7800704Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7801115Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7801546Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7801962Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7802373Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7802827Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7803323Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7803913Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7804526Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7805129Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7805732Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7806311Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7806903Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7807545Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7808150Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7808680Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7809170Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7810157Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7810577Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7810957Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7811362Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7811755Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7812155Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7812594Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7813052Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7813595Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7814008Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7814394Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7814801Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7815214Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7815643Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7816053Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7816447Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7816874Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7817300Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7817721Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7818111Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7818515Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7818932Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7819313Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7819728Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7820185Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7820594Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7820976Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7821373Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7821790Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7822189Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7822566Z test_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7822926Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7823276Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7823650Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7824024Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7824374Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7824755Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7825133Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7825558Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7825908Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7826284Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7826678Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7827051Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7827414Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7827756Z test_isend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7828115Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7828512Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7828921Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7829369Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7829803Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7830212Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7830627Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7831101Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7831523Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7831948Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7832344Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7832755Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7833162Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7833564Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7833947Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7834343Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7834801Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7835275Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7835739Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7836182Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7836639Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7837070Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7837494Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7837910Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7838316Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7838748Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7839198Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7839673Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7840164Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7840621Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7841011Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7841392Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7841790Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7842176Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7842602Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7842983Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7843369Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7843737Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7844077Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7844450Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7844827Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7845175Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7845539Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7845920Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7846283Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7846645Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7847009Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7847383Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7847735Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7848174Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7848563Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7848923Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7849301Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7850293Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7850673Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7851090Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7851528Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7851956Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7852331Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7852730Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7853156Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7853546Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7853936Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7854345Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7854759Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7855168Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7855568Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7855969Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7856355Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7856734Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7857153Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7857580Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:45.7858015Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:46.9219966Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:46.9234908Z 
2022-05-18T04:58:46.9235051Z Running tests...
2022-05-18T04:58:46.9236126Z ----------------------------------------------------------------------
2022-05-18T04:58:48.5573688Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:48.5949936Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42701
2022-05-18T04:58:48.6057517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42702
2022-05-18T04:58:49.7764772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:49.7765331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:49.7766102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:49.7766807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:49.7873411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:49.8780115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:51.0617768Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:58:51.0618822Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:58:51.1735374Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:58:52.0383471Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:58:52.0384194Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:58:52.0384766Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:58:52.0385614Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:58:52.0386529Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:58:52.0527262Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:58:52.0527861Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:58:52.0528714Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:58:52.0529895Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:58:52.0671786Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:58:52.0672663Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:58:52.0673335Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:58:52.0674137Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:58:52.4149764Z ok (5.491s)
2022-05-18T04:58:52.4149952Z 
2022-05-18T04:58:52.4150332Z ----------------------------------------------------------------------
2022-05-18T04:58:52.4150909Z Ran 1 test in 5.491s
2022-05-18T04:58:52.4151158Z 
2022-05-18T04:58:52.4151254Z OK
2022-05-18T04:58:52.4151391Z 
2022-05-18T04:58:52.4151523Z Generating XML reports...
2022-05-18T04:58:52.4208958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045846.xml
2022-05-18T04:58:53.8615895Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:53.8630664Z 
2022-05-18T04:58:53.8631155Z Running tests...
2022-05-18T04:58:53.8631669Z ----------------------------------------------------------------------
2022-05-18T04:58:53.8677415Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.005s)
2022-05-18T04:58:53.8677740Z 
2022-05-18T04:58:53.8678020Z ----------------------------------------------------------------------
2022-05-18T04:58:53.8678335Z Ran 1 test in 0.005s
2022-05-18T04:58:53.8678501Z 
2022-05-18T04:58:53.8678618Z OK (skipped=1)
2022-05-18T04:58:53.8678775Z 
2022-05-18T04:58:53.8678899Z Generating XML reports...
2022-05-18T04:58:53.8721569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045853.xml
2022-05-18T04:58:55.1393576Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:55.1408868Z 
2022-05-18T04:58:55.1409288Z Running tests...
2022-05-18T04:58:55.1409986Z ----------------------------------------------------------------------
2022-05-18T04:58:56.7964735Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:56.8333879Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42854
2022-05-18T04:58:56.8442493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42855
2022-05-18T04:58:57.9999254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:57.9999834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:58.0000648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:58.0001350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:58.0109608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:58.1010489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:58.2490747Z ok (3.108s)
2022-05-18T04:58:58.2491748Z 
2022-05-18T04:58:58.2492513Z ----------------------------------------------------------------------
2022-05-18T04:58:58.2492836Z Ran 1 test in 3.108s
2022-05-18T04:58:58.2493004Z 
2022-05-18T04:58:58.2493100Z OK
2022-05-18T04:58:58.2493244Z 
2022-05-18T04:58:58.2493383Z Generating XML reports...
2022-05-18T04:58:58.2559344Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045855.xml
2022-05-18T04:58:59.6528455Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:59.6543492Z 
2022-05-18T04:58:59.6543636Z Running tests...
2022-05-18T04:58:59.6544092Z ----------------------------------------------------------------------
2022-05-18T04:59:01.2734231Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:01.2851145Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.630s)
2022-05-18T04:59:01.2851715Z 
2022-05-18T04:59:01.2851991Z ----------------------------------------------------------------------
2022-05-18T04:59:01.2852323Z Ran 1 test in 1.631s
2022-05-18T04:59:01.2852487Z 
2022-05-18T04:59:01.2852603Z OK (skipped=1)
2022-05-18T04:59:01.2853053Z 
2022-05-18T04:59:01.2853183Z Generating XML reports...
2022-05-18T04:59:01.2890068Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045859.xml
2022-05-18T04:59:02.6823958Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:02.6839244Z 
2022-05-18T04:59:02.6839661Z Running tests...
2022-05-18T04:59:02.6840144Z ----------------------------------------------------------------------
2022-05-18T04:59:04.3148035Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:04.3511319Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43005
2022-05-18T04:59:04.3621427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43006
2022-05-18T04:59:05.5659478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:05.5660040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:05.5660818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:05.5661740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:05.5770432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:05.5862317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmq6vpf6s
2022-05-18T04:59:05.5864830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmq6vpf6s/_remote_module_non_scriptable.py
2022-05-18T04:59:05.6670786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:05.6767238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp437f3w9p
2022-05-18T04:59:05.6769866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp437f3w9p/_remote_module_non_scriptable.py
2022-05-18T04:59:05.6971176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:05.6971695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:05.8670438Z ok (3.183s)
2022-05-18T04:59:05.8670714Z 
2022-05-18T04:59:05.8671093Z ----------------------------------------------------------------------
2022-05-18T04:59:05.8671436Z Ran 1 test in 3.183s
2022-05-18T04:59:05.8671581Z 
2022-05-18T04:59:05.8671684Z OK
2022-05-18T04:59:05.8671820Z 
2022-05-18T04:59:05.8671954Z Generating XML reports...
2022-05-18T04:59:05.8729564Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045902.xml
2022-05-18T04:59:07.2965488Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:07.2981437Z 
2022-05-18T04:59:07.2981694Z Running tests...
2022-05-18T04:59:07.2982130Z ----------------------------------------------------------------------
2022-05-18T04:59:08.9360203Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:08.9725703Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43124
2022-05-18T04:59:08.9832981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43125
2022-05-18T04:59:10.2035210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:10.2035759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:10.2036545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:10.2037486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:10.2044250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:10.2045154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:10.2137477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppbts3rxz
2022-05-18T04:59:10.2140054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppbts3rxz/_remote_module_non_scriptable.py
2022-05-18T04:59:10.2143682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8pqyfvb9
2022-05-18T04:59:10.2146874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8pqyfvb9/_remote_module_non_scriptable.py
2022-05-18T04:59:10.2350915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:10.2351426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:10.3880690Z ok (3.090s)
2022-05-18T04:59:10.3880997Z 
2022-05-18T04:59:10.3881448Z ----------------------------------------------------------------------
2022-05-18T04:59:10.3882035Z Ran 1 test in 3.090s
2022-05-18T04:59:10.3882200Z 
2022-05-18T04:59:10.3882291Z OK
2022-05-18T04:59:10.3882427Z 
2022-05-18T04:59:10.3882558Z Generating XML reports...
2022-05-18T04:59:10.3940368Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045907.xml
2022-05-18T04:59:11.7924781Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:11.7940316Z 
2022-05-18T04:59:11.7940806Z Running tests...
2022-05-18T04:59:11.7941292Z ----------------------------------------------------------------------
2022-05-18T04:59:13.4122863Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:13.4485121Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43243
2022-05-18T04:59:13.4596516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43244
2022-05-18T04:59:14.6275086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:14.6275651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:14.6276439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:14.6277116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:14.6285011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:14.6285898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:15.9699654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkm09f4n
2022-05-18T04:59:15.9700786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkm09f4n/_remote_module_non_scriptable.py
2022-05-18T04:59:15.9994008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62lgc45_
2022-05-18T04:59:15.9996669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62lgc45_/_remote_module_non_scriptable.py
2022-05-18T04:59:16.5642907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:16.5643906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:16.5896264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:16.5897233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:16.6226360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:16.6227337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:16.6475040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:16.6475986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:16.7781659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:16.7782606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:16.8032580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:16.8033558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:17.1686832Z ok (5.374s)
2022-05-18T04:59:17.1687058Z 
2022-05-18T04:59:17.1687493Z ----------------------------------------------------------------------
2022-05-18T04:59:17.1687839Z Ran 1 test in 5.375s
2022-05-18T04:59:17.1688002Z 
2022-05-18T04:59:17.1688083Z OK
2022-05-18T04:59:17.1688221Z 
2022-05-18T04:59:17.1688353Z Generating XML reports...
2022-05-18T04:59:17.1744759Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045911.xml
2022-05-18T04:59:18.5906062Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:18.5920339Z 
2022-05-18T04:59:18.5920663Z Running tests...
2022-05-18T04:59:18.5921105Z ----------------------------------------------------------------------
2022-05-18T04:59:20.2068448Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:20.2434296Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43364
2022-05-18T04:59:20.2546124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43365
2022-05-18T04:59:21.4682703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:21.4683292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:21.4684102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:21.4684801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:21.4793449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:21.5698882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:22.7771829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bie8ltn
2022-05-18T04:59:22.7772457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bie8ltn/_remote_module_non_scriptable.py
2022-05-18T04:59:22.8729409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5rvmim5
2022-05-18T04:59:22.8730950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5rvmim5/_remote_module_non_scriptable.py
2022-05-18T04:59:22.8985232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:22.8985908Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:22.9156139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:22.9156830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:23.1672371Z ok (4.570s)
2022-05-18T04:59:23.1672675Z 
2022-05-18T04:59:23.1673104Z ----------------------------------------------------------------------
2022-05-18T04:59:23.1673437Z Ran 1 test in 4.570s
2022-05-18T04:59:23.1673837Z 
2022-05-18T04:59:23.1673938Z OK
2022-05-18T04:59:23.1674069Z 
2022-05-18T04:59:23.1674182Z Generating XML reports...
2022-05-18T04:59:23.1678041Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045918.xml
2022-05-18T04:59:24.6070763Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:24.6086476Z 
2022-05-18T04:59:24.6086613Z Running tests...
2022-05-18T04:59:24.6087262Z ----------------------------------------------------------------------
2022-05-18T04:59:26.2678092Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:26.3053045Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43485
2022-05-18T04:59:26.3162771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43486
2022-05-18T04:59:27.5103415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:27.5103997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:27.5105035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:27.5105731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:27.5112639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:27.5113544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:28.8287492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3caz3vtw
2022-05-18T04:59:28.8288098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3caz3vtw/_remote_module_non_scriptable.py
2022-05-18T04:59:28.8426623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph79ipupl
2022-05-18T04:59:28.8429252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph79ipupl/_remote_module_non_scriptable.py
2022-05-18T04:59:28.8689434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:28.8690397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:28.8878476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:28.8879154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:29.2237740Z ok (4.615s)
2022-05-18T04:59:29.2238153Z 
2022-05-18T04:59:29.2238808Z ----------------------------------------------------------------------
2022-05-18T04:59:29.2239447Z Ran 1 test in 4.615s
2022-05-18T04:59:29.2239752Z 
2022-05-18T04:59:29.2239912Z OK
2022-05-18T04:59:29.2240133Z 
2022-05-18T04:59:29.2240396Z Generating XML reports...
2022-05-18T04:59:29.2298047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045924.xml
2022-05-18T04:59:30.6720515Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:30.6736553Z 
2022-05-18T04:59:30.6736695Z Running tests...
2022-05-18T04:59:30.6737440Z ----------------------------------------------------------------------
2022-05-18T04:59:32.3460973Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:32.3834016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43606
2022-05-18T04:59:32.3944799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43607
2022-05-18T04:59:33.5434267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:33.5435045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:33.5435830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:33.5436532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:33.5545676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:33.6448513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:34.8967069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuw94k80
2022-05-18T04:59:34.8967653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuw94k80/_remote_module_non_scriptable.py
2022-05-18T04:59:34.9316784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_7hqmg0
2022-05-18T04:59:34.9318951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_7hqmg0/_remote_module_non_scriptable.py
2022-05-18T04:59:34.9535253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:34.9536009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:35.5024129Z ok (4.828s)
2022-05-18T04:59:35.5024317Z 
2022-05-18T04:59:35.5024710Z ----------------------------------------------------------------------
2022-05-18T04:59:35.5025038Z Ran 1 test in 4.829s
2022-05-18T04:59:35.5025203Z 
2022-05-18T04:59:35.5025299Z OK
2022-05-18T04:59:35.5025435Z 
2022-05-18T04:59:35.5025565Z Generating XML reports...
2022-05-18T04:59:35.5082186Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045930.xml
2022-05-18T04:59:36.9432865Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:36.9448322Z 
2022-05-18T04:59:36.9448474Z Running tests...
2022-05-18T04:59:36.9449168Z ----------------------------------------------------------------------
2022-05-18T04:59:38.5906311Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:38.6282318Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43727
2022-05-18T04:59:38.6394079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43728
2022-05-18T04:59:39.8364962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:39.8365529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:39.8366328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:39.8367035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:39.8373816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:39.8374623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:41.1754603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvzdnfplu
2022-05-18T04:59:41.1755557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvzdnfplu/_remote_module_non_scriptable.py
2022-05-18T04:59:41.1863891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ycd3rzq
2022-05-18T04:59:41.1866667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ycd3rzq/_remote_module_non_scriptable.py
2022-05-18T04:59:41.7477213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:41.7478034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:41.7745074Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:41.7745877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:42.1495733Z ok (5.204s)
2022-05-18T04:59:42.1495976Z 
2022-05-18T04:59:42.1496376Z ----------------------------------------------------------------------
2022-05-18T04:59:42.1496718Z Ran 1 test in 5.205s
2022-05-18T04:59:42.1496885Z 
2022-05-18T04:59:42.1496986Z OK
2022-05-18T04:59:42.1497100Z 
2022-05-18T04:59:42.1497237Z Generating XML reports...
2022-05-18T04:59:42.1554756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045936.xml
2022-05-18T04:59:43.5808393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:43.5823279Z 
2022-05-18T04:59:43.5823631Z Running tests...
2022-05-18T04:59:43.5824149Z ----------------------------------------------------------------------
2022-05-18T04:59:45.1993403Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:45.2359902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43848
2022-05-18T04:59:45.2470140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43849
2022-05-18T04:59:46.4496472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:46.4497163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:46.4497947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:46.4498670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:46.4606510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:46.5511129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:47.7662872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqk8t65vc
2022-05-18T04:59:47.7664263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqk8t65vc/_remote_module_non_scriptable.py
2022-05-18T04:59:47.8425925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpczhjmghf
2022-05-18T04:59:47.8426870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpczhjmghf/_remote_module_non_scriptable.py
2022-05-18T04:59:48.1633501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:48.1634069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:48.1859339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:48.1859842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:48.4548926Z ok (4.872s)
2022-05-18T04:59:48.4549110Z 
2022-05-18T04:59:48.4549494Z ----------------------------------------------------------------------
2022-05-18T04:59:48.4549818Z Ran 1 test in 4.873s
2022-05-18T04:59:48.4549986Z 
2022-05-18T04:59:48.4550082Z OK
2022-05-18T04:59:48.4550219Z 
2022-05-18T04:59:48.4550348Z Generating XML reports...
2022-05-18T04:59:48.4606991Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045943.xml
2022-05-18T04:59:49.8964956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:49.8980495Z 
2022-05-18T04:59:49.8980904Z Running tests...
2022-05-18T04:59:49.8981373Z ----------------------------------------------------------------------
2022-05-18T04:59:51.5748564Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:51.6122276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43969
2022-05-18T04:59:51.6232563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43970
2022-05-18T04:59:52.7853935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:52.7854494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:52.7855293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:52.7856005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:52.7962584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:52.8869152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:54.0978525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xgxgqqt
2022-05-18T04:59:54.0979366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xgxgqqt/_remote_module_non_scriptable.py
2022-05-18T04:59:54.1809194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoydlo5s7
2022-05-18T04:59:54.1810422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoydlo5s7/_remote_module_non_scriptable.py
2022-05-18T04:59:54.2062546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:54.2063034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:54.2233180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:54.2233688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:54.5308510Z ok (4.632s)
2022-05-18T04:59:54.5308722Z 
2022-05-18T04:59:54.5309134Z ----------------------------------------------------------------------
2022-05-18T04:59:54.5309473Z Ran 1 test in 4.633s
2022-05-18T04:59:54.5309626Z 
2022-05-18T04:59:54.5309721Z OK
2022-05-18T04:59:54.5309855Z 
2022-05-18T04:59:54.5309986Z Generating XML reports...
2022-05-18T04:59:54.5366132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045949.xml
2022-05-18T04:59:55.9520598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:55.9535861Z 
2022-05-18T04:59:55.9536027Z Running tests...
2022-05-18T04:59:55.9536465Z ----------------------------------------------------------------------
2022-05-18T04:59:57.5622034Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:57.5737852Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.620s)
2022-05-18T04:59:57.5738417Z 
2022-05-18T04:59:57.5738715Z ----------------------------------------------------------------------
2022-05-18T04:59:57.5739027Z Ran 1 test in 1.620s
2022-05-18T04:59:57.5739194Z 
2022-05-18T04:59:57.5739304Z OK (skipped=1)
2022-05-18T04:59:57.5739460Z 
2022-05-18T04:59:57.5739585Z Generating XML reports...
2022-05-18T04:59:57.5776961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045955.xml
2022-05-18T04:59:58.9644136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:58.9660759Z 
2022-05-18T04:59:58.9660932Z Running tests...
2022-05-18T04:59:58.9661386Z ----------------------------------------------------------------------
2022-05-18T05:00:00.6233832Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:00.6607405Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44126
2022-05-18T05:00:00.6719067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44127
2022-05-18T05:00:01.8720331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:01.8720884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:01.8721672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:01.8722383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:01.8829253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:01.9735499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:02.2773485Z ok (3.311s)
2022-05-18T05:00:02.2773716Z 
2022-05-18T05:00:02.2774099Z ----------------------------------------------------------------------
2022-05-18T05:00:02.2774422Z Ran 1 test in 3.311s
2022-05-18T05:00:02.2774591Z 
2022-05-18T05:00:02.2774693Z OK
2022-05-18T05:00:02.2774826Z 
2022-05-18T05:00:02.2777798Z Generating XML reports...
2022-05-18T05:00:02.2831422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045958.xml
2022-05-18T05:00:03.7452127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:03.7468371Z 
2022-05-18T05:00:03.7468856Z Running tests...
2022-05-18T05:00:03.7469345Z ----------------------------------------------------------------------
2022-05-18T05:00:05.4105061Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:05.4226734Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.676s)
2022-05-18T05:00:05.4227392Z 
2022-05-18T05:00:05.4227675Z ----------------------------------------------------------------------
2022-05-18T05:00:05.4228015Z Ran 1 test in 1.676s
2022-05-18T05:00:05.4228160Z 
2022-05-18T05:00:05.4228269Z OK (skipped=1)
2022-05-18T05:00:05.4228421Z 
2022-05-18T05:00:05.4228545Z Generating XML reports...
2022-05-18T05:00:05.4267450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050003.xml
2022-05-18T05:00:06.8276380Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:06.8292574Z 
2022-05-18T05:00:06.8292945Z Running tests...
2022-05-18T05:00:06.8293395Z ----------------------------------------------------------------------
2022-05-18T05:00:08.4869285Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:08.5248751Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44277
2022-05-18T05:00:08.5359019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44278
2022-05-18T05:00:09.7794471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:09.7795029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:09.7796062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:09.7796802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:09.7902808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:09.8809834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:11.4435885Z ok (4.614s)
2022-05-18T05:00:11.4436111Z 
2022-05-18T05:00:11.4436495Z ----------------------------------------------------------------------
2022-05-18T05:00:11.4436810Z Ran 1 test in 4.614s
2022-05-18T05:00:11.4436971Z 
2022-05-18T05:00:11.4437067Z OK
2022-05-18T05:00:11.4437200Z 
2022-05-18T05:00:11.4437331Z Generating XML reports...
2022-05-18T05:00:11.4493837Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050006.xml
2022-05-18T05:00:12.8454867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:12.8470489Z 
2022-05-18T05:00:12.8470809Z Running tests...
2022-05-18T05:00:12.8471547Z ----------------------------------------------------------------------
2022-05-18T05:00:12.8493942Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-05-18T05:00:12.8494566Z 
2022-05-18T05:00:12.8494872Z ----------------------------------------------------------------------
2022-05-18T05:00:12.8495214Z Ran 1 test in 0.002s
2022-05-18T05:00:12.8495380Z 
2022-05-18T05:00:12.8495474Z OK (skipped=1)
2022-05-18T05:00:12.8495627Z 
2022-05-18T05:00:12.8495751Z Generating XML reports...
2022-05-18T05:00:12.8540085Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050012.xml
2022-05-18T05:00:14.1238629Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:14.1254773Z 
2022-05-18T05:00:14.1255100Z Running tests...
2022-05-18T05:00:14.1255548Z ----------------------------------------------------------------------
2022-05-18T05:00:14.1274319Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T05:00:15.7876110Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:15.8241738Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44429
2022-05-18T05:00:15.8350542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44430
2022-05-18T05:00:17.0133142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:17.0133701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:17.0134477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:17.0135176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:17.0143210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:17.0143692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:17.0266442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9d9k01_n
2022-05-18T05:00:17.0267483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6yu62pkh
2022-05-18T05:00:17.0269059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9d9k01_n/_remote_module_non_scriptable.py
2022-05-18T05:00:17.0270151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6yu62pkh/_remote_module_non_scriptable.py
2022-05-18T05:00:17.2399154Z ok (3.114s)
2022-05-18T05:00:17.2399377Z 
2022-05-18T05:00:17.2400031Z ----------------------------------------------------------------------
2022-05-18T05:00:17.2400391Z Ran 1 test in 3.114s
2022-05-18T05:00:17.2400544Z 
2022-05-18T05:00:17.2400649Z OK
2022-05-18T05:00:17.2400782Z 
2022-05-18T05:00:17.2400912Z Generating XML reports...
2022-05-18T05:00:17.2458314Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050014.xml
2022-05-18T05:00:18.6655645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:18.6671168Z 
2022-05-18T05:00:18.6671538Z Running tests...
2022-05-18T05:00:18.6671977Z ----------------------------------------------------------------------
2022-05-18T05:00:18.6696608Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T05:00:20.3112312Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:20.3501734Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44548
2022-05-18T05:00:20.3612856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44549
2022-05-18T05:00:21.5334045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:21.5334609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:21.5335379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:21.5336072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:21.5443913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:21.5547571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6ia2wj6
2022-05-18T05:00:21.5550762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6ia2wj6/_remote_module_non_scriptable.py
2022-05-18T05:00:21.6346634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:21.6444827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxr0c5g_4
2022-05-18T05:00:21.6447891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxr0c5g_4/_remote_module_non_scriptable.py
2022-05-18T05:00:21.6695968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:00:21.6696465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:00:21.8663040Z ok (3.199s)
2022-05-18T05:00:21.8663251Z 
2022-05-18T05:00:21.8663634Z ----------------------------------------------------------------------
2022-05-18T05:00:21.8664032Z Ran 1 test in 3.199s
2022-05-18T05:00:21.8664313Z 
2022-05-18T05:00:21.8664466Z OK
2022-05-18T05:00:21.8664702Z 
2022-05-18T05:00:21.8667030Z Generating XML reports...
2022-05-18T05:00:21.8722096Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050018.xml
2022-05-18T05:00:23.3164306Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:23.3180571Z 
2022-05-18T05:00:23.3180823Z Running tests...
2022-05-18T05:00:23.3181240Z ----------------------------------------------------------------------
2022-05-18T05:00:23.3207368Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T05:00:24.9679379Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:25.0062514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44667
2022-05-18T05:00:25.0170245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44668
2022-05-18T05:00:26.2399259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:26.2399842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:26.2400663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:26.2401390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:26.2408798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:26.2409319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:26.2510172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo11rphid
2022-05-18T05:00:26.2513957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo11rphid/_remote_module_non_scriptable.py
2022-05-18T05:00:26.2517515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_otk3n88
2022-05-18T05:00:26.2520809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_otk3n88/_remote_module_non_scriptable.py
2022-05-18T05:00:26.2774576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:00:26.2775687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:00:26.4218685Z ok (3.104s)
2022-05-18T05:00:26.4219038Z 
2022-05-18T05:00:26.4219796Z ----------------------------------------------------------------------
2022-05-18T05:00:26.4220463Z Ran 1 test in 3.104s
2022-05-18T05:00:26.4220676Z 
2022-05-18T05:00:26.4220775Z OK
2022-05-18T05:00:26.4220909Z 
2022-05-18T05:00:26.4221038Z Generating XML reports...
2022-05-18T05:00:26.4279281Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050023.xml
2022-05-18T05:00:27.8477729Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:27.8493069Z 
2022-05-18T05:00:27.8493959Z Running tests...
2022-05-18T05:00:27.8494844Z ----------------------------------------------------------------------
2022-05-18T05:00:27.8513566Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T05:00:29.5134710Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:29.5508889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44786
2022-05-18T05:00:29.5620844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44787
2022-05-18T05:00:30.7983169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:30.7983710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:30.7984518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:30.7985198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:30.7992497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:30.7992980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:30.8090583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn97e1myd
2022-05-18T05:00:30.8093576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn97e1myd/_remote_module_non_scriptable.py
2022-05-18T05:00:30.8096506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzo06ert
2022-05-18T05:00:30.8099742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzo06ert/_remote_module_non_scriptable.py
2022-05-18T05:00:31.0672553Z ok (3.218s)
2022-05-18T05:00:31.0672787Z 
2022-05-18T05:00:31.0673170Z ----------------------------------------------------------------------
2022-05-18T05:00:31.0673525Z Ran 1 test in 3.218s
2022-05-18T05:00:31.0673673Z 
2022-05-18T05:00:31.0673773Z OK
2022-05-18T05:00:31.0673907Z 
2022-05-18T05:00:31.0674038Z Generating XML reports...
2022-05-18T05:00:31.0730476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050027.xml
2022-05-18T05:00:32.4600685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:32.4616900Z 
2022-05-18T05:00:32.4617052Z Running tests...
2022-05-18T05:00:32.4617490Z ----------------------------------------------------------------------
2022-05-18T05:00:34.1313994Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:34.1688994Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44905
2022-05-18T05:00:34.1799503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44906
2022-05-18T05:00:35.3465321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:35.3466118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:35.3466912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:35.3467591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:35.3475040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:35.3475530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:35.5850731Z ok (3.123s)
2022-05-18T05:00:35.5850938Z 
2022-05-18T05:00:35.5851344Z ----------------------------------------------------------------------
2022-05-18T05:00:35.5851681Z Ran 1 test in 3.123s
2022-05-18T05:00:35.5851845Z 
2022-05-18T05:00:35.5851944Z OK
2022-05-18T05:00:35.5852088Z 
2022-05-18T05:00:35.5852205Z Generating XML reports...
2022-05-18T05:00:35.5908733Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050032.xml
2022-05-18T05:00:36.9896540Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:36.9911296Z 
2022-05-18T05:00:36.9911630Z Running tests...
2022-05-18T05:00:36.9912078Z ----------------------------------------------------------------------
2022-05-18T05:00:38.6349095Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:38.6719084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45024
2022-05-18T05:00:38.6828816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45025
2022-05-18T05:00:39.8713039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:39.8713616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:39.8714412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:39.8715089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:39.8822446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:39.9725388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:40.1879074Z ok (3.196s)
2022-05-18T05:00:40.1879337Z 
2022-05-18T05:00:40.1879982Z ----------------------------------------------------------------------
2022-05-18T05:00:40.1880342Z Ran 1 test in 3.197s
2022-05-18T05:00:40.1880504Z 
2022-05-18T05:00:40.1880603Z OK
2022-05-18T05:00:40.1880743Z 
2022-05-18T05:00:40.1880863Z Generating XML reports...
2022-05-18T05:00:40.1938015Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050036.xml
2022-05-18T05:00:41.6181563Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:41.6197130Z 
2022-05-18T05:00:41.6197537Z Running tests...
2022-05-18T05:00:41.6198012Z ----------------------------------------------------------------------
2022-05-18T05:00:43.2746149Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:43.3120188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45143
2022-05-18T05:00:43.3231545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45144
2022-05-18T05:00:44.5591206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:44.5591753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:44.5592800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:44.5593471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:44.5599961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:44.5601267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:44.5710569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:00:44.5711078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:00:44.5711757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:00:44.5712553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:00:44.8281164Z ok (3.208s)
2022-05-18T05:00:44.8281363Z 
2022-05-18T05:00:44.8281734Z ----------------------------------------------------------------------
2022-05-18T05:00:44.8282068Z Ran 1 test in 3.208s
2022-05-18T05:00:44.8282235Z 
2022-05-18T05:00:44.8282328Z OK
2022-05-18T05:00:44.8282462Z 
2022-05-18T05:00:44.8282581Z Generating XML reports...
2022-05-18T05:00:44.8339126Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050041.xml
2022-05-18T05:00:46.2529263Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:46.2545841Z 
2022-05-18T05:00:46.2546241Z Running tests...
2022-05-18T05:00:46.2546746Z ----------------------------------------------------------------------
2022-05-18T05:00:47.9204181Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:47.9581263Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45268
2022-05-18T05:00:47.9692443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45269
2022-05-18T05:00:49.2118252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:49.2118822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:49.2119625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:49.2120503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:49.2128269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:49.2129185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:49.3741686Z skip: Skipped due to small world size. (3.119s)
2022-05-18T05:00:49.3742009Z 
2022-05-18T05:00:49.3742564Z ----------------------------------------------------------------------
2022-05-18T05:00:49.3742912Z Ran 1 test in 3.120s
2022-05-18T05:00:49.3743081Z 
2022-05-18T05:00:49.3743173Z OK (skipped=1)
2022-05-18T05:00:49.3743330Z 
2022-05-18T05:00:49.3743460Z Generating XML reports...
2022-05-18T05:00:49.3801073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050046.xml
2022-05-18T05:00:50.7964877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:50.7979893Z 
2022-05-18T05:00:50.7980068Z Running tests...
2022-05-18T05:00:50.7980733Z ----------------------------------------------------------------------
2022-05-18T05:00:52.4553418Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:52.4928315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45383
2022-05-18T05:00:52.5039310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45384
2022-05-18T05:00:53.7167937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:53.7168472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:53.7169278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:53.7170327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:53.7176868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:53.7177816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:53.9086269Z ok (3.110s)
2022-05-18T05:00:53.9086498Z 
2022-05-18T05:00:53.9087112Z ----------------------------------------------------------------------
2022-05-18T05:00:53.9087435Z Ran 1 test in 3.111s
2022-05-18T05:00:53.9087599Z 
2022-05-18T05:00:53.9087692Z OK
2022-05-18T05:00:53.9087829Z 
2022-05-18T05:00:53.9088061Z Generating XML reports...
2022-05-18T05:00:53.9143967Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050050.xml
2022-05-18T05:00:55.3457440Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:55.3472799Z 
2022-05-18T05:00:55.3473304Z Running tests...
2022-05-18T05:00:55.3473930Z ----------------------------------------------------------------------
2022-05-18T05:00:57.0134079Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:57.0510042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45502
2022-05-18T05:00:57.0621108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45503
2022-05-18T05:00:58.3142491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:58.3143302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:58.3144267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:58.3145066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:58.3151549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:58.3152314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:58.5731075Z ok (3.225s)
2022-05-18T05:00:58.5731463Z 
2022-05-18T05:00:58.5732123Z ----------------------------------------------------------------------
2022-05-18T05:00:58.5732750Z Ran 1 test in 3.226s
2022-05-18T05:00:58.5733032Z 
2022-05-18T05:00:58.5733200Z OK
2022-05-18T05:00:58.5733442Z 
2022-05-18T05:00:58.5733679Z Generating XML reports...
2022-05-18T05:00:58.5791045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050055.xml
2022-05-18T05:00:59.9989305Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:00.0004328Z 
2022-05-18T05:01:00.0004798Z Running tests...
2022-05-18T05:01:00.0005300Z ----------------------------------------------------------------------
2022-05-18T05:01:01.6662093Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:01.7048432Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45621
2022-05-18T05:01:01.7165457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45622
2022-05-18T05:01:02.9550353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:02.9550936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:02.9551725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:02.9552423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:02.9560217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:02.9560795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:03.1217701Z ok (3.121s)
2022-05-18T05:01:03.1218430Z 
2022-05-18T05:01:03.1218855Z ----------------------------------------------------------------------
2022-05-18T05:01:03.1219216Z Ran 1 test in 3.121s
2022-05-18T05:01:03.1219364Z 
2022-05-18T05:01:03.1219463Z OK
2022-05-18T05:01:03.1219605Z 
2022-05-18T05:01:03.1219746Z Generating XML reports...
2022-05-18T05:01:03.1275650Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050059.xml
2022-05-18T05:01:04.5500453Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:04.5515568Z 
2022-05-18T05:01:04.5515878Z Running tests...
2022-05-18T05:01:04.5516380Z ----------------------------------------------------------------------
2022-05-18T05:01:04.5537436Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-05-18T05:01:04.5537853Z 
2022-05-18T05:01:04.5538140Z ----------------------------------------------------------------------
2022-05-18T05:01:04.5538455Z Ran 1 test in 0.002s
2022-05-18T05:01:04.5538619Z 
2022-05-18T05:01:04.5538727Z OK (skipped=1)
2022-05-18T05:01:04.5538880Z 
2022-05-18T05:01:04.5539005Z Generating XML reports...
2022-05-18T05:01:04.5581370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050104.xml
2022-05-18T05:01:05.8247386Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:05.8263367Z 
2022-05-18T05:01:05.8263608Z Running tests...
2022-05-18T05:01:05.8264050Z ----------------------------------------------------------------------
2022-05-18T05:01:05.8285811Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-05-18T05:01:05.8286424Z 
2022-05-18T05:01:05.8286737Z ----------------------------------------------------------------------
2022-05-18T05:01:05.8287050Z Ran 1 test in 0.002s
2022-05-18T05:01:05.8287212Z 
2022-05-18T05:01:05.8287335Z OK (skipped=1)
2022-05-18T05:01:05.8287495Z 
2022-05-18T05:01:05.8287619Z Generating XML reports...
2022-05-18T05:01:05.8330864Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050105.xml
2022-05-18T05:01:07.1140293Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:07.1155512Z 
2022-05-18T05:01:07.1155948Z Running tests...
2022-05-18T05:01:07.1156432Z ----------------------------------------------------------------------
2022-05-18T05:01:08.7541734Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:08.7919414Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45810
2022-05-18T05:01:08.8028574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45811
2022-05-18T05:01:10.0193202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:10.0194216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:10.0195006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:10.0195678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:10.0201882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:10.0202835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:10.0409215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:01:10.0410154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:01:10.0410934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:10.0411632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:10.3078383Z ok (3.192s)
2022-05-18T05:01:10.3078605Z 
2022-05-18T05:01:10.3079321Z ----------------------------------------------------------------------
2022-05-18T05:01:10.3080001Z Ran 1 test in 3.192s
2022-05-18T05:01:10.3080344Z 
2022-05-18T05:01:10.3080517Z OK
2022-05-18T05:01:10.3080704Z 
2022-05-18T05:01:10.3080838Z Generating XML reports...
2022-05-18T05:01:10.3137509Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050107.xml
2022-05-18T05:01:11.7383788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:11.7398499Z 
2022-05-18T05:01:11.7398693Z Running tests...
2022-05-18T05:01:11.7399139Z ----------------------------------------------------------------------
2022-05-18T05:01:13.3869227Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:13.4236744Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45935
2022-05-18T05:01:13.4346422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45936
2022-05-18T05:01:14.6235824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:14.6236399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:14.6237186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:14.6238125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:14.6245152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:14.6245930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:14.8393158Z skip: Skipped due to small world size. (3.099s)
2022-05-18T05:01:14.8393629Z 
2022-05-18T05:01:14.8394359Z ----------------------------------------------------------------------
2022-05-18T05:01:14.8394837Z Ran 1 test in 3.099s
2022-05-18T05:01:14.8395002Z 
2022-05-18T05:01:14.8395094Z OK (skipped=1)
2022-05-18T05:01:14.8395253Z 
2022-05-18T05:01:14.8395378Z Generating XML reports...
2022-05-18T05:01:14.8452125Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050111.xml
2022-05-18T05:01:16.2539506Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:16.2553656Z 
2022-05-18T05:01:16.2553887Z Running tests...
2022-05-18T05:01:16.2554557Z ----------------------------------------------------------------------
2022-05-18T05:01:16.2576027Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-05-18T05:01:16.2576749Z 
2022-05-18T05:01:16.2577304Z ----------------------------------------------------------------------
2022-05-18T05:01:16.2577649Z Ran 1 test in 0.002s
2022-05-18T05:01:16.2577812Z 
2022-05-18T05:01:16.2577921Z OK (skipped=1)
2022-05-18T05:01:16.2578077Z 
2022-05-18T05:01:16.2578183Z Generating XML reports...
2022-05-18T05:01:16.2618849Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050116.xml
2022-05-18T05:01:17.5246606Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:17.5262397Z 
2022-05-18T05:01:17.5262664Z Running tests...
2022-05-18T05:01:17.5263082Z ----------------------------------------------------------------------
2022-05-18T05:01:17.5285258Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-05-18T05:01:17.5285855Z 
2022-05-18T05:01:17.5286202Z ----------------------------------------------------------------------
2022-05-18T05:01:17.5286533Z Ran 1 test in 0.002s
2022-05-18T05:01:17.5286695Z 
2022-05-18T05:01:17.5286784Z OK (skipped=1)
2022-05-18T05:01:17.5286937Z 
2022-05-18T05:01:17.5287061Z Generating XML reports...
2022-05-18T05:01:17.5328839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050117.xml
2022-05-18T05:01:18.8107774Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:18.8122670Z 
2022-05-18T05:01:18.8122801Z Running tests...
2022-05-18T05:01:18.8123263Z ----------------------------------------------------------------------
2022-05-18T05:01:20.4847293Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:20.5224707Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46120
2022-05-18T05:01:20.5338428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46121
2022-05-18T05:01:21.7448254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:21.7448809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:21.7449868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:21.7450577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:21.7557102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:21.8460302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:22.0394498Z ok (3.227s)
2022-05-18T05:01:22.0394720Z 
2022-05-18T05:01:22.0395093Z ----------------------------------------------------------------------
2022-05-18T05:01:22.0395663Z Ran 1 test in 3.227s
2022-05-18T05:01:22.0395829Z 
2022-05-18T05:01:22.0395930Z OK
2022-05-18T05:01:22.0396069Z 
2022-05-18T05:01:22.0396205Z Generating XML reports...
2022-05-18T05:01:22.0454548Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050118.xml
2022-05-18T05:01:23.4852094Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:23.4867716Z 
2022-05-18T05:01:23.4868152Z Running tests...
2022-05-18T05:01:23.4868662Z ----------------------------------------------------------------------
2022-05-18T05:01:25.1348588Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:25.1725810Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46235
2022-05-18T05:01:25.1837406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46236
2022-05-18T05:01:26.3800499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:26.3801048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:26.3801826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:26.3802517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:26.3809856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:26.3810749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:26.4125025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:01:26.4226247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:01:26.4227028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:26.4227712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:26.4472255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:01:26.4472776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:01:26.4473491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:01:26.4474184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:01:26.4697083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T05:01:26.4697609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T05:01:26.4698299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T05:01:26.4698986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T05:01:26.6886085Z ok (3.201s)
2022-05-18T05:01:26.6886310Z 
2022-05-18T05:01:26.6886695Z ----------------------------------------------------------------------
2022-05-18T05:01:26.6887332Z Ran 1 test in 3.202s
2022-05-18T05:01:26.6887517Z 
2022-05-18T05:01:26.6887614Z OK
2022-05-18T05:01:26.6887733Z 
2022-05-18T05:01:26.6887868Z Generating XML reports...
2022-05-18T05:01:26.6944459Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050123.xml
2022-05-18T05:01:28.1136281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:28.1151591Z 
2022-05-18T05:01:28.1152212Z Running tests...
2022-05-18T05:01:28.1152705Z ----------------------------------------------------------------------
2022-05-18T05:01:29.7779381Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:29.8153773Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46374
2022-05-18T05:01:29.8264766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46375
2022-05-18T05:01:31.0278748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:31.0279281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:31.0280332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:31.0281031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:31.0287671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:31.0288168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:31.0496914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:01:31.0497437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:01:31.0498149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:31.0499052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:31.2312022Z ok (3.116s)
2022-05-18T05:01:31.2312729Z 
2022-05-18T05:01:31.2313637Z ----------------------------------------------------------------------
2022-05-18T05:01:31.2314180Z Ran 1 test in 3.116s
2022-05-18T05:01:31.2314360Z 
2022-05-18T05:01:31.2314452Z OK
2022-05-18T05:01:31.2314587Z 
2022-05-18T05:01:31.2314719Z Generating XML reports...
2022-05-18T05:01:31.2371021Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050128.xml
2022-05-18T05:01:32.6529988Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:32.6544884Z 
2022-05-18T05:01:32.6545267Z Running tests...
2022-05-18T05:01:32.6546215Z ----------------------------------------------------------------------
2022-05-18T05:01:34.2929400Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:34.3299956Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46499
2022-05-18T05:01:34.3409482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46500
2022-05-18T05:01:35.5058432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:35.5058992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:35.5059777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:35.5060700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:35.5067688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:35.5068491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:35.5175721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:01:35.5176236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:01:35.5176938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:35.5177630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:35.7460027Z ok (3.091s)
2022-05-18T05:01:35.7460233Z 
2022-05-18T05:01:35.7460618Z ----------------------------------------------------------------------
2022-05-18T05:01:35.7460958Z Ran 1 test in 3.091s
2022-05-18T05:01:35.7461122Z 
2022-05-18T05:01:35.7461219Z OK
2022-05-18T05:01:35.7461354Z 
2022-05-18T05:01:35.7461486Z Generating XML reports...
2022-05-18T05:01:35.7518729Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050132.xml
2022-05-18T05:01:37.1499268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:37.1513915Z 
2022-05-18T05:01:37.1514224Z Running tests...
2022-05-18T05:01:37.1514918Z ----------------------------------------------------------------------
2022-05-18T05:01:38.7732565Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:38.8098797Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46624
2022-05-18T05:01:38.8211890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46625
2022-05-18T05:01:39.9927394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:39.9928222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:39.9929140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:39.9930192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:39.9936077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:39.9936659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:40.0044489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:01:40.0044985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:01:40.0045677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:40.0046373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:40.2264526Z ok (3.075s)
2022-05-18T05:01:40.2264744Z 
2022-05-18T05:01:40.2265346Z ----------------------------------------------------------------------
2022-05-18T05:01:40.2265714Z Ran 1 test in 3.075s
2022-05-18T05:01:40.2265883Z 
2022-05-18T05:01:40.2265984Z OK
2022-05-18T05:01:40.2266117Z 
2022-05-18T05:01:40.2266283Z Generating XML reports...
2022-05-18T05:01:40.2322277Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050137.xml
2022-05-18T05:01:41.6469132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:41.6484755Z 
2022-05-18T05:01:41.6484912Z Running tests...
2022-05-18T05:01:41.6485858Z ----------------------------------------------------------------------
2022-05-18T05:01:43.2956428Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:43.3334240Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46749
2022-05-18T05:01:43.3443459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46750
2022-05-18T05:01:44.5028407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:44.5028968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:44.5029752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:44.5030435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:44.5137295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:44.6040028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:44.6154583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:01:44.6155403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:01:44.6156137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:44.6157083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:44.8495630Z ok (3.201s)
2022-05-18T05:01:44.8495851Z 
2022-05-18T05:01:44.8496407Z ----------------------------------------------------------------------
2022-05-18T05:01:44.8496845Z Ran 1 test in 3.201s
2022-05-18T05:01:44.8497011Z 
2022-05-18T05:01:44.8497118Z OK
2022-05-18T05:01:44.8497254Z 
2022-05-18T05:01:44.8497391Z Generating XML reports...
2022-05-18T05:01:44.8553159Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050141.xml
2022-05-18T05:01:46.2720535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:46.2736508Z 
2022-05-18T05:01:46.2736812Z Running tests...
2022-05-18T05:01:46.2737250Z ----------------------------------------------------------------------
2022-05-18T05:01:47.9123163Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:47.9491834Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46874
2022-05-18T05:01:47.9599583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46875
2022-05-18T05:01:49.1849103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:49.1849661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:49.1850702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:49.1851372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:49.1959813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:49.2864410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:49.4648616Z skip: Skipped due to small world size. (3.191s)
2022-05-18T05:01:49.4648869Z 
2022-05-18T05:01:49.4649255Z ----------------------------------------------------------------------
2022-05-18T05:01:49.4649841Z Ran 1 test in 3.191s
2022-05-18T05:01:49.4650011Z 
2022-05-18T05:01:49.4650345Z OK (skipped=1)
2022-05-18T05:01:49.4650525Z 
2022-05-18T05:01:49.4650634Z Generating XML reports...
2022-05-18T05:01:49.4707358Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050146.xml
2022-05-18T05:01:50.9015138Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:50.9030827Z 
2022-05-18T05:01:50.9031240Z Running tests...
2022-05-18T05:01:50.9031726Z ----------------------------------------------------------------------
2022-05-18T05:01:52.5562668Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:52.5927219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46989
2022-05-18T05:01:52.6037841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46990
2022-05-18T05:01:53.8347918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:53.8348484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:53.8349265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:53.8350204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:53.8457025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:53.9362569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:54.1087449Z skip: Skipped due to small world size. (3.205s)
2022-05-18T05:01:54.1087717Z 
2022-05-18T05:01:54.1088094Z ----------------------------------------------------------------------
2022-05-18T05:01:54.1088438Z Ran 1 test in 3.206s
2022-05-18T05:01:54.1088603Z 
2022-05-18T05:01:54.1088712Z OK (skipped=1)
2022-05-18T05:01:54.1088880Z 
2022-05-18T05:01:54.1089007Z Generating XML reports...
2022-05-18T05:01:54.1148324Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050150.xml
2022-05-18T05:01:55.5268835Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:55.5283433Z 
2022-05-18T05:01:55.5283970Z Running tests...
2022-05-18T05:01:55.5284477Z ----------------------------------------------------------------------
2022-05-18T05:01:57.1874289Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:57.2240936Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47104
2022-05-18T05:01:57.2350440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47105
2022-05-18T05:01:58.4076798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:58.4077611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:58.4078375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:58.4079083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:58.4187328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:58.5091698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:58.6397096Z skip: Skipped due to small world size. (3.111s)
2022-05-18T05:01:58.6397491Z 
2022-05-18T05:01:58.6397927Z ----------------------------------------------------------------------
2022-05-18T05:01:58.6398299Z Ran 1 test in 3.111s
2022-05-18T05:01:58.6398445Z 
2022-05-18T05:01:58.6398808Z OK (skipped=1)
2022-05-18T05:01:58.6398980Z 
2022-05-18T05:01:58.6399105Z Generating XML reports...
2022-05-18T05:01:58.6456688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050155.xml
2022-05-18T05:02:00.0623079Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:00.0637594Z 
2022-05-18T05:02:00.0638004Z Running tests...
2022-05-18T05:02:00.0638485Z ----------------------------------------------------------------------
2022-05-18T05:02:01.6994740Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:01.7370101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47219
2022-05-18T05:02:01.7480286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47220
2022-05-18T05:02:02.9209086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:02.9209917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:02.9210683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:02.9211624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:02.9218762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:02.9219251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:03.1527733Z skip: Skipped due to small world size. (3.089s)
2022-05-18T05:02:03.1528192Z 
2022-05-18T05:02:03.1528603Z ----------------------------------------------------------------------
2022-05-18T05:02:03.1528944Z Ran 1 test in 3.089s
2022-05-18T05:02:03.1529108Z 
2022-05-18T05:02:03.1529221Z OK (skipped=1)
2022-05-18T05:02:03.1529376Z 
2022-05-18T05:02:03.1529732Z Generating XML reports...
2022-05-18T05:02:03.1587757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050200.xml
2022-05-18T05:02:04.5801811Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:04.5817604Z 
2022-05-18T05:02:04.5817858Z Running tests...
2022-05-18T05:02:04.5818297Z ----------------------------------------------------------------------
2022-05-18T05:02:06.2267757Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:06.2644303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47334
2022-05-18T05:02:06.2755857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47335
2022-05-18T05:02:07.4860174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:07.4860734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:07.4861520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:07.4862226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:07.4868652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:07.4869528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:07.6806251Z ok (3.098s)
2022-05-18T05:02:07.6806664Z 
2022-05-18T05:02:07.6807402Z ----------------------------------------------------------------------
2022-05-18T05:02:07.6808112Z Ran 1 test in 3.099s
2022-05-18T05:02:07.6808463Z 
2022-05-18T05:02:07.6808622Z OK
2022-05-18T05:02:07.6808790Z 
2022-05-18T05:02:07.6809212Z Generating XML reports...
2022-05-18T05:02:07.6865822Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050204.xml
2022-05-18T05:02:09.1083329Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:09.1105459Z 
2022-05-18T05:02:09.1106029Z Running tests...
2022-05-18T05:02:09.1106517Z ----------------------------------------------------------------------
2022-05-18T05:02:10.7817328Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:10.8194575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47453
2022-05-18T05:02:10.8307875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47454
2022-05-18T05:02:11.9760851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:11.9761434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:11.9762233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:11.9763148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:11.9770197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:11.9770687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:12.1355642Z ok (3.025s)
2022-05-18T05:02:12.1355861Z 
2022-05-18T05:02:12.1356245Z ----------------------------------------------------------------------
2022-05-18T05:02:12.1356578Z Ran 1 test in 3.025s
2022-05-18T05:02:12.1356743Z 
2022-05-18T05:02:12.1356840Z OK
2022-05-18T05:02:12.1356977Z 
2022-05-18T05:02:12.1357117Z Generating XML reports...
2022-05-18T05:02:12.1414895Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050209.xml
2022-05-18T05:02:13.5622447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:13.5636918Z 
2022-05-18T05:02:13.5637324Z Running tests...
2022-05-18T05:02:13.5637849Z ----------------------------------------------------------------------
2022-05-18T05:02:15.2050796Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:15.2422327Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47568
2022-05-18T05:02:15.2532797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47569
2022-05-18T05:02:16.4361108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:16.4361727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:16.4362514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:16.4363210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:16.4472684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:16.5372437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:16.7584456Z ok (3.194s)
2022-05-18T05:02:16.7584858Z 
2022-05-18T05:02:16.7585529Z ----------------------------------------------------------------------
2022-05-18T05:02:16.7586140Z Ran 1 test in 3.195s
2022-05-18T05:02:16.7586450Z 
2022-05-18T05:02:16.7586624Z OK
2022-05-18T05:02:16.7586862Z 
2022-05-18T05:02:16.7587096Z Generating XML reports...
2022-05-18T05:02:16.7645262Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050213.xml
2022-05-18T05:02:18.2051334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:18.2067030Z 
2022-05-18T05:02:18.2067486Z Running tests...
2022-05-18T05:02:18.2067970Z ----------------------------------------------------------------------
2022-05-18T05:02:19.8736952Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:19.9110627Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47687
2022-05-18T05:02:19.9222094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47688
2022-05-18T05:02:21.1214629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:21.1215210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:21.1216012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:21.1216708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:21.1324416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:21.2228383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:21.4271692Z ok (3.220s)
2022-05-18T05:02:21.4271881Z 
2022-05-18T05:02:21.4272440Z ----------------------------------------------------------------------
2022-05-18T05:02:21.4272756Z Ran 1 test in 3.220s
2022-05-18T05:02:21.4272918Z 
2022-05-18T05:02:21.4273015Z OK
2022-05-18T05:02:21.4273217Z 
2022-05-18T05:02:21.4273350Z Generating XML reports...
2022-05-18T05:02:21.4329385Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050218.xml
2022-05-18T05:02:22.8313335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:22.8327326Z 
2022-05-18T05:02:22.8327589Z Running tests...
2022-05-18T05:02:22.8328036Z ----------------------------------------------------------------------
2022-05-18T05:02:24.4526291Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:24.4893710Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47806
2022-05-18T05:02:24.5007784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47807
2022-05-18T05:02:25.7243370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:25.7243939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:25.7244753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:25.7245434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:25.7352777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:25.8255146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:26.0057302Z ok (3.173s)
2022-05-18T05:02:26.0057526Z 
2022-05-18T05:02:26.0057915Z ----------------------------------------------------------------------
2022-05-18T05:02:26.0058254Z Ran 1 test in 3.173s
2022-05-18T05:02:26.0058424Z 
2022-05-18T05:02:26.0058520Z OK
2022-05-18T05:02:26.0058639Z 
2022-05-18T05:02:26.0058773Z Generating XML reports...
2022-05-18T05:02:26.0115049Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050222.xml
2022-05-18T05:02:27.4104749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:27.4119583Z 
2022-05-18T05:02:27.4119916Z Running tests...
2022-05-18T05:02:27.4120414Z ----------------------------------------------------------------------
2022-05-18T05:02:29.0703297Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:29.1079886Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47925
2022-05-18T05:02:29.1190228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47926
2022-05-18T05:02:30.3298984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:30.3299542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:30.3300309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:30.3301026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:30.3307713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:30.3308428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:30.5238814Z ok (3.112s)
2022-05-18T05:02:30.5239036Z 
2022-05-18T05:02:30.5239419Z ----------------------------------------------------------------------
2022-05-18T05:02:30.5239759Z Ran 1 test in 3.112s
2022-05-18T05:02:30.5239922Z 
2022-05-18T05:02:30.5240003Z OK
2022-05-18T05:02:30.5240143Z 
2022-05-18T05:02:30.5240278Z Generating XML reports...
2022-05-18T05:02:30.5297447Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050227.xml
2022-05-18T05:02:31.9522797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:31.9538664Z 
2022-05-18T05:02:31.9538924Z Running tests...
2022-05-18T05:02:31.9539362Z ----------------------------------------------------------------------
2022-05-18T05:02:33.6240827Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:33.6617212Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48040
2022-05-18T05:02:33.6730331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48041
2022-05-18T05:02:34.8744051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:34.8744618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:34.8745408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:34.8746101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:34.8753740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:34.8754232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:34.8863352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:02:34.8863859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:02:34.8864515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:02:34.8865300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:02:35.0781248Z ok (3.124s)
2022-05-18T05:02:35.0781549Z 
2022-05-18T05:02:35.0782412Z ----------------------------------------------------------------------
2022-05-18T05:02:35.0782826Z Ran 1 test in 3.124s
2022-05-18T05:02:35.0782973Z 
2022-05-18T05:02:35.0783067Z OK
2022-05-18T05:02:35.0783203Z 
2022-05-18T05:02:35.0783344Z Generating XML reports...
2022-05-18T05:02:35.0840120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050231.xml
2022-05-18T05:02:36.5035093Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:36.5049955Z 
2022-05-18T05:02:36.5050166Z Running tests...
2022-05-18T05:02:36.5050601Z ----------------------------------------------------------------------
2022-05-18T05:02:38.1762873Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:38.2139167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48165
2022-05-18T05:02:38.2251396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48166
2022-05-18T05:02:39.3436595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:39.3437203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:39.3438267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:39.3438962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:39.3445497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:39.3445985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:39.3653574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:02:39.3654089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:02:39.3654759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:02:39.3655612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:02:39.6300210Z ok (3.125s)
2022-05-18T05:02:39.6300590Z 
2022-05-18T05:02:39.6301323Z ----------------------------------------------------------------------
2022-05-18T05:02:39.6301837Z Ran 1 test in 3.125s
2022-05-18T05:02:39.6302004Z 
2022-05-18T05:02:39.6302102Z OK
2022-05-18T05:02:39.6302245Z 
2022-05-18T05:02:39.6302764Z Generating XML reports...
2022-05-18T05:02:39.6358599Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050236.xml
2022-05-18T05:02:41.0520320Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:41.0535831Z 
2022-05-18T05:02:41.0536014Z Running tests...
2022-05-18T05:02:41.0536468Z ----------------------------------------------------------------------
2022-05-18T05:02:42.7101029Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:42.7474788Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48290
2022-05-18T05:02:42.7586314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48291
2022-05-18T05:02:43.9358435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:43.9358984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:43.9359756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:43.9360679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:43.9467167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:44.0370608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:44.0482055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:02:44.0482574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:02:44.0483263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:02:44.0483954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:02:44.2637927Z ok (3.210s)
2022-05-18T05:02:44.2638322Z 
2022-05-18T05:02:44.2639040Z ----------------------------------------------------------------------
2022-05-18T05:02:44.2639759Z Ran 1 test in 3.210s
2022-05-18T05:02:44.2639947Z 
2022-05-18T05:02:44.2640043Z OK
2022-05-18T05:02:44.2640162Z 
2022-05-18T05:02:44.2640301Z Generating XML reports...
2022-05-18T05:02:44.2696091Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050241.xml
2022-05-18T05:02:45.6918654Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:45.6933224Z 
2022-05-18T05:02:45.6933557Z Running tests...
2022-05-18T05:02:45.6934496Z ----------------------------------------------------------------------
2022-05-18T05:02:47.3272432Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:47.3642550Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48415
2022-05-18T05:02:47.3751202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48416
2022-05-18T05:02:48.5425766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:48.5426325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:48.5427129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:48.5427808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:48.5534945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:48.6436978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:48.6646727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:02:48.6647218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:02:48.6647926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:02:48.6648970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:02:48.8800021Z ok (3.186s)
2022-05-18T05:02:48.8800217Z 
2022-05-18T05:02:48.8800623Z ----------------------------------------------------------------------
2022-05-18T05:02:48.8800971Z Ran 1 test in 3.187s
2022-05-18T05:02:48.8801140Z 
2022-05-18T05:02:48.8801234Z OK
2022-05-18T05:02:48.8801369Z 
2022-05-18T05:02:48.8801483Z Generating XML reports...
2022-05-18T05:02:48.8857987Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050245.xml
2022-05-18T05:02:50.2836327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:50.2850658Z 
2022-05-18T05:02:50.2850900Z Running tests...
2022-05-18T05:02:50.2851648Z ----------------------------------------------------------------------
2022-05-18T05:02:51.8974459Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:51.9341900Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48540
2022-05-18T05:02:51.9451778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48541
2022-05-18T05:02:53.1552401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:53.1553006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:53.1553811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:53.1554508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:53.1561267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:53.1561934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:53.3501663Z skip: Skipped due to small world size. (3.065s)
2022-05-18T05:02:53.3502477Z 
2022-05-18T05:02:53.3503108Z ----------------------------------------------------------------------
2022-05-18T05:02:53.3503454Z Ran 1 test in 3.065s
2022-05-18T05:02:53.3503598Z 
2022-05-18T05:02:53.3503707Z OK (skipped=1)
2022-05-18T05:02:53.3503859Z 
2022-05-18T05:02:53.3503985Z Generating XML reports...
2022-05-18T05:02:53.3560291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050250.xml
2022-05-18T05:02:54.7464513Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:54.7478645Z 
2022-05-18T05:02:54.7479146Z Running tests...
2022-05-18T05:02:54.7479650Z ----------------------------------------------------------------------
2022-05-18T05:02:56.3524962Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:56.3890610Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48655
2022-05-18T05:02:56.3999366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48656
2022-05-18T05:02:57.5780307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:57.5780916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:57.5781711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:57.5782407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:57.5888902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:57.6795512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:57.9048201Z skip: Skipped due to small world size. (3.157s)
2022-05-18T05:02:57.9048433Z 
2022-05-18T05:02:57.9048837Z ----------------------------------------------------------------------
2022-05-18T05:02:57.9049174Z Ran 1 test in 3.157s
2022-05-18T05:02:57.9049337Z 
2022-05-18T05:02:57.9049446Z OK (skipped=1)
2022-05-18T05:02:57.9049833Z 
2022-05-18T05:02:57.9049944Z Generating XML reports...
2022-05-18T05:02:57.9107096Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050254.xml
2022-05-18T05:02:59.3349281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:59.3364290Z 
2022-05-18T05:02:59.3364612Z Running tests...
2022-05-18T05:02:59.3365285Z ----------------------------------------------------------------------
2022-05-18T05:03:00.9474251Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:00.9846961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48770
2022-05-18T05:03:00.9956177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48771
2022-05-18T05:03:02.2347348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:02.2348148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:02.2348967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:02.2349916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:02.2456572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:02.3362109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:02.5006829Z skip: Skipped due to small world size. (3.164s)
2022-05-18T05:03:02.5007276Z 
2022-05-18T05:03:02.5007711Z ----------------------------------------------------------------------
2022-05-18T05:03:02.5008050Z Ran 1 test in 3.164s
2022-05-18T05:03:02.5008211Z 
2022-05-18T05:03:02.5008336Z OK (skipped=1)
2022-05-18T05:03:02.5008627Z 
2022-05-18T05:03:02.5008826Z Generating XML reports...
2022-05-18T05:03:02.5067045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050259.xml
2022-05-18T05:03:03.9273430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:03.9288706Z 
2022-05-18T05:03:03.9289059Z Running tests...
2022-05-18T05:03:03.9289761Z ----------------------------------------------------------------------
2022-05-18T05:03:05.5708353Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:05.6077899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48885
2022-05-18T05:03:05.6187829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48886
2022-05-18T05:03:06.8258471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:06.8259012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:06.8259805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:06.8260500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:06.8268235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:06.8268723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:07.0237438Z skip: Skipped due to small world size. (3.095s)
2022-05-18T05:03:07.0237717Z 
2022-05-18T05:03:07.0238092Z ----------------------------------------------------------------------
2022-05-18T05:03:07.0238425Z Ran 1 test in 3.095s
2022-05-18T05:03:07.0238589Z 
2022-05-18T05:03:07.0238707Z OK (skipped=1)
2022-05-18T05:03:07.0238845Z 
2022-05-18T05:03:07.0238971Z Generating XML reports...
2022-05-18T05:03:07.0295380Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050303.xml
2022-05-18T05:03:08.4532459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:08.4547788Z 
2022-05-18T05:03:08.4548301Z Running tests...
2022-05-18T05:03:08.4548802Z ----------------------------------------------------------------------
2022-05-18T05:03:10.1152039Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:10.1527873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49000
2022-05-18T05:03:10.1639686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49001
2022-05-18T05:03:11.3850669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:11.3851256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:11.3852034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:11.3852732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:11.3859725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:11.3860212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:11.5688742Z ok (3.114s)
2022-05-18T05:03:11.5688969Z 
2022-05-18T05:03:11.5690048Z ----------------------------------------------------------------------
2022-05-18T05:03:11.5690391Z Ran 1 test in 3.114s
2022-05-18T05:03:11.5690554Z 
2022-05-18T05:03:11.5690627Z OK
2022-05-18T05:03:11.5690773Z 
2022-05-18T05:03:11.5690910Z Generating XML reports...
2022-05-18T05:03:11.5747815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050308.xml
2022-05-18T05:03:13.0058517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:13.0073496Z 
2022-05-18T05:03:13.0073643Z Running tests...
2022-05-18T05:03:13.0074856Z ----------------------------------------------------------------------
2022-05-18T05:03:14.6592368Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:14.6961001Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49119
2022-05-18T05:03:14.7071186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49120
2022-05-18T05:03:15.9054406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:15.9054952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:15.9055728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:15.9056418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:15.9062885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:15.9063708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:16.1119156Z ok (3.104s)
2022-05-18T05:03:16.1119371Z 
2022-05-18T05:03:16.1119765Z ----------------------------------------------------------------------
2022-05-18T05:03:16.1120130Z Ran 1 test in 3.105s
2022-05-18T05:03:16.1120277Z 
2022-05-18T05:03:16.1120377Z OK
2022-05-18T05:03:16.1120517Z 
2022-05-18T05:03:16.1120650Z Generating XML reports...
2022-05-18T05:03:16.1177460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050313.xml
2022-05-18T05:03:17.5356856Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:17.5371623Z 
2022-05-18T05:03:17.5371894Z Running tests...
2022-05-18T05:03:17.5372340Z ----------------------------------------------------------------------
2022-05-18T05:03:19.1946735Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:19.2322236Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49238
2022-05-18T05:03:19.2433030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49239
2022-05-18T05:03:20.4554316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:20.4554869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:20.4555655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:20.4556352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:20.4563432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:20.4564238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:22.4512201Z ok (4.914s)
2022-05-18T05:03:22.4513547Z 
2022-05-18T05:03:22.4514034Z ----------------------------------------------------------------------
2022-05-18T05:03:22.4514379Z Ran 1 test in 4.914s
2022-05-18T05:03:22.4514813Z 
2022-05-18T05:03:22.4514907Z OK
2022-05-18T05:03:22.4515046Z 
2022-05-18T05:03:22.4515168Z Generating XML reports...
2022-05-18T05:03:22.4569847Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050317.xml
2022-05-18T05:03:23.8725671Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:23.8740858Z 
2022-05-18T05:03:23.8741324Z Running tests...
2022-05-18T05:03:23.8741814Z ----------------------------------------------------------------------
2022-05-18T05:03:25.4783134Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:25.5149012Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49359
2022-05-18T05:03:25.5258316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49360
2022-05-18T05:03:26.7114409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:26.7115005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:26.7115815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:26.7116493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:26.7123023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:26.7123512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:28.7337052Z ok (4.859s)
2022-05-18T05:03:28.7337278Z 
2022-05-18T05:03:28.7337694Z ----------------------------------------------------------------------
2022-05-18T05:03:28.7338015Z Ran 1 test in 4.860s
2022-05-18T05:03:28.7338184Z 
2022-05-18T05:03:28.7338287Z OK
2022-05-18T05:03:28.7338438Z 
2022-05-18T05:03:28.7338572Z Generating XML reports...
2022-05-18T05:03:28.7394541Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050323.xml
2022-05-18T05:03:30.1753940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:30.1768820Z 
2022-05-18T05:03:30.1769246Z Running tests...
2022-05-18T05:03:30.1770001Z ----------------------------------------------------------------------
2022-05-18T05:03:31.8229034Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:31.8599085Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49480
2022-05-18T05:03:31.8709182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49481
2022-05-18T05:03:32.9937927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:32.9938523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:32.9939313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:32.9939988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:33.0048151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:33.0949234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:33.2762213Z ok (3.099s)
2022-05-18T05:03:33.2762440Z 
2022-05-18T05:03:33.2762809Z ----------------------------------------------------------------------
2022-05-18T05:03:33.2763149Z Ran 1 test in 3.099s
2022-05-18T05:03:33.2763315Z 
2022-05-18T05:03:33.2763412Z OK
2022-05-18T05:03:33.2763549Z 
2022-05-18T05:03:33.2763683Z Generating XML reports...
2022-05-18T05:03:33.2823379Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050330.xml
2022-05-18T05:03:34.7092113Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:34.7107123Z 
2022-05-18T05:03:34.7107548Z Running tests...
2022-05-18T05:03:34.7108021Z ----------------------------------------------------------------------
2022-05-18T05:03:36.3646936Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:36.4017039Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49599
2022-05-18T05:03:36.4128057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49600
2022-05-18T05:03:37.6545996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:37.6546564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:37.6547340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:37.6548040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:37.6657053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:37.7561137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:39.4205349Z ok (4.709s)
2022-05-18T05:03:39.4205565Z 
2022-05-18T05:03:39.4205936Z ----------------------------------------------------------------------
2022-05-18T05:03:39.4206255Z Ran 1 test in 4.710s
2022-05-18T05:03:39.4206442Z 
2022-05-18T05:03:39.4206537Z OK
2022-05-18T05:03:39.4206679Z 
2022-05-18T05:03:39.4206818Z Generating XML reports...
2022-05-18T05:03:39.4264031Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050334.xml
2022-05-18T05:03:40.8350327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:40.8365504Z 
2022-05-18T05:03:40.8366016Z Running tests...
2022-05-18T05:03:40.8366635Z ----------------------------------------------------------------------
2022-05-18T05:03:42.4906766Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:42.5281536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49716
2022-05-18T05:03:42.5392309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49717
2022-05-18T05:03:43.7290983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:43.7291561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:43.7292358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:43.7293069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:43.7399513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:43.8302673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:44.0442470Z ok (3.207s)
2022-05-18T05:03:44.0442691Z 
2022-05-18T05:03:44.0443067Z ----------------------------------------------------------------------
2022-05-18T05:03:44.0443412Z Ran 1 test in 3.208s
2022-05-18T05:03:44.0446223Z 
2022-05-18T05:03:44.0446443Z OK
2022-05-18T05:03:44.0446622Z 
2022-05-18T05:03:44.0447090Z Generating XML reports...
2022-05-18T05:03:44.0502449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050340.xml
2022-05-18T05:03:45.4678394Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:45.4693418Z 
2022-05-18T05:03:45.4693888Z Running tests...
2022-05-18T05:03:45.4694381Z ----------------------------------------------------------------------
2022-05-18T05:03:47.1088246Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:47.1458245Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49835
2022-05-18T05:03:47.1568327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49836
2022-05-18T05:03:48.3320861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:48.3321448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:48.3322221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:48.3322928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:48.3429692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:48.4334817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:48.6620126Z ok (3.192s)
2022-05-18T05:03:48.6620385Z 
2022-05-18T05:03:48.6620798Z ----------------------------------------------------------------------
2022-05-18T05:03:48.6621142Z Ran 1 test in 3.193s
2022-05-18T05:03:48.6621306Z 
2022-05-18T05:03:48.6621401Z OK
2022-05-18T05:03:48.6621517Z 
2022-05-18T05:03:48.6621655Z Generating XML reports...
2022-05-18T05:03:48.6678145Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050345.xml
2022-05-18T05:03:50.0979869Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:50.0994949Z 
2022-05-18T05:03:50.0995250Z Running tests...
2022-05-18T05:03:50.0995703Z ----------------------------------------------------------------------
2022-05-18T05:03:51.7531604Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:51.7899544Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49954
2022-05-18T05:03:51.8009358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49955
2022-05-18T05:03:52.9921868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:52.9922642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:52.9923449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:52.9924125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:52.9931084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:52.9931686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:53.2059324Z ok (3.106s)
2022-05-18T05:03:53.2059550Z 
2022-05-18T05:03:53.2059930Z ----------------------------------------------------------------------
2022-05-18T05:03:53.2060271Z Ran 1 test in 3.106s
2022-05-18T05:03:53.2060437Z 
2022-05-18T05:03:53.2060534Z OK
2022-05-18T05:03:53.2060669Z 
2022-05-18T05:03:53.2060805Z Generating XML reports...
2022-05-18T05:03:53.2117425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050350.xml
2022-05-18T05:03:54.6290048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:54.6305493Z 
2022-05-18T05:03:54.6306090Z Running tests...
2022-05-18T05:03:54.6306639Z ----------------------------------------------------------------------
2022-05-18T05:03:56.2711345Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:56.3087999Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50073
2022-05-18T05:03:56.3199191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50074
2022-05-18T05:03:57.5100779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:57.5101315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:57.5102130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:57.5102830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:57.5212361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:57.6116124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:01.6328723Z ok (7.002s)
2022-05-18T05:04:01.6329026Z 
2022-05-18T05:04:01.6329869Z ----------------------------------------------------------------------
2022-05-18T05:04:01.6330222Z Ran 1 test in 7.002s
2022-05-18T05:04:01.6330440Z 
2022-05-18T05:04:01.6330612Z OK
2022-05-18T05:04:01.6330863Z 
2022-05-18T05:04:01.6331031Z Generating XML reports...
2022-05-18T05:04:01.6386507Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050354.xml
2022-05-18T05:04:03.1019120Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:03.1033907Z 
2022-05-18T05:04:03.1034161Z Running tests...
2022-05-18T05:04:03.1034613Z ----------------------------------------------------------------------
2022-05-18T05:04:04.7725878Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:04.8102628Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50196
2022-05-18T05:04:04.8214292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50197
2022-05-18T05:04:05.9961388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:05.9962206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:05.9963465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:05.9964284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:06.0069696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:06.0976497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:10.0346104Z ok (6.931s)
2022-05-18T05:04:10.0346364Z 
2022-05-18T05:04:10.0346972Z ----------------------------------------------------------------------
2022-05-18T05:04:10.0347300Z Ran 1 test in 6.931s
2022-05-18T05:04:10.0347467Z 
2022-05-18T05:04:10.0348558Z OK
2022-05-18T05:04:10.0348978Z 
2022-05-18T05:04:10.0349358Z Generating XML reports...
2022-05-18T05:04:10.0403404Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050403.xml
2022-05-18T05:04:11.4812505Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:11.4827889Z 
2022-05-18T05:04:11.4828198Z Running tests...
2022-05-18T05:04:11.4828616Z ----------------------------------------------------------------------
2022-05-18T05:04:13.1397410Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:13.1775144Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50319
2022-05-18T05:04:13.1886345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50320
2022-05-18T05:04:14.3593026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:14.3593575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:14.3594346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:14.3595106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:14.3602272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:14.3602773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:18.4005944Z ok (6.917s)
2022-05-18T05:04:18.4006187Z 
2022-05-18T05:04:18.4006779Z ----------------------------------------------------------------------
2022-05-18T05:04:18.4007135Z Ran 1 test in 6.918s
2022-05-18T05:04:18.4007308Z 
2022-05-18T05:04:18.4007403Z OK
2022-05-18T05:04:18.4007564Z 
2022-05-18T05:04:18.4007782Z Generating XML reports...
2022-05-18T05:04:18.4064893Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050411.xml
2022-05-18T05:04:19.8450584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:19.8466183Z 
2022-05-18T05:04:19.8466680Z Running tests...
2022-05-18T05:04:19.8467186Z ----------------------------------------------------------------------
2022-05-18T05:04:19.8486531Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T05:04:19.8486842Z 
2022-05-18T05:04:19.8487125Z ----------------------------------------------------------------------
2022-05-18T05:04:19.8487436Z Ran 1 test in 0.002s
2022-05-18T05:04:19.8487603Z 
2022-05-18T05:04:19.8487713Z OK (skipped=1)
2022-05-18T05:04:19.8487868Z 
2022-05-18T05:04:19.8487992Z Generating XML reports...
2022-05-18T05:04:19.8531070Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050419.xml
2022-05-18T05:04:21.1172828Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:21.1188033Z 
2022-05-18T05:04:21.1188186Z Running tests...
2022-05-18T05:04:21.1189213Z ----------------------------------------------------------------------
2022-05-18T05:04:21.1208488Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T05:04:21.1209202Z 
2022-05-18T05:04:21.1209716Z ----------------------------------------------------------------------
2022-05-18T05:04:21.1210160Z Ran 1 test in 0.002s
2022-05-18T05:04:21.1210442Z 
2022-05-18T05:04:21.1210610Z OK (skipped=1)
2022-05-18T05:04:21.1210766Z 
2022-05-18T05:04:21.1210892Z Generating XML reports...
2022-05-18T05:04:21.1252282Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050421.xml
2022-05-18T05:04:22.3895573Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:22.3910893Z 
2022-05-18T05:04:22.3911181Z Running tests...
2022-05-18T05:04:22.3911609Z ----------------------------------------------------------------------
2022-05-18T05:04:22.3933423Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T05:04:22.3933870Z 
2022-05-18T05:04:22.3934267Z ----------------------------------------------------------------------
2022-05-18T05:04:22.3934858Z Ran 1 test in 0.002s
2022-05-18T05:04:22.3935017Z 
2022-05-18T05:04:22.3935123Z OK (skipped=1)
2022-05-18T05:04:22.3935276Z 
2022-05-18T05:04:22.3987423Z Generating XML reports...
2022-05-18T05:04:22.3988661Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050422.xml
2022-05-18T05:04:23.6772386Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:23.6788093Z 
2022-05-18T05:04:23.6788647Z Running tests...
2022-05-18T05:04:23.6789140Z ----------------------------------------------------------------------
2022-05-18T05:04:23.6809186Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T05:04:23.6809510Z 
2022-05-18T05:04:23.6810265Z ----------------------------------------------------------------------
2022-05-18T05:04:23.6810618Z Ran 1 test in 0.002s
2022-05-18T05:04:23.6810791Z 
2022-05-18T05:04:23.6810901Z OK (skipped=1)
2022-05-18T05:04:23.6811055Z 
2022-05-18T05:04:23.6811184Z Generating XML reports...
2022-05-18T05:04:23.6853794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050423.xml
2022-05-18T05:04:24.9618981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:24.9634039Z 
2022-05-18T05:04:24.9634383Z Running tests...
2022-05-18T05:04:24.9634827Z ----------------------------------------------------------------------
2022-05-18T05:04:24.9654407Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T05:04:24.9654695Z 
2022-05-18T05:04:24.9655201Z ----------------------------------------------------------------------
2022-05-18T05:04:24.9655602Z Ran 1 test in 0.002s
2022-05-18T05:04:24.9655764Z 
2022-05-18T05:04:24.9655881Z OK (skipped=1)
2022-05-18T05:04:24.9656044Z 
2022-05-18T05:04:24.9656150Z Generating XML reports...
2022-05-18T05:04:24.9698400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050424.xml
2022-05-18T05:04:26.2445142Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:26.2461001Z 
2022-05-18T05:04:26.2461361Z Running tests...
2022-05-18T05:04:26.2461818Z ----------------------------------------------------------------------
2022-05-18T05:04:26.2482740Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T05:04:26.2483086Z 
2022-05-18T05:04:26.2483414Z ----------------------------------------------------------------------
2022-05-18T05:04:26.2484033Z Ran 1 test in 0.002s
2022-05-18T05:04:26.2484217Z 
2022-05-18T05:04:26.2484309Z OK (skipped=1)
2022-05-18T05:04:26.2484464Z 
2022-05-18T05:04:26.2484590Z Generating XML reports...
2022-05-18T05:04:26.2526614Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050426.xml
2022-05-18T05:04:27.5249936Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:27.5265531Z 
2022-05-18T05:04:27.5266070Z Running tests...
2022-05-18T05:04:27.5266540Z ----------------------------------------------------------------------
2022-05-18T05:04:27.5286050Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T05:04:27.5286357Z 
2022-05-18T05:04:27.5286634Z ----------------------------------------------------------------------
2022-05-18T05:04:27.5286961Z Ran 1 test in 0.002s
2022-05-18T05:04:27.5287106Z 
2022-05-18T05:04:27.5287214Z OK (skipped=1)
2022-05-18T05:04:27.5287387Z 
2022-05-18T05:04:27.5287514Z Generating XML reports...
2022-05-18T05:04:27.5329886Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050427.xml
2022-05-18T05:04:28.7768147Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:28.7783374Z 
2022-05-18T05:04:28.7783772Z Running tests...
2022-05-18T05:04:28.7784208Z ----------------------------------------------------------------------
2022-05-18T05:04:28.7804879Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:04:28.7805201Z 
2022-05-18T05:04:28.7805481Z ----------------------------------------------------------------------
2022-05-18T05:04:28.7805815Z Ran 1 test in 0.002s
2022-05-18T05:04:28.7805984Z 
2022-05-18T05:04:28.7806094Z OK (skipped=1)
2022-05-18T05:04:28.7806250Z 
2022-05-18T05:04:28.7806370Z Generating XML reports...
2022-05-18T05:04:28.7848808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050428.xml
2022-05-18T05:04:30.0141338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:30.0157725Z 
2022-05-18T05:04:30.0158351Z Running tests...
2022-05-18T05:04:30.0158855Z ----------------------------------------------------------------------
2022-05-18T05:04:30.0179141Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:04:30.0179596Z 
2022-05-18T05:04:30.0179899Z ----------------------------------------------------------------------
2022-05-18T05:04:30.0180211Z Ran 1 test in 0.002s
2022-05-18T05:04:30.0180377Z 
2022-05-18T05:04:30.0180485Z OK (skipped=1)
2022-05-18T05:04:30.0180639Z 
2022-05-18T05:04:30.0180761Z Generating XML reports...
2022-05-18T05:04:30.0223673Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050430.xml
2022-05-18T05:04:31.2655082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:31.2671096Z 
2022-05-18T05:04:31.2671463Z Running tests...
2022-05-18T05:04:31.2671901Z ----------------------------------------------------------------------
2022-05-18T05:04:31.2693066Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:04:31.2693434Z 
2022-05-18T05:04:31.2693736Z ----------------------------------------------------------------------
2022-05-18T05:04:31.2694117Z Ran 1 test in 0.002s
2022-05-18T05:04:31.2694279Z 
2022-05-18T05:04:31.2694391Z OK (skipped=1)
2022-05-18T05:04:31.2694544Z 
2022-05-18T05:04:31.2694669Z Generating XML reports...
2022-05-18T05:04:31.2739260Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050431.xml
2022-05-18T05:04:32.5495965Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:32.5511579Z 
2022-05-18T05:04:32.5511949Z Running tests...
2022-05-18T05:04:32.5512376Z ----------------------------------------------------------------------
2022-05-18T05:04:32.5534038Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:04:32.5534367Z 
2022-05-18T05:04:32.5534661Z ----------------------------------------------------------------------
2022-05-18T05:04:32.5534968Z Ran 1 test in 0.002s
2022-05-18T05:04:32.5535129Z 
2022-05-18T05:04:32.5535236Z OK (skipped=1)
2022-05-18T05:04:32.5535387Z 
2022-05-18T05:04:32.5535518Z Generating XML reports...
2022-05-18T05:04:32.5578092Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050432.xml
2022-05-18T05:04:33.8328596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:33.8344276Z 
2022-05-18T05:04:33.8344560Z Running tests...
2022-05-18T05:04:33.8344996Z ----------------------------------------------------------------------
2022-05-18T05:04:33.8366125Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:04:33.8366457Z 
2022-05-18T05:04:33.8366746Z ----------------------------------------------------------------------
2022-05-18T05:04:33.8367055Z Ran 1 test in 0.002s
2022-05-18T05:04:33.8367214Z 
2022-05-18T05:04:33.8367322Z OK (skipped=1)
2022-05-18T05:04:33.8367477Z 
2022-05-18T05:04:33.8367601Z Generating XML reports...
2022-05-18T05:04:33.8409913Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050433.xml
2022-05-18T05:04:35.0963208Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:35.0978080Z 
2022-05-18T05:04:35.0978598Z Running tests...
2022-05-18T05:04:35.0979134Z ----------------------------------------------------------------------
2022-05-18T05:04:35.0997957Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:04:35.0998307Z 
2022-05-18T05:04:35.0998591Z ----------------------------------------------------------------------
2022-05-18T05:04:35.0998899Z Ran 1 test in 0.002s
2022-05-18T05:04:35.0999067Z 
2022-05-18T05:04:35.0999174Z OK (skipped=1)
2022-05-18T05:04:35.0999327Z 
2022-05-18T05:04:35.0999454Z Generating XML reports...
2022-05-18T05:04:35.1039998Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050435.xml
2022-05-18T05:04:36.3755311Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:36.3770297Z 
2022-05-18T05:04:36.3770610Z Running tests...
2022-05-18T05:04:36.3771293Z ----------------------------------------------------------------------
2022-05-18T05:04:36.3792346Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:04:36.3792776Z 
2022-05-18T05:04:36.3793059Z ----------------------------------------------------------------------
2022-05-18T05:04:36.3793421Z Ran 1 test in 0.002s
2022-05-18T05:04:36.3793720Z 
2022-05-18T05:04:36.3793898Z OK (skipped=1)
2022-05-18T05:04:36.3794089Z 
2022-05-18T05:04:36.3794196Z Generating XML reports...
2022-05-18T05:04:36.3836585Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050436.xml
2022-05-18T05:04:37.6578017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:37.6594060Z 
2022-05-18T05:04:37.6594336Z Running tests...
2022-05-18T05:04:37.6595318Z ----------------------------------------------------------------------
2022-05-18T05:04:37.6616229Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:04:37.6616572Z 
2022-05-18T05:04:37.6617336Z ----------------------------------------------------------------------
2022-05-18T05:04:37.6617771Z Ran 1 test in 0.002s
2022-05-18T05:04:37.6617940Z 
2022-05-18T05:04:37.6618052Z OK (skipped=1)
2022-05-18T05:04:37.6618217Z 
2022-05-18T05:04:37.6618342Z Generating XML reports...
2022-05-18T05:04:37.6661290Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050437.xml
2022-05-18T05:04:38.9417186Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:38.9432565Z 
2022-05-18T05:04:38.9433042Z Running tests...
2022-05-18T05:04:38.9433541Z ----------------------------------------------------------------------
2022-05-18T05:04:38.9454943Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:04:38.9455541Z 
2022-05-18T05:04:38.9455836Z ----------------------------------------------------------------------
2022-05-18T05:04:38.9456142Z Ran 1 test in 0.002s
2022-05-18T05:04:38.9456301Z 
2022-05-18T05:04:38.9456409Z OK (skipped=1)
2022-05-18T05:04:38.9456564Z 
2022-05-18T05:04:38.9456692Z Generating XML reports...
2022-05-18T05:04:38.9499011Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050438.xml
2022-05-18T05:04:40.2248865Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:40.2264361Z 
2022-05-18T05:04:40.2264619Z Running tests...
2022-05-18T05:04:40.2265057Z ----------------------------------------------------------------------
2022-05-18T05:04:40.2284752Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:04:40.2285086Z 
2022-05-18T05:04:40.2285346Z ----------------------------------------------------------------------
2022-05-18T05:04:40.2285688Z Ran 1 test in 0.002s
2022-05-18T05:04:40.2285854Z 
2022-05-18T05:04:40.2285962Z OK (skipped=1)
2022-05-18T05:04:40.2286114Z 
2022-05-18T05:04:40.2286245Z Generating XML reports...
2022-05-18T05:04:40.2329641Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050440.xml
2022-05-18T05:04:41.5009338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:41.5024667Z 
2022-05-18T05:04:41.5025007Z Running tests...
2022-05-18T05:04:41.5025458Z ----------------------------------------------------------------------
2022-05-18T05:04:41.5045587Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:04:41.5046056Z 
2022-05-18T05:04:41.5046557Z ----------------------------------------------------------------------
2022-05-18T05:04:41.5046904Z Ran 1 test in 0.002s
2022-05-18T05:04:41.5047067Z 
2022-05-18T05:04:41.5047177Z OK (skipped=1)
2022-05-18T05:04:41.5047314Z 
2022-05-18T05:04:41.5047437Z Generating XML reports...
2022-05-18T05:04:41.5088637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050441.xml
2022-05-18T05:04:42.7533429Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:42.7548670Z 
2022-05-18T05:04:42.7549033Z Running tests...
2022-05-18T05:04:42.7549471Z ----------------------------------------------------------------------
2022-05-18T05:04:42.7570484Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:04:42.7571051Z 
2022-05-18T05:04:42.7571353Z ----------------------------------------------------------------------
2022-05-18T05:04:42.7571664Z Ran 1 test in 0.002s
2022-05-18T05:04:42.7571833Z 
2022-05-18T05:04:42.7571939Z OK (skipped=1)
2022-05-18T05:04:42.7572092Z 
2022-05-18T05:04:42.7572221Z Generating XML reports...
2022-05-18T05:04:42.7615251Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050442.xml
2022-05-18T05:04:44.0426275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:44.0441208Z 
2022-05-18T05:04:44.0441571Z Running tests...
2022-05-18T05:04:44.0442014Z ----------------------------------------------------------------------
2022-05-18T05:04:44.0463984Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:04:44.0464400Z 
2022-05-18T05:04:44.0464705Z ----------------------------------------------------------------------
2022-05-18T05:04:44.0465033Z Ran 1 test in 0.002s
2022-05-18T05:04:44.0465193Z 
2022-05-18T05:04:44.0465301Z OK (skipped=1)
2022-05-18T05:04:44.0465747Z 
2022-05-18T05:04:44.0465853Z Generating XML reports...
2022-05-18T05:04:44.0507912Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050444.xml
2022-05-18T05:04:45.3266005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:45.3281096Z 
2022-05-18T05:04:45.3281468Z Running tests...
2022-05-18T05:04:45.3281977Z ----------------------------------------------------------------------
2022-05-18T05:04:45.3301667Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:04:45.3302021Z 
2022-05-18T05:04:45.3302330Z ----------------------------------------------------------------------
2022-05-18T05:04:45.3302639Z Ran 1 test in 0.002s
2022-05-18T05:04:45.3302801Z 
2022-05-18T05:04:45.3302910Z OK (skipped=1)
2022-05-18T05:04:45.3303064Z 
2022-05-18T05:04:45.3303187Z Generating XML reports...
2022-05-18T05:04:45.3345864Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050445.xml
2022-05-18T05:04:46.6139167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:46.6154220Z 
2022-05-18T05:04:46.6154796Z Running tests...
2022-05-18T05:04:46.6155345Z ----------------------------------------------------------------------
2022-05-18T05:04:46.6176165Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:04:46.6176530Z 
2022-05-18T05:04:46.6176954Z ----------------------------------------------------------------------
2022-05-18T05:04:46.6177426Z Ran 1 test in 0.002s
2022-05-18T05:04:46.6177591Z 
2022-05-18T05:04:46.6177700Z OK (skipped=1)
2022-05-18T05:04:46.6177854Z 
2022-05-18T05:04:46.6177960Z Generating XML reports...
2022-05-18T05:04:46.6221026Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050446.xml
2022-05-18T05:04:47.9041958Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:47.9057019Z 
2022-05-18T05:04:47.9057483Z Running tests...
2022-05-18T05:04:47.9058144Z ----------------------------------------------------------------------
2022-05-18T05:04:47.9077625Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:04:47.9078139Z 
2022-05-18T05:04:47.9078432Z ----------------------------------------------------------------------
2022-05-18T05:04:47.9078767Z Ran 1 test in 0.002s
2022-05-18T05:04:47.9079038Z 
2022-05-18T05:04:47.9079517Z OK (skipped=1)
2022-05-18T05:04:47.9079684Z 
2022-05-18T05:04:47.9079813Z Generating XML reports...
2022-05-18T05:04:47.9122209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050447.xml
2022-05-18T05:04:49.1888177Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:49.1903782Z 
2022-05-18T05:04:49.1904132Z Running tests...
2022-05-18T05:04:49.1904606Z ----------------------------------------------------------------------
2022-05-18T05:04:49.1925581Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:04:49.1925929Z 
2022-05-18T05:04:49.1926216Z ----------------------------------------------------------------------
2022-05-18T05:04:49.1926526Z Ran 1 test in 0.002s
2022-05-18T05:04:49.1926688Z 
2022-05-18T05:04:49.1926797Z OK (skipped=1)
2022-05-18T05:04:49.1926952Z 
2022-05-18T05:04:49.1927086Z Generating XML reports...
2022-05-18T05:04:49.1969624Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050449.xml
2022-05-18T05:04:50.4685582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:50.4703646Z 
2022-05-18T05:04:50.4704060Z Running tests...
2022-05-18T05:04:50.4704557Z ----------------------------------------------------------------------
2022-05-18T05:04:52.1516595Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:52.1888015Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51282
2022-05-18T05:04:52.1998448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51283
2022-05-18T05:04:53.3533289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:53.3533875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:53.3534651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:53.3535358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:53.3542493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:53.3543260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:55.6946738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:04:55.6947299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:04:55.6948075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:04:55.6948774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:04:56.1089967Z ok (5.638s)
2022-05-18T05:04:56.1090502Z 
2022-05-18T05:04:56.1090890Z ----------------------------------------------------------------------
2022-05-18T05:04:56.1091233Z Ran 1 test in 5.639s
2022-05-18T05:04:56.1091395Z 
2022-05-18T05:04:56.1091487Z OK
2022-05-18T05:04:56.1091623Z 
2022-05-18T05:04:56.1091759Z Generating XML reports...
2022-05-18T05:04:56.1149136Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050450.xml
2022-05-18T05:04:57.5563489Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:57.5578927Z 
2022-05-18T05:04:57.5579351Z Running tests...
2022-05-18T05:04:57.5579832Z ----------------------------------------------------------------------
2022-05-18T05:04:59.2085639Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:59.2451698Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51411
2022-05-18T05:04:59.2561876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51412
2022-05-18T05:05:00.4724932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:00.4725492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:00.4726251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:00.4726947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:00.4734283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:00.4734870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:00.6608942Z skip: Need at least 3 CUDA devices (3.103s)
2022-05-18T05:05:00.6609212Z 
2022-05-18T05:05:00.6610284Z ----------------------------------------------------------------------
2022-05-18T05:05:00.6610638Z Ran 1 test in 3.103s
2022-05-18T05:05:00.6610802Z 
2022-05-18T05:05:00.6610893Z OK (skipped=1)
2022-05-18T05:05:00.6611051Z 
2022-05-18T05:05:00.6611177Z Generating XML reports...
2022-05-18T05:05:00.6666835Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050457.xml
2022-05-18T05:05:02.0952018Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:02.0968965Z 
2022-05-18T05:05:02.0969181Z Running tests...
2022-05-18T05:05:02.0969594Z ----------------------------------------------------------------------
2022-05-18T05:05:02.0992041Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-05-18T05:05:02.0992670Z 
2022-05-18T05:05:02.0993337Z ----------------------------------------------------------------------
2022-05-18T05:05:02.0993991Z Ran 1 test in 0.002s
2022-05-18T05:05:02.0994320Z 
2022-05-18T05:05:02.0994514Z OK (skipped=1)
2022-05-18T05:05:02.0994802Z 
2022-05-18T05:05:02.0995043Z Generating XML reports...
2022-05-18T05:05:02.1039077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050502.xml
2022-05-18T05:05:03.3752033Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:03.3768752Z 
2022-05-18T05:05:03.3769277Z Running tests...
2022-05-18T05:05:03.3770120Z ----------------------------------------------------------------------
2022-05-18T05:05:05.0403552Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:05.0779066Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51561
2022-05-18T05:05:05.0889369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51562
2022-05-18T05:05:06.2480818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:06.2481862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:06.2483197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:06.2484582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:06.2489287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:06.2491702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:07.2950901Z ok (3.918s)
2022-05-18T05:05:07.2951377Z 
2022-05-18T05:05:07.2951791Z ----------------------------------------------------------------------
2022-05-18T05:05:07.2952137Z Ran 1 test in 3.918s
2022-05-18T05:05:07.2952296Z 
2022-05-18T05:05:07.2952388Z OK
2022-05-18T05:05:07.2952528Z 
2022-05-18T05:05:07.2952665Z Generating XML reports...
2022-05-18T05:05:07.3009436Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050503.xml
2022-05-18T05:05:08.7407906Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:08.7424959Z 
2022-05-18T05:05:08.7425476Z Running tests...
2022-05-18T05:05:08.7425990Z ----------------------------------------------------------------------
2022-05-18T05:05:10.3936212Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:10.4313344Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51676
2022-05-18T05:05:10.4425325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51677
2022-05-18T05:05:11.6303786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:11.6304616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:11.6305418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:11.6306098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:11.6412695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:11.7318471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:14.1514777Z ok (5.409s)
2022-05-18T05:05:14.1515020Z 
2022-05-18T05:05:14.1515401Z ----------------------------------------------------------------------
2022-05-18T05:05:14.1515740Z Ran 1 test in 5.409s
2022-05-18T05:05:14.1515907Z 
2022-05-18T05:05:14.1516004Z OK
2022-05-18T05:05:14.1516139Z 
2022-05-18T05:05:14.1516263Z Generating XML reports...
2022-05-18T05:05:14.1575225Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050508.xml
2022-05-18T05:05:15.5961725Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:15.5976881Z 
2022-05-18T05:05:15.5977137Z Running tests...
2022-05-18T05:05:15.5977569Z ----------------------------------------------------------------------
2022-05-18T05:05:17.2274263Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:17.2644406Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51793
2022-05-18T05:05:17.2751503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51794
2022-05-18T05:05:18.4667458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:18.4668008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:18.4668801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:18.4669501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:18.4776846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:18.5679206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:18.5887123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:05:18.5888212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:05:18.5889247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:05:18.5890231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:05:19.5813150Z ok (3.983s)
2022-05-18T05:05:19.5813370Z 
2022-05-18T05:05:19.5813760Z ----------------------------------------------------------------------
2022-05-18T05:05:19.5814100Z Ran 1 test in 3.984s
2022-05-18T05:05:19.5814267Z 
2022-05-18T05:05:19.5814370Z OK
2022-05-18T05:05:19.5814485Z 
2022-05-18T05:05:19.5814620Z Generating XML reports...
2022-05-18T05:05:19.5871126Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050515.xml
2022-05-18T05:05:21.0223043Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:21.0238379Z 
2022-05-18T05:05:21.0238537Z Running tests...
2022-05-18T05:05:21.0238978Z ----------------------------------------------------------------------
2022-05-18T05:05:22.6716338Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:22.7093497Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51914
2022-05-18T05:05:22.7204574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51915
2022-05-18T05:05:23.9058010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:23.9058570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:23.9059362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:23.9060079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:23.9166592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:24.0073123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:24.2255151Z skip: Skipped due to small world size. (3.201s)
2022-05-18T05:05:24.2255622Z 
2022-05-18T05:05:24.2256258Z ----------------------------------------------------------------------
2022-05-18T05:05:24.2256897Z Ran 1 test in 3.202s
2022-05-18T05:05:24.2257183Z 
2022-05-18T05:05:24.2257387Z OK (skipped=1)
2022-05-18T05:05:24.2257676Z 
2022-05-18T05:05:24.2257902Z Generating XML reports...
2022-05-18T05:05:24.2315225Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050521.xml
2022-05-18T05:05:25.6532259Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:25.6547464Z 
2022-05-18T05:05:25.6547734Z Running tests...
2022-05-18T05:05:25.6548303Z ----------------------------------------------------------------------
2022-05-18T05:05:27.3104891Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:27.3479552Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52029
2022-05-18T05:05:27.3590721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52030
2022-05-18T05:05:28.5817278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:28.5817847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:28.5818642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:28.5819564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:28.5925592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:28.6832132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:28.8641870Z skip: Skipped due to small world size. (3.209s)
2022-05-18T05:05:28.8642109Z 
2022-05-18T05:05:28.8642503Z ----------------------------------------------------------------------
2022-05-18T05:05:28.8642842Z Ran 1 test in 3.209s
2022-05-18T05:05:28.8643008Z 
2022-05-18T05:05:28.8643117Z OK (skipped=1)
2022-05-18T05:05:28.8643273Z 
2022-05-18T05:05:28.8643378Z Generating XML reports...
2022-05-18T05:05:28.8700922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050525.xml
2022-05-18T05:05:30.2863643Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:30.2878455Z 
2022-05-18T05:05:30.2878898Z Running tests...
2022-05-18T05:05:30.2879402Z ----------------------------------------------------------------------
2022-05-18T05:05:31.9440512Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:31.9819538Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52144
2022-05-18T05:05:31.9933473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52145
2022-05-18T05:05:33.1798282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:33.1798835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:33.1799610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:33.1800300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:33.1908093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:33.2812542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:33.4986435Z skip: Skipped due to small world size. (3.210s)
2022-05-18T05:05:33.4986680Z 
2022-05-18T05:05:33.4987079Z ----------------------------------------------------------------------
2022-05-18T05:05:33.4987419Z Ran 1 test in 3.211s
2022-05-18T05:05:33.4987594Z 
2022-05-18T05:05:33.4987685Z OK (skipped=1)
2022-05-18T05:05:33.4987843Z 
2022-05-18T05:05:33.4987973Z Generating XML reports...
2022-05-18T05:05:33.5045069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050530.xml
2022-05-18T05:05:34.9482267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:34.9498176Z 
2022-05-18T05:05:34.9498842Z Running tests...
2022-05-18T05:05:34.9499368Z ----------------------------------------------------------------------
2022-05-18T05:05:36.6021349Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:36.6400234Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52259
2022-05-18T05:05:36.6513295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52260
2022-05-18T05:05:37.9029750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:37.9030300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:37.9031092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:37.9031791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:37.9038785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:37.9039280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:37.9146483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:05:37.9146986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:05:37.9147662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:05:37.9148331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:05:39.1577487Z ok (4.208s)
2022-05-18T05:05:39.1577829Z 
2022-05-18T05:05:39.1578292Z ----------------------------------------------------------------------
2022-05-18T05:05:39.1578635Z Ran 1 test in 4.208s
2022-05-18T05:05:39.1578814Z 
2022-05-18T05:05:39.1578890Z OK
2022-05-18T05:05:39.1579027Z 
2022-05-18T05:05:39.1579161Z Generating XML reports...
2022-05-18T05:05:39.1635958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050534.xml
2022-05-18T05:05:40.6170318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:40.6184528Z 
2022-05-18T05:05:40.6184755Z Running tests...
2022-05-18T05:05:40.6185188Z ----------------------------------------------------------------------
2022-05-18T05:05:40.6210086Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s)
2022-05-18T05:05:40.6210930Z 
2022-05-18T05:05:40.6211225Z ----------------------------------------------------------------------
2022-05-18T05:05:40.6211555Z Ran 1 test in 0.003s
2022-05-18T05:05:40.6211722Z 
2022-05-18T05:05:40.6211833Z OK (skipped=1)
2022-05-18T05:05:40.6211989Z 
2022-05-18T05:05:40.6212118Z Generating XML reports...
2022-05-18T05:05:40.6254520Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050540.xml
2022-05-18T05:05:41.8881332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:41.8896329Z 
2022-05-18T05:05:41.8896656Z Running tests...
2022-05-18T05:05:41.8897088Z ----------------------------------------------------------------------
2022-05-18T05:05:43.5402285Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:43.5777104Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52415
2022-05-18T05:05:43.5887972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52416
2022-05-18T05:05:44.8051147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:44.8051678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:44.8052470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:44.8053179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:44.8159968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:44.9066379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:45.0937136Z skip: Skipped due to small world size. (3.204s)
2022-05-18T05:05:45.0937390Z 
2022-05-18T05:05:45.0937765Z ----------------------------------------------------------------------
2022-05-18T05:05:45.0938080Z Ran 1 test in 3.204s
2022-05-18T05:05:45.0938243Z 
2022-05-18T05:05:45.0938599Z OK (skipped=1)
2022-05-18T05:05:45.0938777Z 
2022-05-18T05:05:45.0938904Z Generating XML reports...
2022-05-18T05:05:45.0995770Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050541.xml
2022-05-18T05:05:46.5399327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:46.5415418Z 
2022-05-18T05:05:46.5415676Z Running tests...
2022-05-18T05:05:46.5416117Z ----------------------------------------------------------------------
2022-05-18T05:05:48.2059310Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:48.2427838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52530
2022-05-18T05:05:48.2539006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52531
2022-05-18T05:05:49.4451168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:49.4451761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:49.4452541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:49.4453481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:49.4560670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:49.5463602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:49.7588843Z ok (3.217s)
2022-05-18T05:05:49.7589185Z 
2022-05-18T05:05:49.7589618Z ----------------------------------------------------------------------
2022-05-18T05:05:49.7589965Z Ran 1 test in 3.217s
2022-05-18T05:05:49.7590110Z 
2022-05-18T05:05:49.7590201Z OK
2022-05-18T05:05:49.7590337Z 
2022-05-18T05:05:49.7590489Z Generating XML reports...
2022-05-18T05:05:49.7646434Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050546.xml
2022-05-18T05:05:51.1460770Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:51.1475756Z 
2022-05-18T05:05:51.1476004Z Running tests...
2022-05-18T05:05:51.1476662Z ----------------------------------------------------------------------
2022-05-18T05:05:52.8093149Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:52.8471843Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52645
2022-05-18T05:05:52.8584379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52646
2022-05-18T05:05:54.0921738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:54.0922313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:54.0923104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:54.0923809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:54.0930476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:54.0931427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:54.2631795Z ok (3.115s)
2022-05-18T05:05:54.2632017Z 
2022-05-18T05:05:54.2632403Z ----------------------------------------------------------------------
2022-05-18T05:05:54.2632725Z Ran 1 test in 3.116s
2022-05-18T05:05:54.2632889Z 
2022-05-18T05:05:54.2632983Z OK
2022-05-18T05:05:54.2633118Z 
2022-05-18T05:05:54.2633248Z Generating XML reports...
2022-05-18T05:05:54.2690581Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050551.xml
2022-05-18T05:05:55.6881291Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:55.6896657Z 
2022-05-18T05:05:55.6897100Z Running tests...
2022-05-18T05:05:55.6897588Z ----------------------------------------------------------------------
2022-05-18T05:05:55.6925001Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T05:05:55.6925336Z 
2022-05-18T05:05:55.6925630Z ----------------------------------------------------------------------
2022-05-18T05:05:55.6925941Z Ran 1 test in 0.003s
2022-05-18T05:05:55.6926105Z 
2022-05-18T05:05:55.6926216Z OK (skipped=1)
2022-05-18T05:05:55.6926367Z 
2022-05-18T05:05:55.6926491Z Generating XML reports...
2022-05-18T05:05:55.6968769Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050555.xml
2022-05-18T05:05:56.9688904Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:56.9704109Z 
2022-05-18T05:05:56.9704530Z Running tests...
2022-05-18T05:05:56.9704977Z ----------------------------------------------------------------------
2022-05-18T05:05:56.9735034Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T05:05:56.9735343Z 
2022-05-18T05:05:56.9735631Z ----------------------------------------------------------------------
2022-05-18T05:05:56.9735936Z Ran 1 test in 0.003s
2022-05-18T05:05:56.9736096Z 
2022-05-18T05:05:56.9736204Z OK (skipped=1)
2022-05-18T05:05:56.9736366Z 
2022-05-18T05:05:56.9736488Z Generating XML reports...
2022-05-18T05:05:56.9778638Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050556.xml
2022-05-18T05:05:58.2320095Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:58.2335064Z 
2022-05-18T05:05:58.2335387Z Running tests...
2022-05-18T05:05:58.2335825Z ----------------------------------------------------------------------
2022-05-18T05:05:58.2365862Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T05:05:58.2366187Z 
2022-05-18T05:05:58.2366453Z ----------------------------------------------------------------------
2022-05-18T05:05:58.2366780Z Ran 1 test in 0.003s
2022-05-18T05:05:58.2366942Z 
2022-05-18T05:05:58.2367050Z OK (skipped=1)
2022-05-18T05:05:58.2367212Z 
2022-05-18T05:05:58.2367337Z Generating XML reports...
2022-05-18T05:05:58.2407935Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050558.xml
2022-05-18T05:05:59.4691758Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:59.4707107Z 
2022-05-18T05:05:59.4707248Z Running tests...
2022-05-18T05:05:59.4707997Z ----------------------------------------------------------------------
2022-05-18T05:05:59.4732385Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T05:05:59.4732687Z 
2022-05-18T05:05:59.4732960Z ----------------------------------------------------------------------
2022-05-18T05:05:59.4733288Z Ran 1 test in 0.003s
2022-05-18T05:05:59.4733449Z 
2022-05-18T05:05:59.4733566Z OK (skipped=1)
2022-05-18T05:05:59.4733720Z 
2022-05-18T05:05:59.4733842Z Generating XML reports...
2022-05-18T05:05:59.4776075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050559.xml
2022-05-18T05:06:00.7541373Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:00.7556360Z 
2022-05-18T05:06:00.7556687Z Running tests...
2022-05-18T05:06:00.7557402Z ----------------------------------------------------------------------
2022-05-18T05:06:00.7579305Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T05:06:00.7579638Z 
2022-05-18T05:06:00.7579922Z ----------------------------------------------------------------------
2022-05-18T05:06:00.7580248Z Ran 1 test in 0.002s
2022-05-18T05:06:00.7580408Z 
2022-05-18T05:06:00.7580518Z OK (skipped=1)
2022-05-18T05:06:00.7581766Z 
2022-05-18T05:06:00.7582353Z Generating XML reports...
2022-05-18T05:06:00.7623838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050600.xml
2022-05-18T05:06:02.0384919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:02.0400041Z 
2022-05-18T05:06:02.0400304Z Running tests...
2022-05-18T05:06:02.0400734Z ----------------------------------------------------------------------
2022-05-18T05:06:02.0429029Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T05:06:02.0429470Z 
2022-05-18T05:06:02.0430167Z ----------------------------------------------------------------------
2022-05-18T05:06:02.0430528Z Ran 1 test in 0.003s
2022-05-18T05:06:02.0430694Z 
2022-05-18T05:06:02.0430818Z OK (skipped=1)
2022-05-18T05:06:02.0430959Z 
2022-05-18T05:06:02.0431085Z Generating XML reports...
2022-05-18T05:06:02.0473901Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050602.xml
2022-05-18T05:06:03.3022237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:03.3036481Z 
2022-05-18T05:06:03.3036627Z Running tests...
2022-05-18T05:06:03.3037518Z ----------------------------------------------------------------------
2022-05-18T05:06:03.3065517Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T05:06:03.3065831Z 
2022-05-18T05:06:03.3066123Z ----------------------------------------------------------------------
2022-05-18T05:06:03.3066459Z Ran 1 test in 0.003s
2022-05-18T05:06:03.3066612Z 
2022-05-18T05:06:03.3066723Z OK (skipped=1)
2022-05-18T05:06:03.3066879Z 
2022-05-18T05:06:03.3067004Z Generating XML reports...
2022-05-18T05:06:03.3107740Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050603.xml
2022-05-18T05:06:04.5748315Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:04.5762787Z 
2022-05-18T05:06:04.5763091Z Running tests...
2022-05-18T05:06:04.5763595Z ----------------------------------------------------------------------
2022-05-18T05:06:04.5786729Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T05:06:04.5787195Z 
2022-05-18T05:06:04.5787707Z ----------------------------------------------------------------------
2022-05-18T05:06:04.5788029Z Ran 1 test in 0.002s
2022-05-18T05:06:04.5788191Z 
2022-05-18T05:06:04.5788311Z OK (skipped=1)
2022-05-18T05:06:04.5788465Z 
2022-05-18T05:06:04.5788589Z Generating XML reports...
2022-05-18T05:06:04.5830840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050604.xml
2022-05-18T05:06:05.8340063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:05.8353897Z 
2022-05-18T05:06:05.8354137Z Running tests...
2022-05-18T05:06:05.8354566Z ----------------------------------------------------------------------
2022-05-18T05:06:07.4685912Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:07.5056585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53040
2022-05-18T05:06:07.5164557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53041
2022-05-18T05:06:08.7046303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:08.7046867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:08.7047648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:08.7048342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:08.7154938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:08.8057355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:09.0213375Z ok (3.186s)
2022-05-18T05:06:09.0213603Z 
2022-05-18T05:06:09.0214003Z ----------------------------------------------------------------------
2022-05-18T05:06:09.0214325Z Ran 1 test in 3.186s
2022-05-18T05:06:09.0214491Z 
2022-05-18T05:06:09.0214586Z OK
2022-05-18T05:06:09.0214722Z 
2022-05-18T05:06:09.0216408Z Generating XML reports...
2022-05-18T05:06:09.0271472Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050605.xml
2022-05-18T05:06:10.4509358Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:10.4529606Z 
2022-05-18T05:06:10.4529947Z Running tests...
2022-05-18T05:06:10.4530667Z ----------------------------------------------------------------------
2022-05-18T05:06:12.1238486Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:12.1615885Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53159
2022-05-18T05:06:12.1726816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53160
2022-05-18T05:06:13.3906134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:13.3906694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:13.3907491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:13.3908165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:13.3914724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:13.3915529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:15.3807727Z ok (4.928s)
2022-05-18T05:06:15.3807949Z 
2022-05-18T05:06:15.3808336Z ----------------------------------------------------------------------
2022-05-18T05:06:15.3808675Z Ran 1 test in 4.928s
2022-05-18T05:06:15.3808845Z 
2022-05-18T05:06:15.3808940Z OK
2022-05-18T05:06:15.3809075Z 
2022-05-18T05:06:15.3809212Z Generating XML reports...
2022-05-18T05:06:15.3864802Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050610.xml
2022-05-18T05:06:16.8136343Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:16.8150632Z 
2022-05-18T05:06:16.8151033Z Running tests...
2022-05-18T05:06:16.8151981Z ----------------------------------------------------------------------
2022-05-18T05:06:18.4476611Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:18.4851486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53280
2022-05-18T05:06:18.4960839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53281
2022-05-18T05:06:19.7230046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:19.7230620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:19.7231411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:19.7232106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:19.7338607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:19.8240275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:19.8449279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:06:19.8449803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:06:19.8450857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:06:19.8451541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:06:20.1010595Z ok (3.286s)
2022-05-18T05:06:20.1011065Z 
2022-05-18T05:06:20.1011571Z ----------------------------------------------------------------------
2022-05-18T05:06:20.1011947Z Ran 1 test in 3.286s
2022-05-18T05:06:20.1012122Z 
2022-05-18T05:06:20.1012210Z OK
2022-05-18T05:06:20.1012348Z 
2022-05-18T05:06:20.1012484Z Generating XML reports...
2022-05-18T05:06:20.1070526Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050616.xml
2022-05-18T05:06:21.5310727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:21.5325766Z 
2022-05-18T05:06:21.5326038Z Running tests...
2022-05-18T05:06:21.5326491Z ----------------------------------------------------------------------
2022-05-18T05:06:23.1802556Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:23.2173337Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53405
2022-05-18T05:06:23.2282981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53406
2022-05-18T05:06:24.4262620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:24.4263190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:24.4263981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:24.4264662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:24.4372020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:24.5276671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:24.7333108Z skip: Skipped due to small world size. (3.200s)
2022-05-18T05:06:24.7333674Z 
2022-05-18T05:06:24.7334393Z ----------------------------------------------------------------------
2022-05-18T05:06:24.7334800Z Ran 1 test in 3.201s
2022-05-18T05:06:24.7334965Z 
2022-05-18T05:06:24.7335077Z OK (skipped=1)
2022-05-18T05:06:24.7335213Z 
2022-05-18T05:06:24.7335338Z Generating XML reports...
2022-05-18T05:06:24.7391909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050621.xml
2022-05-18T05:06:26.1437341Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:26.1452108Z 
2022-05-18T05:06:26.1452357Z Running tests...
2022-05-18T05:06:26.1453070Z ----------------------------------------------------------------------
2022-05-18T05:06:27.7661226Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:27.8029200Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53520
2022-05-18T05:06:27.8141094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53521
2022-05-18T05:06:29.0308744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:29.0309302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:29.0310100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:29.0310781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:29.0317426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:29.0317914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:30.7215156Z ok (4.576s)
2022-05-18T05:06:30.7215676Z 
2022-05-18T05:06:30.7216060Z ----------------------------------------------------------------------
2022-05-18T05:06:30.7216379Z Ran 1 test in 4.576s
2022-05-18T05:06:30.7216543Z 
2022-05-18T05:06:30.7216633Z OK
2022-05-18T05:06:30.7216767Z 
2022-05-18T05:06:30.7216897Z Generating XML reports...
2022-05-18T05:06:30.7273664Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050626.xml
2022-05-18T05:06:32.1675344Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:32.1690298Z 
2022-05-18T05:06:32.1690726Z Running tests...
2022-05-18T05:06:32.1691230Z ----------------------------------------------------------------------
2022-05-18T05:06:33.8252285Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:33.8628588Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53637
2022-05-18T05:06:33.8739836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53638
2022-05-18T05:06:35.0460288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:35.0460840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:35.0461589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:35.0462290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:35.0569284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:35.1472106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:35.3790842Z ok (3.210s)
2022-05-18T05:06:35.3791211Z 
2022-05-18T05:06:35.3791653Z ----------------------------------------------------------------------
2022-05-18T05:06:35.3792018Z Ran 1 test in 3.210s
2022-05-18T05:06:35.3792182Z 
2022-05-18T05:06:35.3792277Z OK
2022-05-18T05:06:35.3792394Z 
2022-05-18T05:06:35.3792530Z Generating XML reports...
2022-05-18T05:06:35.3850664Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050632.xml
2022-05-18T05:06:36.8131149Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:36.8146384Z 
2022-05-18T05:06:36.8146785Z Running tests...
2022-05-18T05:06:36.8147277Z ----------------------------------------------------------------------
2022-05-18T05:06:38.4704867Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:38.5071796Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53752
2022-05-18T05:06:38.5180996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53753
2022-05-18T05:06:39.7066146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:39.7066706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:39.7067477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:39.7068175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:39.7177574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:39.8078084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:39.8193707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:06:39.8195184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:06:39.8195948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:06:39.8196643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:06:39.8405800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:06:39.8406870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:06:39.8407820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:06:39.8408513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:06:41.1680706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjfy65hph
2022-05-18T05:06:41.1681937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjfy65hph/_remote_module_non_scriptable.py
2022-05-18T05:06:41.1798662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizuut7u0
2022-05-18T05:06:41.1801214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizuut7u0/_remote_module_non_scriptable.py
2022-05-18T05:06:41.5259272Z ok (4.711s)
2022-05-18T05:06:41.5259580Z 
2022-05-18T05:06:41.5260329Z ----------------------------------------------------------------------
2022-05-18T05:06:41.5260979Z Ran 1 test in 4.711s
2022-05-18T05:06:41.5261181Z 
2022-05-18T05:06:41.5261279Z OK
2022-05-18T05:06:41.5261414Z 
2022-05-18T05:06:41.5261567Z Generating XML reports...
2022-05-18T05:06:41.5317667Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050636.xml
2022-05-18T05:06:42.9739173Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:42.9753599Z 
2022-05-18T05:06:42.9754064Z Running tests...
2022-05-18T05:06:42.9754969Z ----------------------------------------------------------------------
2022-05-18T05:06:44.6318736Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:44.6695030Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53881
2022-05-18T05:06:44.6805396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53882
2022-05-18T05:06:45.8491285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:45.8492086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:45.8492880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:45.8493591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:45.8600811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:45.9503062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:45.9618156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:06:45.9618691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:06:45.9619412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:06:45.9620095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:06:45.9826624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:06:45.9827159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:06:45.9827870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:06:45.9828549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:06:47.3112859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbvkzk18
2022-05-18T05:06:47.3114048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbvkzk18/_remote_module_non_scriptable.py
2022-05-18T05:06:47.3217917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz72j2mdu
2022-05-18T05:06:47.3220066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz72j2mdu/_remote_module_non_scriptable.py
2022-05-18T05:06:47.6881465Z ok (4.712s)
2022-05-18T05:06:47.6881686Z 
2022-05-18T05:06:47.6882106Z ----------------------------------------------------------------------
2022-05-18T05:06:47.6882462Z Ran 1 test in 4.713s
2022-05-18T05:06:47.6882612Z 
2022-05-18T05:06:47.6882710Z OK
2022-05-18T05:06:47.6882849Z 
2022-05-18T05:06:47.6882987Z Generating XML reports...
2022-05-18T05:06:47.6941784Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050642.xml
2022-05-18T05:06:49.1300649Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:49.1315799Z 
2022-05-18T05:06:49.1316033Z Running tests...
2022-05-18T05:06:49.1316461Z ----------------------------------------------------------------------
2022-05-18T05:06:50.7990492Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:50.8371487Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54010
2022-05-18T05:06:50.8482605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54011
2022-05-18T05:06:52.1119075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:52.1119657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:52.1120442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:52.1121139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:52.1228882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:52.2134181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:53.4471643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpml77fv1l
2022-05-18T05:06:53.4472722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpml77fv1l/_remote_module_non_scriptable.py
2022-05-18T05:06:53.5107837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9o_c57kq
2022-05-18T05:06:53.5109240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9o_c57kq/_remote_module_non_scriptable.py
2022-05-18T05:06:53.8175624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:53.8176175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:54.1563971Z ok (5.024s)
2022-05-18T05:06:54.1564198Z 
2022-05-18T05:06:54.1564610Z ----------------------------------------------------------------------
2022-05-18T05:06:54.1564951Z Ran 1 test in 5.025s
2022-05-18T05:06:54.1565116Z 
2022-05-18T05:06:54.1567561Z OK
2022-05-18T05:06:54.1568602Z 
2022-05-18T05:06:54.1569033Z Generating XML reports...
2022-05-18T05:06:54.1623477Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050649.xml
2022-05-18T05:06:55.6037187Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:55.6052844Z 
2022-05-18T05:06:55.6053331Z Running tests...
2022-05-18T05:06:55.6053853Z ----------------------------------------------------------------------
2022-05-18T05:06:57.2550327Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:57.2924952Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54131
2022-05-18T05:06:57.3035662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54132
2022-05-18T05:06:58.4793988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:58.4794587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:58.4795405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:58.4796112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:58.4803010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:58.4803480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:59.8182997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqtmmvqp
2022-05-18T05:06:59.8183615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqtmmvqp/_remote_module_non_scriptable.py
2022-05-18T05:06:59.8248068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjwezkmx
2022-05-18T05:06:59.8250783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjwezkmx/_remote_module_non_scriptable.py
2022-05-18T05:07:00.1323486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:00.1324082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:00.1334944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:00.1335439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:00.4114713Z ok (4.806s)
2022-05-18T05:07:00.4114928Z 
2022-05-18T05:07:00.4115533Z ----------------------------------------------------------------------
2022-05-18T05:07:00.4115884Z Ran 1 test in 4.806s
2022-05-18T05:07:00.4116050Z 
2022-05-18T05:07:00.4116389Z OK
2022-05-18T05:07:00.4116547Z 
2022-05-18T05:07:00.4116662Z Generating XML reports...
2022-05-18T05:07:00.4173014Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050655.xml
2022-05-18T05:07:01.8439982Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:01.8454378Z 
2022-05-18T05:07:01.8454775Z Running tests...
2022-05-18T05:07:01.8455217Z ----------------------------------------------------------------------
2022-05-18T05:07:03.4674832Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:03.5041111Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54252
2022-05-18T05:07:03.5153702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54253
2022-05-18T05:07:04.7322146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:04.7322714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:04.7323491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:04.7324442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:04.7431079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:04.8336939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:06.0645659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdzq7g1k
2022-05-18T05:07:06.0646534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdzq7g1k/_remote_module_non_scriptable.py
2022-05-18T05:07:06.1092272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgbp50kg
2022-05-18T05:07:06.1094727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgbp50kg/_remote_module_non_scriptable.py
2022-05-18T05:07:06.4170764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:06.4171310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:06.4186243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:06.4186736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:06.4348766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:06.4349245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:06.4362721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:06.4363219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:06.7234596Z ok (4.878s)
2022-05-18T05:07:06.7234820Z 
2022-05-18T05:07:06.7235201Z ----------------------------------------------------------------------
2022-05-18T05:07:06.7235542Z Ran 1 test in 4.878s
2022-05-18T05:07:06.7235709Z 
2022-05-18T05:07:06.7235810Z OK
2022-05-18T05:07:06.7235943Z 
2022-05-18T05:07:06.7236076Z Generating XML reports...
2022-05-18T05:07:06.7292112Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050701.xml
2022-05-18T05:07:08.1359115Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:08.1375174Z 
2022-05-18T05:07:08.1375455Z Running tests...
2022-05-18T05:07:08.1375890Z ----------------------------------------------------------------------
2022-05-18T05:07:09.8030164Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:09.8150440Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.677s)
2022-05-18T05:07:09.8151645Z 
2022-05-18T05:07:09.8152093Z ----------------------------------------------------------------------
2022-05-18T05:07:09.8152418Z Ran 1 test in 1.677s
2022-05-18T05:07:09.8152582Z 
2022-05-18T05:07:09.8152691Z OK (skipped=1)
2022-05-18T05:07:09.8152846Z 
2022-05-18T05:07:09.8152971Z Generating XML reports...
2022-05-18T05:07:09.8191707Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050708.xml
2022-05-18T05:07:11.2136152Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:11.2151569Z 
2022-05-18T05:07:11.2151941Z Running tests...
2022-05-18T05:07:11.2152414Z ----------------------------------------------------------------------
2022-05-18T05:07:12.8861242Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:12.9236124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54409
2022-05-18T05:07:12.9346801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54410
2022-05-18T05:07:14.1424958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:14.1425499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:14.1426312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:14.1427035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:14.1434137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:14.1434927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:15.4754670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7svyl73
2022-05-18T05:07:15.4755289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7svyl73/_remote_module_non_scriptable.py
2022-05-18T05:07:15.4945926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9uz87nu
2022-05-18T05:07:15.4948904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9uz87nu/_remote_module_non_scriptable.py
2022-05-18T05:07:15.5160884Z <class 'torch.nn.parameter.Parameter'>
2022-05-18T05:07:15.8422788Z ok (4.627s)
2022-05-18T05:07:15.8423007Z 
2022-05-18T05:07:15.8423391Z ----------------------------------------------------------------------
2022-05-18T05:07:15.8423743Z Ran 1 test in 4.627s
2022-05-18T05:07:15.8423888Z 
2022-05-18T05:07:15.8423982Z OK
2022-05-18T05:07:15.8424117Z 
2022-05-18T05:07:15.8424249Z Generating XML reports...
2022-05-18T05:07:15.8481333Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050711.xml
2022-05-18T05:07:17.2815122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:17.2830989Z 
2022-05-18T05:07:17.2831351Z Running tests...
2022-05-18T05:07:17.2831798Z ----------------------------------------------------------------------
2022-05-18T05:07:18.9391363Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:18.9767863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54526
2022-05-18T05:07:18.9879822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54527
2022-05-18T05:07:20.1877832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:20.1878413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:20.1879227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:20.1879903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:20.1886317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:20.1887508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:21.5197451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dnuei14
2022-05-18T05:07:21.5198130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dnuei14/_remote_module_non_scriptable.py
2022-05-18T05:07:21.5239901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05ep6h_3
2022-05-18T05:07:21.5242776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05ep6h_3/_remote_module_non_scriptable.py
2022-05-18T05:07:21.7952649Z ok (4.512s)
2022-05-18T05:07:21.7952844Z 
2022-05-18T05:07:21.7953436Z ----------------------------------------------------------------------
2022-05-18T05:07:21.7953807Z Ran 1 test in 4.512s
2022-05-18T05:07:21.7953955Z 
2022-05-18T05:07:21.7954058Z OK
2022-05-18T05:07:21.7954196Z 
2022-05-18T05:07:21.7954332Z Generating XML reports...
2022-05-18T05:07:21.8011557Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050717.xml
2022-05-18T05:07:23.2132360Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:23.2147043Z 
2022-05-18T05:07:23.2147275Z Running tests...
2022-05-18T05:07:23.2147723Z ----------------------------------------------------------------------
2022-05-18T05:07:24.8205281Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:24.8571750Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54643
2022-05-18T05:07:24.8679968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54644
2022-05-18T05:07:26.0560718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:26.0561289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:26.0562086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:26.0562766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:26.0670169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:26.1576728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:27.3636678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnuzisd8q
2022-05-18T05:07:27.3637711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnuzisd8q/_remote_module_non_scriptable.py
2022-05-18T05:07:27.4522010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgy6wugm
2022-05-18T05:07:27.4523114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgy6wugm/_remote_module_non_scriptable.py
2022-05-18T05:07:27.7734212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:27.7734782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:28.0761649Z ok (4.861s)
2022-05-18T05:07:28.0762368Z 
2022-05-18T05:07:28.0763125Z ----------------------------------------------------------------------
2022-05-18T05:07:28.0763517Z Ran 1 test in 4.861s
2022-05-18T05:07:28.0763661Z 
2022-05-18T05:07:28.0763770Z OK
2022-05-18T05:07:28.0763908Z 
2022-05-18T05:07:28.0764045Z Generating XML reports...
2022-05-18T05:07:28.0820840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050723.xml
2022-05-18T05:07:29.5113379Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:29.5128508Z 
2022-05-18T05:07:29.5128812Z Running tests...
2022-05-18T05:07:29.5129253Z ----------------------------------------------------------------------
2022-05-18T05:07:31.1515924Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:31.1886677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54764
2022-05-18T05:07:31.1996356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54765
2022-05-18T05:07:32.4072515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:32.4073736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:32.4074535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:32.4075231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:32.4182221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:32.5087471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:33.7023445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd39av23z
2022-05-18T05:07:33.7024707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd39av23z/_remote_module_non_scriptable.py
2022-05-18T05:07:33.7921028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2hsh4t5t
2022-05-18T05:07:33.7922461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2hsh4t5t/_remote_module_non_scriptable.py
2022-05-18T05:07:34.1006331Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:07:34.4077293Z ok (4.895s)
2022-05-18T05:07:34.4077684Z 
2022-05-18T05:07:34.4078441Z ----------------------------------------------------------------------
2022-05-18T05:07:34.4079004Z Ran 1 test in 4.895s
2022-05-18T05:07:34.4079173Z 
2022-05-18T05:07:34.4079257Z OK
2022-05-18T05:07:34.4079394Z 
2022-05-18T05:07:34.4079529Z Generating XML reports...
2022-05-18T05:07:34.4137848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050729.xml
2022-05-18T05:07:35.8434468Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:35.8450222Z 
2022-05-18T05:07:35.8450467Z Running tests...
2022-05-18T05:07:35.8450887Z ----------------------------------------------------------------------
2022-05-18T05:07:37.5127417Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:37.5498480Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54885
2022-05-18T05:07:37.5609467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54886
2022-05-18T05:07:38.7407746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:38.7408863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:38.7410647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:38.7411805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:38.7518004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:38.8424586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:40.0615376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvdsad70j
2022-05-18T05:07:40.0616561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvdsad70j/_remote_module_non_scriptable.py
2022-05-18T05:07:40.1579613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2vsxdez
2022-05-18T05:07:40.1581057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2vsxdez/_remote_module_non_scriptable.py
2022-05-18T05:07:40.4667912Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:07:40.4669551Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:07:40.8691825Z ok (5.024s)
2022-05-18T05:07:40.8692052Z 
2022-05-18T05:07:40.8692694Z ----------------------------------------------------------------------
2022-05-18T05:07:40.8693057Z Ran 1 test in 5.024s
2022-05-18T05:07:40.8693227Z 
2022-05-18T05:07:40.8693323Z OK
2022-05-18T05:07:40.8693472Z 
2022-05-18T05:07:40.8693607Z Generating XML reports...
2022-05-18T05:07:40.8750710Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050735.xml
2022-05-18T05:07:42.3092601Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:42.3108071Z 
2022-05-18T05:07:42.3108318Z Running tests...
2022-05-18T05:07:42.3108740Z ----------------------------------------------------------------------
2022-05-18T05:07:43.9697031Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:44.0073202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55006
2022-05-18T05:07:44.0185106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55007
2022-05-18T05:07:45.2231832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:45.2232402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:45.2233398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:45.2234127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:45.2240583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:45.2241362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:45.2328709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgv8rdfb
2022-05-18T05:07:45.2331169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgv8rdfb/_remote_module_non_scriptable.py
2022-05-18T05:07:45.2333573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3k9vgbd7
2022-05-18T05:07:45.2336414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3k9vgbd7/_remote_module_non_scriptable.py
2022-05-18T05:07:45.2484178Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:07:45.2486059Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:07:45.2490263Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T05:07:45.2491302Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-05-18T05:07:45.2492720Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T05:07:45.2493707Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-05-18T05:07:45.2495148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:45.2495627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:45.2498906Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:07:45.2500525Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:07:45.2505199Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:07:45.2506695Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:07:45.2511985Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:07:45.2513459Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:07:45.2517666Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:07:45.2519149Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:07:45.2523746Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:07:45.2525238Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:07:45.4233197Z ok (3.112s)
2022-05-18T05:07:45.4233380Z 
2022-05-18T05:07:45.4234023Z ----------------------------------------------------------------------
2022-05-18T05:07:45.4234417Z Ran 1 test in 3.113s
2022-05-18T05:07:45.4234566Z 
2022-05-18T05:07:45.4234670Z OK
2022-05-18T05:07:45.4234809Z 
2022-05-18T05:07:45.4234940Z Generating XML reports...
2022-05-18T05:07:45.4290968Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050742.xml
2022-05-18T05:07:46.8431769Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:46.8446432Z 
2022-05-18T05:07:46.8446862Z Running tests...
2022-05-18T05:07:46.8447346Z ----------------------------------------------------------------------
2022-05-18T05:07:48.4899564Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:48.5268656Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55125
2022-05-18T05:07:48.5379089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55126
2022-05-18T05:07:49.7057179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:49.7057768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:49.7058558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:49.7059258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:49.7168051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:49.8072152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:51.0388708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1iaew8l
2022-05-18T05:07:51.0389315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1iaew8l/_remote_module_non_scriptable.py
2022-05-18T05:07:51.0809215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5mzko9cm
2022-05-18T05:07:51.0810817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5mzko9cm/_remote_module_non_scriptable.py
2022-05-18T05:07:51.3855331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:51.3855894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:51.7459954Z ok (4.901s)
2022-05-18T05:07:51.7460201Z 
2022-05-18T05:07:51.7460589Z ----------------------------------------------------------------------
2022-05-18T05:07:51.7460929Z Ran 1 test in 4.901s
2022-05-18T05:07:51.7461097Z 
2022-05-18T05:07:51.7461219Z OK
2022-05-18T05:07:51.7461336Z 
2022-05-18T05:07:51.7461473Z Generating XML reports...
2022-05-18T05:07:51.7518285Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050746.xml
2022-05-18T05:07:53.1685743Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:53.1700231Z 
2022-05-18T05:07:53.1700675Z Running tests...
2022-05-18T05:07:53.1701184Z ----------------------------------------------------------------------
2022-05-18T05:07:54.7837154Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:54.8205457Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55246
2022-05-18T05:07:54.8317894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55247
2022-05-18T05:07:56.0372382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:56.0373230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:56.0374065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:56.0374985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:56.0382699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:56.0383292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:57.3962972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8v4hjtj
2022-05-18T05:07:57.3963594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8v4hjtj/_remote_module_non_scriptable.py
2022-05-18T05:07:57.4158819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzi5w4x0
2022-05-18T05:07:57.4161407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzi5w4x0/_remote_module_non_scriptable.py
2022-05-18T05:07:57.4377243Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T05:07:57.4379037Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T05:07:57.4381168Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T05:07:57.4382899Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T05:07:58.0399065Z ok (4.870s)
2022-05-18T05:07:58.0399270Z 
2022-05-18T05:07:58.0399946Z ----------------------------------------------------------------------
2022-05-18T05:07:58.0400267Z Ran 1 test in 4.870s
2022-05-18T05:07:58.0400436Z 
2022-05-18T05:07:58.0400537Z OK
2022-05-18T05:07:58.0400674Z 
2022-05-18T05:07:58.0400808Z Generating XML reports...
2022-05-18T05:07:58.0470006Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050753.xml
2022-05-18T05:07:59.4850809Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:59.4865781Z 
2022-05-18T05:07:59.4866230Z Running tests...
2022-05-18T05:07:59.4866656Z ----------------------------------------------------------------------
2022-05-18T05:08:01.1284344Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:01.1662922Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55397
2022-05-18T05:08:01.1774880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55398
2022-05-18T05:08:02.3387091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:02.3387938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:02.3388798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:02.3389497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:02.3396214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:02.3396985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:03.6569143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmcz_g2ou
2022-05-18T05:08:03.6570024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmcz_g2ou/_remote_module_non_scriptable.py
2022-05-18T05:08:03.6697678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw84mwjf1
2022-05-18T05:08:03.6700503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw84mwjf1/_remote_module_non_scriptable.py
2022-05-18T05:08:03.9774309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:03.9774887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:04.0001888Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpmcz_g2ou'>
2022-05-18T05:08:04.0002398Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:08:04.0003194Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpw84mwjf1'>
2022-05-18T05:08:04.0003672Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:08:04.2853061Z ok (4.798s)
2022-05-18T05:08:04.2853278Z 
2022-05-18T05:08:04.2853645Z ----------------------------------------------------------------------
2022-05-18T05:08:04.2853967Z Ran 1 test in 4.799s
2022-05-18T05:08:04.2854137Z 
2022-05-18T05:08:04.2854231Z OK
2022-05-18T05:08:04.2854368Z 
2022-05-18T05:08:04.2854502Z Generating XML reports...
2022-05-18T05:08:04.2911084Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050759.xml
2022-05-18T05:08:05.7152376Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:05.7166793Z 
2022-05-18T05:08:05.7167227Z Running tests...
2022-05-18T05:08:05.7167743Z ----------------------------------------------------------------------
2022-05-18T05:08:07.3458792Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:07.3577488Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.641s)
2022-05-18T05:08:07.3578334Z 
2022-05-18T05:08:07.3578631Z ----------------------------------------------------------------------
2022-05-18T05:08:07.3578946Z Ran 1 test in 1.641s
2022-05-18T05:08:07.3579113Z 
2022-05-18T05:08:07.3579221Z OK (skipped=1)
2022-05-18T05:08:07.3579376Z 
2022-05-18T05:08:07.3579503Z Generating XML reports...
2022-05-18T05:08:07.3617763Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050805.xml
2022-05-18T05:08:08.7382682Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:08.7397015Z 
2022-05-18T05:08:08.7397663Z Running tests...
2022-05-18T05:08:08.7398112Z ----------------------------------------------------------------------
2022-05-18T05:08:10.3834582Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:10.4206383Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55554
2022-05-18T05:08:10.4318465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55555
2022-05-18T05:08:11.6652722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:11.6653447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:11.6657883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:11.6659234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:11.6763211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:11.7666495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:11.7776124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:08:11.7776642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:08:11.7777334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:08:11.7778038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:08:13.1310539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpksmamb12
2022-05-18T05:08:13.1311416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpksmamb12/_remote_module_non_scriptable.py
2022-05-18T05:08:13.1319893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpio8hunaq
2022-05-18T05:08:13.1322718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpio8hunaq/_remote_module_non_scriptable.py
2022-05-18T05:08:13.4437610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:13.4438193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:13.4453959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:13.4454688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:14.0408930Z ok (5.301s)
2022-05-18T05:08:14.0409257Z 
2022-05-18T05:08:14.0410187Z ----------------------------------------------------------------------
2022-05-18T05:08:14.0410752Z Ran 1 test in 5.301s
2022-05-18T05:08:14.0411121Z 
2022-05-18T05:08:14.0411294Z OK
2022-05-18T05:08:14.0412579Z 
2022-05-18T05:08:14.0412862Z Generating XML reports...
2022-05-18T05:08:14.0466364Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050808.xml
2022-05-18T05:08:15.4876929Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:15.4892058Z 
2022-05-18T05:08:15.4892237Z Running tests...
2022-05-18T05:08:15.4892987Z ----------------------------------------------------------------------
2022-05-18T05:08:17.1328167Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:17.1697138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55681
2022-05-18T05:08:17.1807417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55682
2022-05-18T05:08:18.3837612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:18.3838177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:18.3838964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:18.3839665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:18.3946258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:18.3948201Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T05:08:18.4852552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:18.4853773Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T05:08:19.6761792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpex7tncw8
2022-05-18T05:08:19.6762969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpex7tncw8/_remote_module_non_scriptable.py
2022-05-18T05:08:19.7809384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8l5fx3h
2022-05-18T05:08:19.7810399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8l5fx3h/_remote_module_non_scriptable.py
2022-05-18T05:08:20.0903395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:20.0903957Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:20.0920748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:20.0921611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:20.1174506Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T05:08:20.1175120Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T05:08:20.3759061Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T05:08:20.3760164Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T05:08:20.3838999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:20.3839609Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:20.3855829Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:20.3856566Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:20.4100620Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T05:08:20.4101444Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T05:08:20.5671942Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T05:08:20.5673014Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T05:08:20.5749717Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:20.5750304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:20.5766480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:20.5767210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:21.1899321Z ok (5.700s)
2022-05-18T05:08:21.1899642Z 
2022-05-18T05:08:21.1900154Z ----------------------------------------------------------------------
2022-05-18T05:08:21.1900498Z Ran 1 test in 5.701s
2022-05-18T05:08:21.1900665Z 
2022-05-18T05:08:21.1900756Z OK
2022-05-18T05:08:21.1900889Z 
2022-05-18T05:08:21.1901020Z Generating XML reports...
2022-05-18T05:08:21.1957730Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050815.xml
2022-05-18T05:08:22.6224235Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:22.6238844Z 
2022-05-18T05:08:22.6239001Z Running tests...
2022-05-18T05:08:22.6239862Z ----------------------------------------------------------------------
2022-05-18T05:08:24.2835575Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:24.3207430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55802
2022-05-18T05:08:24.3317174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55803
2022-05-18T05:08:25.5146306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:25.5146951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:25.5147743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:25.5148429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:25.5254989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:25.5257805Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T05:08:25.6161262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:25.6162482Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T05:08:26.8241354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79kc8iqn
2022-05-18T05:08:26.8241970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79kc8iqn/_remote_module_non_scriptable.py
2022-05-18T05:08:26.9182962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05fx5mde
2022-05-18T05:08:26.9184283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05fx5mde/_remote_module_non_scriptable.py
2022-05-18T05:08:27.2281728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:27.2282282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:27.2300094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:27.2300574Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:27.2306600Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T05:08:27.2307182Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T05:08:27.2340286Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T05:08:27.2340898Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T05:08:27.2342072Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T05:08:27.2342736Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T05:08:27.2343557Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T05:08:27.2344210Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T05:08:27.5785943Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T05:08:27.5787046Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T05:08:27.5868470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:27.5868949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:27.5885951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:27.5886468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:27.5891645Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T05:08:27.5892202Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T05:08:27.5923994Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T05:08:27.5924605Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T05:08:27.5925431Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T05:08:27.5926075Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T05:08:28.2411296Z ok (5.617s)
2022-05-18T05:08:28.2411504Z 
2022-05-18T05:08:28.2412143Z ----------------------------------------------------------------------
2022-05-18T05:08:28.2412486Z Ran 1 test in 5.617s
2022-05-18T05:08:28.2412651Z 
2022-05-18T05:08:28.2412747Z OK
2022-05-18T05:08:28.2412883Z 
2022-05-18T05:08:28.2413014Z Generating XML reports...
2022-05-18T05:08:28.2470491Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050822.xml
2022-05-18T05:08:29.6714405Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:29.6729439Z 
2022-05-18T05:08:29.6729858Z Running tests...
2022-05-18T05:08:29.6730688Z ----------------------------------------------------------------------
2022-05-18T05:08:31.3018147Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:31.3391284Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55923
2022-05-18T05:08:31.3500655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55924
2022-05-18T05:08:32.5056186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:32.5056741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:32.5057538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:32.5058240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:32.5165301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:32.6069953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:33.8929243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp91abigjc
2022-05-18T05:08:33.8930738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp91abigjc/_remote_module_non_scriptable.py
2022-05-18T05:08:33.9235146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ajqn2hb
2022-05-18T05:08:33.9236564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ajqn2hb/_remote_module_non_scriptable.py
2022-05-18T05:08:34.2794270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:34.2794837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:34.3473112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:34.3473932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:34.7585405Z ok (5.085s)
2022-05-18T05:08:34.7585613Z 
2022-05-18T05:08:34.7586020Z ----------------------------------------------------------------------
2022-05-18T05:08:34.7586358Z Ran 1 test in 5.086s
2022-05-18T05:08:34.7586522Z 
2022-05-18T05:08:34.7586615Z OK
2022-05-18T05:08:34.7586750Z 
2022-05-18T05:08:34.7586881Z Generating XML reports...
2022-05-18T05:08:34.7643913Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050829.xml
2022-05-18T05:08:36.1960640Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:36.1976105Z 
2022-05-18T05:08:36.1976501Z Running tests...
2022-05-18T05:08:36.1977023Z ----------------------------------------------------------------------
2022-05-18T05:08:37.8101622Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:37.8475942Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56074
2022-05-18T05:08:37.8589544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56075
2022-05-18T05:08:39.0109530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:39.0110077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:39.0110883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:39.0111574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:39.0118717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:39.0119566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:40.3785123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpld0xj1ch
2022-05-18T05:08:40.3785743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpld0xj1ch/_remote_module_non_scriptable.py
2022-05-18T05:08:40.3996048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptk80g8ne
2022-05-18T05:08:40.3997845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptk80g8ne/_remote_module_non_scriptable.py
2022-05-18T05:08:40.7540208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:40.7540761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:40.8199761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:40.8200308Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:41.1673223Z ok (4.969s)
2022-05-18T05:08:41.1673492Z 
2022-05-18T05:08:41.1673868Z ----------------------------------------------------------------------
2022-05-18T05:08:41.1674205Z Ran 1 test in 4.970s
2022-05-18T05:08:41.1674379Z 
2022-05-18T05:08:41.1674482Z OK
2022-05-18T05:08:41.1674599Z 
2022-05-18T05:08:41.1674730Z Generating XML reports...
2022-05-18T05:08:41.1731574Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050836.xml
2022-05-18T05:08:42.6203587Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:42.6219323Z 
2022-05-18T05:08:42.6219610Z Running tests...
2022-05-18T05:08:42.6220045Z ----------------------------------------------------------------------
2022-05-18T05:08:44.2787421Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:44.3167259Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56225
2022-05-18T05:08:44.3280927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56226
2022-05-18T05:08:45.5205317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:45.5205871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:45.5206657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:45.5207355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:45.5215323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:45.5215820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:46.8858882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8j2ersb
2022-05-18T05:08:46.8859884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8j2ersb/_remote_module_non_scriptable.py
2022-05-18T05:08:46.8883886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnn3o7lo3
2022-05-18T05:08:46.8886628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnn3o7lo3/_remote_module_non_scriptable.py
2022-05-18T05:08:47.2425419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:47.2425976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:47.3094936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:47.3095480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:47.7367060Z ok (5.114s)
2022-05-18T05:08:47.7367283Z 
2022-05-18T05:08:47.7367694Z ----------------------------------------------------------------------
2022-05-18T05:08:47.7368035Z Ran 1 test in 5.115s
2022-05-18T05:08:47.7368200Z 
2022-05-18T05:08:47.7368280Z OK
2022-05-18T05:08:47.7368426Z 
2022-05-18T05:08:47.7368558Z Generating XML reports...
2022-05-18T05:08:47.7425135Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050842.xml
2022-05-18T05:08:49.1835769Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:49.1851867Z 
2022-05-18T05:08:49.1852196Z Running tests...
2022-05-18T05:08:49.1852651Z ----------------------------------------------------------------------
2022-05-18T05:08:50.8444686Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:50.8822184Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56376
2022-05-18T05:08:50.8933361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56377
2022-05-18T05:08:52.0894985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:52.0895570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:52.0896339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:52.0897038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:52.0903836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:52.0904373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:53.4258024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8ngjn5d
2022-05-18T05:08:53.4259358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8ngjn5d/_remote_module_non_scriptable.py
2022-05-18T05:08:53.4647591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6qkj58s6
2022-05-18T05:08:53.4648948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6qkj58s6/_remote_module_non_scriptable.py
2022-05-18T05:08:53.8201181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:53.8201727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:53.8941112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:53.8941630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:54.3017479Z ok (5.116s)
2022-05-18T05:08:54.3017730Z 
2022-05-18T05:08:54.3018140Z ----------------------------------------------------------------------
2022-05-18T05:08:54.3018481Z Ran 1 test in 5.116s
2022-05-18T05:08:54.3018650Z 
2022-05-18T05:08:54.3018725Z OK
2022-05-18T05:08:54.3018859Z 
2022-05-18T05:08:54.3018998Z Generating XML reports...
2022-05-18T05:08:54.3084678Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050849.xml
2022-05-18T05:08:55.7541663Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:55.7556571Z 
2022-05-18T05:08:55.7556838Z Running tests...
2022-05-18T05:08:55.7557271Z ----------------------------------------------------------------------
2022-05-18T05:08:57.4070550Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:57.4445786Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56527
2022-05-18T05:08:57.4559528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56528
2022-05-18T05:08:58.6340455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:58.6341007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:58.6341795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:58.6342495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:58.6349243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:58.6349708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:59.9736781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnmhzu8x
2022-05-18T05:08:59.9737942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnmhzu8x/_remote_module_non_scriptable.py
2022-05-18T05:08:59.9873287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpulzljp_w
2022-05-18T05:08:59.9875717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpulzljp_w/_remote_module_non_scriptable.py
2022-05-18T05:09:00.3530121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:00.3530867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:00.4353976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:00.4354524Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:00.8643231Z ok (5.108s)
2022-05-18T05:09:00.8643471Z 
2022-05-18T05:09:00.8643874Z ----------------------------------------------------------------------
2022-05-18T05:09:00.8644453Z Ran 1 test in 5.109s
2022-05-18T05:09:00.8644641Z 
2022-05-18T05:09:00.8644735Z OK
2022-05-18T05:09:00.8644869Z 
2022-05-18T05:09:00.8645003Z Generating XML reports...
2022-05-18T05:09:00.8702707Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050855.xml
2022-05-18T05:09:02.2927142Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:02.2943276Z 
2022-05-18T05:09:02.2943636Z Running tests...
2022-05-18T05:09:02.2944547Z ----------------------------------------------------------------------
2022-05-18T05:09:03.9119268Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:03.9485511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56678
2022-05-18T05:09:03.9599274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56679
2022-05-18T05:09:05.1335895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:05.1336910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:05.1338592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:05.1339931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:05.1444478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:05.2352900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:06.4434857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyu73gunq
2022-05-18T05:09:06.4436079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyu73gunq/_remote_module_non_scriptable.py
2022-05-18T05:09:06.5381376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44xrfvwl
2022-05-18T05:09:06.5382573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44xrfvwl/_remote_module_non_scriptable.py
2022-05-18T05:09:06.9007753Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:06.9008343Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:06.9711516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:06.9712064Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:07.3681971Z ok (5.074s)
2022-05-18T05:09:07.3682199Z 
2022-05-18T05:09:07.3682586Z ----------------------------------------------------------------------
2022-05-18T05:09:07.3682920Z Ran 1 test in 5.074s
2022-05-18T05:09:07.3683103Z 
2022-05-18T05:09:07.3683179Z OK
2022-05-18T05:09:07.3683317Z 
2022-05-18T05:09:07.3683460Z Generating XML reports...
2022-05-18T05:09:07.3740607Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050902.xml
2022-05-18T05:09:08.7740458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:08.7755511Z 
2022-05-18T05:09:08.7755947Z Running tests...
2022-05-18T05:09:08.7756428Z ----------------------------------------------------------------------
2022-05-18T05:09:10.4238745Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:10.4612170Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56829
2022-05-18T05:09:10.4722318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56830
2022-05-18T05:09:11.6370572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:11.6371178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:11.6371995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:11.6372691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:11.6479092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:11.7386357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:12.9483278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9g71l6k
2022-05-18T05:09:12.9483918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9g71l6k/_remote_module_non_scriptable.py
2022-05-18T05:09:13.0691025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ukrrk5b
2022-05-18T05:09:13.0692308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ukrrk5b/_remote_module_non_scriptable.py
2022-05-18T05:09:13.4310577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:13.4311079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:13.4986617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:13.4987136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:13.8805693Z ok (5.105s)
2022-05-18T05:09:13.8805987Z 
2022-05-18T05:09:13.8806741Z ----------------------------------------------------------------------
2022-05-18T05:09:13.8807394Z Ran 1 test in 5.105s
2022-05-18T05:09:13.8807560Z 
2022-05-18T05:09:13.8807656Z OK
2022-05-18T05:09:13.8807807Z 
2022-05-18T05:09:13.8807930Z Generating XML reports...
2022-05-18T05:09:13.8864369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050908.xml
2022-05-18T05:09:15.3292860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:15.3307570Z 
2022-05-18T05:09:15.3308010Z Running tests...
2022-05-18T05:09:15.3308955Z ----------------------------------------------------------------------
2022-05-18T05:09:16.9815206Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:17.0194297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56980
2022-05-18T05:09:17.0306265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56981
2022-05-18T05:09:18.2606477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:18.2607064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:18.2607871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:18.2608570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:18.2615857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:18.2616876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:19.6277846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vvpcuj0
2022-05-18T05:09:19.6279139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vvpcuj0/_remote_module_non_scriptable.py
2022-05-18T05:09:19.6377985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeseiaedd
2022-05-18T05:09:19.6380474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeseiaedd/_remote_module_non_scriptable.py
2022-05-18T05:09:19.9903134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:19.9903691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:20.0572233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:20.0572772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:20.4392485Z ok (5.108s)
2022-05-18T05:09:20.4392667Z 
2022-05-18T05:09:20.4393048Z ----------------------------------------------------------------------
2022-05-18T05:09:20.4393359Z Ran 1 test in 5.108s
2022-05-18T05:09:20.4393527Z 
2022-05-18T05:09:20.4393623Z OK
2022-05-18T05:09:20.4393758Z 
2022-05-18T05:09:20.4393897Z Generating XML reports...
2022-05-18T05:09:20.4450965Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050915.xml
2022-05-18T05:09:21.8749189Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:21.8763990Z 
2022-05-18T05:09:21.8764534Z Running tests...
2022-05-18T05:09:21.8765022Z ----------------------------------------------------------------------
2022-05-18T05:09:23.4969116Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:23.5337413Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57131
2022-05-18T05:09:23.5450457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57132
2022-05-18T05:09:24.7719053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:24.7719594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:24.7720386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:24.7721100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:24.7830133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:24.8734717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:26.0944508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy401dmht
2022-05-18T05:09:26.0945926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy401dmht/_remote_module_non_scriptable.py
2022-05-18T05:09:26.2143954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6fwx7pu
2022-05-18T05:09:26.2144759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6fwx7pu/_remote_module_non_scriptable.py
2022-05-18T05:09:26.5882901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:26.5883427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:26.6663644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:26.6664185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:27.0535352Z ok (5.177s)
2022-05-18T05:09:27.0535623Z 
2022-05-18T05:09:27.0536208Z ----------------------------------------------------------------------
2022-05-18T05:09:27.0536557Z Ran 1 test in 5.177s
2022-05-18T05:09:27.0536723Z 
2022-05-18T05:09:27.0536797Z OK
2022-05-18T05:09:27.0536939Z 
2022-05-18T05:09:27.0537070Z Generating XML reports...
2022-05-18T05:09:27.0595649Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050921.xml
2022-05-18T05:09:28.5018677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:28.5037865Z 
2022-05-18T05:09:28.5038261Z Running tests...
2022-05-18T05:09:28.5038768Z ----------------------------------------------------------------------
2022-05-18T05:09:30.1386746Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:30.1764536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57282
2022-05-18T05:09:30.1875089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57283
2022-05-18T05:09:31.3771972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:31.3772513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:31.3773310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:31.3774249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:31.3883366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:31.4787137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:32.6921410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0c55zi4
2022-05-18T05:09:32.6922529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0c55zi4/_remote_module_non_scriptable.py
2022-05-18T05:09:32.8031037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6g5l0xqk
2022-05-18T05:09:32.8032254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6g5l0xqk/_remote_module_non_scriptable.py
2022-05-18T05:09:33.1703004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:33.1703612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:33.2359905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:33.2360463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:33.5961716Z ok (5.092s)
2022-05-18T05:09:33.5961932Z 
2022-05-18T05:09:33.5962326Z ----------------------------------------------------------------------
2022-05-18T05:09:33.5962650Z Ran 1 test in 5.092s
2022-05-18T05:09:33.5962825Z 
2022-05-18T05:09:33.5962921Z OK
2022-05-18T05:09:33.5963059Z 
2022-05-18T05:09:33.5963192Z Generating XML reports...
2022-05-18T05:09:33.6020024Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050928.xml
2022-05-18T05:09:35.0450569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:35.0465833Z 
2022-05-18T05:09:35.0466259Z Running tests...
2022-05-18T05:09:35.0466739Z ----------------------------------------------------------------------
2022-05-18T05:09:36.6785844Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:36.7164346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57433
2022-05-18T05:09:36.7273824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57434
2022-05-18T05:09:37.9510702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:37.9511621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:37.9512429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:37.9513134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:37.9621649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:38.0526221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:39.3041571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3lih07u
2022-05-18T05:09:39.3043093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3lih07u/_remote_module_non_scriptable.py
2022-05-18T05:09:39.3861752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthg6nixq
2022-05-18T05:09:39.3862676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthg6nixq/_remote_module_non_scriptable.py
2022-05-18T05:09:39.7374644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:39.7375182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:39.7981163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:39.7981664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:40.1357924Z ok (5.089s)
2022-05-18T05:09:40.1358152Z 
2022-05-18T05:09:40.1358546Z ----------------------------------------------------------------------
2022-05-18T05:09:40.1358867Z Ran 1 test in 5.089s
2022-05-18T05:09:40.1359036Z 
2022-05-18T05:09:40.1359127Z OK
2022-05-18T05:09:40.1359265Z 
2022-05-18T05:09:40.1359400Z Generating XML reports...
2022-05-18T05:09:40.1417029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050935.xml
2022-05-18T05:09:41.5835914Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:41.5850991Z 
2022-05-18T05:09:41.5851261Z Running tests...
2022-05-18T05:09:41.5851722Z ----------------------------------------------------------------------
2022-05-18T05:09:43.2329387Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:43.2705462Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57584
2022-05-18T05:09:43.2815888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57585
2022-05-18T05:09:44.4401995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:44.4402563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:44.4403347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:44.4404050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:44.4510495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:44.5417577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:45.7561269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsup80wuj
2022-05-18T05:09:45.7563379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsup80wuj/_remote_module_non_scriptable.py
2022-05-18T05:09:45.8821606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv935kkgy
2022-05-18T05:09:45.8822430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv935kkgy/_remote_module_non_scriptable.py
2022-05-18T05:09:46.2332575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:46.2333118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:46.2982672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:46.2983189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:46.6899942Z ok (5.105s)
2022-05-18T05:09:46.6900164Z 
2022-05-18T05:09:46.6900543Z ----------------------------------------------------------------------
2022-05-18T05:09:46.6900877Z Ran 1 test in 5.105s
2022-05-18T05:09:46.6901040Z 
2022-05-18T05:09:46.6901115Z OK
2022-05-18T05:09:46.6901256Z 
2022-05-18T05:09:46.6901388Z Generating XML reports...
2022-05-18T05:09:46.6958287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050941.xml
2022-05-18T05:09:48.1112076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:48.1126827Z 
2022-05-18T05:09:48.1126970Z Running tests...
2022-05-18T05:09:48.1127705Z ----------------------------------------------------------------------
2022-05-18T05:09:49.7172362Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:49.7287560Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.616s)
2022-05-18T05:09:49.7288109Z 
2022-05-18T05:09:49.7288399Z ----------------------------------------------------------------------
2022-05-18T05:09:49.7288731Z Ran 1 test in 1.616s
2022-05-18T05:09:49.7288877Z 
2022-05-18T05:09:49.7288986Z OK (skipped=1)
2022-05-18T05:09:49.7289140Z 
2022-05-18T05:09:49.7289286Z Generating XML reports...
2022-05-18T05:09:49.7327010Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050948.xml
2022-05-18T05:09:51.1149082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:51.1164333Z 
2022-05-18T05:09:51.1164563Z Running tests...
2022-05-18T05:09:51.1165000Z ----------------------------------------------------------------------
2022-05-18T05:09:52.7531253Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:52.7900799Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57771
2022-05-18T05:09:52.8008575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57772
2022-05-18T05:09:54.0170130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:54.0170735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:54.0171531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:54.0172239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:54.0179259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:54.0179762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:55.3311581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_piqugh
2022-05-18T05:09:55.3312184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48m4oqq5
2022-05-18T05:09:55.3312730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_piqugh/_remote_module_non_scriptable.py
2022-05-18T05:09:55.3313486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48m4oqq5/_remote_module_non_scriptable.py
2022-05-18T05:09:56.3096491Z ok (5.193s)
2022-05-18T05:09:56.3096715Z 
2022-05-18T05:09:56.3097104Z ----------------------------------------------------------------------
2022-05-18T05:09:56.3097449Z Ran 1 test in 5.193s
2022-05-18T05:09:56.3097620Z 
2022-05-18T05:09:56.3097725Z OK
2022-05-18T05:09:56.3097864Z 
2022-05-18T05:09:56.3098001Z Generating XML reports...
2022-05-18T05:09:56.3156732Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050951.xml
2022-05-18T05:09:57.7523646Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:57.7538860Z 
2022-05-18T05:09:57.7539325Z Running tests...
2022-05-18T05:09:57.7539835Z ----------------------------------------------------------------------
2022-05-18T05:09:59.3978791Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:59.4348713Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57888
2022-05-18T05:09:59.4459105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57889
2022-05-18T05:10:00.6248041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:00.6248611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:00.6249387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:00.6250505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:00.6359318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:00.7264478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:02.1975162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8pj6iioz
2022-05-18T05:10:02.1976251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8pj6iioz/_remote_module_non_scriptable.py
2022-05-18T05:10:02.3113326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftw7g03_
2022-05-18T05:10:02.3114168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftw7g03_/_remote_module_non_scriptable.py
2022-05-18T05:10:02.3304243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:02.3304775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:02.3404348Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpftw7g03_'>
2022-05-18T05:10:02.3404829Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:10:02.3405433Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp8pj6iioz'>
2022-05-18T05:10:02.3405868Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:10:02.6540575Z ok (4.900s)
2022-05-18T05:10:02.6540766Z 
2022-05-18T05:10:02.6541149Z ----------------------------------------------------------------------
2022-05-18T05:10:02.6541491Z Ran 1 test in 4.900s
2022-05-18T05:10:02.6541656Z 
2022-05-18T05:10:02.6541755Z OK
2022-05-18T05:10:02.6541891Z 
2022-05-18T05:10:02.6542009Z Generating XML reports...
2022-05-18T05:10:02.6599147Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050957.xml
2022-05-18T05:10:04.0997744Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:04.1013086Z 
2022-05-18T05:10:04.1013331Z Running tests...
2022-05-18T05:10:04.1013762Z ----------------------------------------------------------------------
2022-05-18T05:10:05.7535145Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:05.7913544Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58009
2022-05-18T05:10:05.8024682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58010
2022-05-18T05:10:06.9789886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:06.9790707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:06.9791521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:06.9792219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:06.9798398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:06.9799111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:06.9893798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2g8q_5zo
2022-05-18T05:10:06.9894583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpop327hz7
2022-05-18T05:10:06.9896129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2g8q_5zo/_remote_module_non_scriptable.py
2022-05-18T05:10:06.9896663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpop327hz7/_remote_module_non_scriptable.py
2022-05-18T05:10:07.0068378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:07.0068895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:07.3074105Z ok (3.206s)
2022-05-18T05:10:07.3074320Z 
2022-05-18T05:10:07.3074836Z ----------------------------------------------------------------------
2022-05-18T05:10:07.3075437Z Ran 1 test in 3.206s
2022-05-18T05:10:07.3075610Z 
2022-05-18T05:10:07.3075711Z OK
2022-05-18T05:10:07.3075846Z 
2022-05-18T05:10:07.3075982Z Generating XML reports...
2022-05-18T05:10:07.3132208Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051004.xml
2022-05-18T05:10:08.7576702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:08.7591747Z 
2022-05-18T05:10:08.7591999Z Running tests...
2022-05-18T05:10:08.7592419Z ----------------------------------------------------------------------
2022-05-18T05:10:10.4402821Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:10.4781539Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58158
2022-05-18T05:10:10.4895625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58159
2022-05-18T05:10:11.6985779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:11.6986381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:11.6987186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:11.6989294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:11.7096652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:11.8000907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:13.0015138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojgtkuov
2022-05-18T05:10:13.0015762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojgtkuov/_remote_module_non_scriptable.py
2022-05-18T05:10:13.0966001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jg3owcb
2022-05-18T05:10:13.0967009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jg3owcb/_remote_module_non_scriptable.py
2022-05-18T05:10:13.4068872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:13.4069425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:13.7980789Z ok (5.039s)
2022-05-18T05:10:13.7981004Z 
2022-05-18T05:10:13.7981373Z ----------------------------------------------------------------------
2022-05-18T05:10:13.7981710Z Ran 1 test in 5.039s
2022-05-18T05:10:13.7981882Z 
2022-05-18T05:10:13.7981979Z OK
2022-05-18T05:10:13.7982112Z 
2022-05-18T05:10:13.7982241Z Generating XML reports...
2022-05-18T05:10:13.8039039Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051008.xml
2022-05-18T05:10:15.2380778Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:15.2395308Z 
2022-05-18T05:10:15.2395568Z Running tests...
2022-05-18T05:10:15.2396341Z ----------------------------------------------------------------------
2022-05-18T05:10:16.8939613Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:16.9317563Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58279
2022-05-18T05:10:16.9429169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58280
2022-05-18T05:10:18.1319909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:18.1320745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:18.1321551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:18.1322254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:18.1430591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:18.2331125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:18.2445766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:10:18.2446534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:10:18.2447240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:10:18.2447937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:10:18.2757044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:10:18.2757900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:10:18.2758683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:10:18.2759371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:10:19.6154853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp575pa5dq
2022-05-18T05:10:19.6156307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp575pa5dq/_remote_module_non_scriptable.py
2022-05-18T05:10:19.6295774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_xvnque
2022-05-18T05:10:19.6298822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_xvnque/_remote_module_non_scriptable.py
2022-05-18T05:10:19.9516457Z ok (4.712s)
2022-05-18T05:10:19.9516678Z 
2022-05-18T05:10:19.9517056Z ----------------------------------------------------------------------
2022-05-18T05:10:19.9517399Z Ran 1 test in 4.712s
2022-05-18T05:10:19.9517572Z 
2022-05-18T05:10:19.9517666Z OK
2022-05-18T05:10:19.9517800Z 
2022-05-18T05:10:19.9517941Z Generating XML reports...
2022-05-18T05:10:19.9573838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051015.xml
2022-05-18T05:10:21.3763300Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:21.3777466Z 
2022-05-18T05:10:21.3777871Z Running tests...
2022-05-18T05:10:21.3778356Z ----------------------------------------------------------------------
2022-05-18T05:10:23.0118200Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:23.0491441Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58408
2022-05-18T05:10:23.0601227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58409
2022-05-18T05:10:24.2696190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:24.2696977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:24.2697752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:24.2698452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:24.2705237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:24.2705733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:24.2914521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:10:24.2915024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:10:24.2915713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:10:24.2916405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:10:24.3123787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:10:24.3124651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:10:24.3125328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:10:24.3126023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:10:25.6410661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyb1mqba
2022-05-18T05:10:25.6411726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyb1mqba/_remote_module_non_scriptable.py
2022-05-18T05:10:25.6577189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphlbc28mx
2022-05-18T05:10:25.6579822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphlbc28mx/_remote_module_non_scriptable.py
2022-05-18T05:10:35.9839389Z ok (14.606s)
2022-05-18T05:10:35.9839576Z 
2022-05-18T05:10:35.9839974Z ----------------------------------------------------------------------
2022-05-18T05:10:35.9840316Z Ran 1 test in 14.606s
2022-05-18T05:10:35.9840487Z 
2022-05-18T05:10:35.9840579Z OK
2022-05-18T05:10:35.9840714Z 
2022-05-18T05:10:35.9840834Z Generating XML reports...
2022-05-18T05:10:35.9897722Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051021.xml
2022-05-18T05:10:37.3984509Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:37.3998653Z 
2022-05-18T05:10:37.3998774Z Running tests...
2022-05-18T05:10:37.3999544Z ----------------------------------------------------------------------
2022-05-18T05:10:39.0251173Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:39.0625252Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58537
2022-05-18T05:10:39.0733521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58538
2022-05-18T05:10:40.2534879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:40.2536015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:40.2536811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:40.2537511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:40.2644996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:40.3549802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:41.5634223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9b52dwck
2022-05-18T05:10:41.5634835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9b52dwck/_remote_module_non_scriptable.py
2022-05-18T05:10:41.6539716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyi4ke4w
2022-05-18T05:10:41.6540668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyi4ke4w/_remote_module_non_scriptable.py
2022-05-18T05:10:42.5817768Z ok (5.182s)
2022-05-18T05:10:42.5818016Z 
2022-05-18T05:10:42.5818413Z ----------------------------------------------------------------------
2022-05-18T05:10:42.5818762Z Ran 1 test in 5.182s
2022-05-18T05:10:42.5818943Z 
2022-05-18T05:10:42.5820719Z OK
2022-05-18T05:10:42.5820918Z 
2022-05-18T05:10:42.5821096Z Generating XML reports...
2022-05-18T05:10:42.5875614Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051037.xml
2022-05-18T05:10:44.0077215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:44.0092655Z 
2022-05-18T05:10:44.0093186Z Running tests...
2022-05-18T05:10:44.0093681Z ----------------------------------------------------------------------
2022-05-18T05:10:45.6338479Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:45.6709297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58658
2022-05-18T05:10:45.6824632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58659
2022-05-18T05:10:46.8866876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:46.8867454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:46.8868246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:46.8868947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:46.8976158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:46.9880944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:48.2192462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvxiwgwy
2022-05-18T05:10:48.2193093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvxiwgwy/_remote_module_non_scriptable.py
2022-05-18T05:10:48.2863156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps793te14
2022-05-18T05:10:48.2864025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps793te14/_remote_module_non_scriptable.py
2022-05-18T05:10:49.1910702Z ok (5.181s)
2022-05-18T05:10:49.1910943Z 
2022-05-18T05:10:49.1911349Z ----------------------------------------------------------------------
2022-05-18T05:10:49.1911689Z Ran 1 test in 5.182s
2022-05-18T05:10:49.1911857Z 
2022-05-18T05:10:49.1911954Z OK
2022-05-18T05:10:49.1912090Z 
2022-05-18T05:10:49.1912207Z Generating XML reports...
2022-05-18T05:10:49.1968421Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051044.xml
2022-05-18T05:10:50.6134954Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:50.6149830Z 
2022-05-18T05:10:50.6150353Z Running tests...
2022-05-18T05:10:50.6150846Z ----------------------------------------------------------------------
2022-05-18T05:10:52.2559237Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:52.2931216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58779
2022-05-18T05:10:52.3041233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58780
2022-05-18T05:10:53.5306258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:53.5306867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:53.5307904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:53.5308628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:53.5315602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:53.5316112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:54.8630899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_cqc59t
2022-05-18T05:10:54.8631559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_cqc59t/_remote_module_non_scriptable.py
2022-05-18T05:10:54.8689266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5g3toy9h
2022-05-18T05:10:54.8693026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5g3toy9h/_remote_module_non_scriptable.py
2022-05-18T05:10:55.5120686Z ok (4.897s)
2022-05-18T05:10:55.5121058Z 
2022-05-18T05:10:55.5121518Z ----------------------------------------------------------------------
2022-05-18T05:10:55.5121868Z Ran 1 test in 4.897s
2022-05-18T05:10:55.5122060Z 
2022-05-18T05:10:55.5122214Z OK
2022-05-18T05:10:55.5122452Z 
2022-05-18T05:10:55.5122647Z Generating XML reports...
2022-05-18T05:10:55.5180375Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051050.xml
2022-05-18T05:10:56.9379782Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:56.9394252Z 
2022-05-18T05:10:56.9394751Z Running tests...
2022-05-18T05:10:56.9395404Z ----------------------------------------------------------------------
2022-05-18T05:10:58.5646172Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:58.6017128Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58896
2022-05-18T05:10:58.6125597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58897
2022-05-18T05:10:59.8065589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:59.8066160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:59.8066956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:59.8067661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:59.8074013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:59.8075210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:01.1385080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rqz44r5
2022-05-18T05:11:01.1386134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rqz44r5/_remote_module_non_scriptable.py
2022-05-18T05:11:01.1584691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_6j6j58
2022-05-18T05:11:01.1587405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_6j6j58/_remote_module_non_scriptable.py
2022-05-18T05:11:01.4560358Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:11:01.4618924Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:11:01.4721195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:01.4722060Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:01.8208258Z ok (4.881s)
2022-05-18T05:11:01.8208498Z 
2022-05-18T05:11:01.8208889Z ----------------------------------------------------------------------
2022-05-18T05:11:01.8209238Z Ran 1 test in 4.881s
2022-05-18T05:11:01.8209403Z 
2022-05-18T05:11:01.8209496Z OK
2022-05-18T05:11:01.8209883Z 
2022-05-18T05:11:01.8210020Z Generating XML reports...
2022-05-18T05:11:01.8266361Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051056.xml
2022-05-18T05:11:03.2595458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:03.2610091Z 
2022-05-18T05:11:03.2610570Z Running tests...
2022-05-18T05:11:03.2611293Z ----------------------------------------------------------------------
2022-05-18T05:11:04.8929404Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:04.9320649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59017
2022-05-18T05:11:04.9431177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59018
2022-05-18T05:11:06.1126184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:06.1126779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:06.1127837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:06.1128567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:06.1237200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:06.2141803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:07.4090584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwxjfghh
2022-05-18T05:11:07.4091202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwxjfghh/_remote_module_non_scriptable.py
2022-05-18T05:11:07.5051062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3xg6lnx5
2022-05-18T05:11:07.5052390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3xg6lnx5/_remote_module_non_scriptable.py
2022-05-18T05:11:07.5225122Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:11:07.5226194Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T05:11:07.5227346Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:11:07.5228176Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T05:11:08.1523124Z ok (4.891s)
2022-05-18T05:11:08.1523367Z 
2022-05-18T05:11:08.1523743Z ----------------------------------------------------------------------
2022-05-18T05:11:08.1524104Z Ran 1 test in 4.891s
2022-05-18T05:11:08.1524268Z 
2022-05-18T05:11:08.1524368Z OK
2022-05-18T05:11:08.1524504Z 
2022-05-18T05:11:08.1524619Z Generating XML reports...
2022-05-18T05:11:08.1581413Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051103.xml
2022-05-18T05:11:09.5780689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:09.5795187Z 
2022-05-18T05:11:09.5795594Z Running tests...
2022-05-18T05:11:09.5796081Z ----------------------------------------------------------------------
2022-05-18T05:11:11.2067579Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:11.2184403Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.639s)
2022-05-18T05:11:11.2184994Z 
2022-05-18T05:11:11.2185249Z ----------------------------------------------------------------------
2022-05-18T05:11:11.2185576Z Ran 1 test in 1.639s
2022-05-18T05:11:11.2185738Z 
2022-05-18T05:11:11.2185846Z OK (skipped=1)
2022-05-18T05:11:11.2186000Z 
2022-05-18T05:11:11.2186124Z Generating XML reports...
2022-05-18T05:11:11.2224075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051109.xml
2022-05-18T05:11:12.6072200Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:12.6087327Z 
2022-05-18T05:11:12.6087750Z Running tests...
2022-05-18T05:11:12.6088519Z ----------------------------------------------------------------------
2022-05-18T05:11:14.2605118Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:14.2989894Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59174
2022-05-18T05:11:14.3100507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59175
2022-05-18T05:11:15.4818120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:15.4818688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:15.4819454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:15.4820156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:15.4827804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:15.4828297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:16.8173141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3wa7_et5
2022-05-18T05:11:16.8174219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3wa7_et5/_remote_module_non_scriptable.py
2022-05-18T05:11:16.8371738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazj_uhtf
2022-05-18T05:11:16.8374353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazj_uhtf/_remote_module_non_scriptable.py
2022-05-18T05:11:17.5551794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:17.5552317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:17.5915517Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:11:17.5917116Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:11:17.9189438Z ok (5.310s)
2022-05-18T05:11:17.9189675Z 
2022-05-18T05:11:17.9190102Z ----------------------------------------------------------------------
2022-05-18T05:11:17.9190689Z Ran 1 test in 5.310s
2022-05-18T05:11:17.9190860Z 
2022-05-18T05:11:17.9190957Z OK
2022-05-18T05:11:17.9191072Z 
2022-05-18T05:11:17.9191204Z Generating XML reports...
2022-05-18T05:11:17.9248863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051112.xml
2022-05-18T05:11:19.3545122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:19.3559915Z 
2022-05-18T05:11:19.3560251Z Running tests...
2022-05-18T05:11:19.3560944Z ----------------------------------------------------------------------
2022-05-18T05:11:21.0016861Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:21.0390050Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59299
2022-05-18T05:11:21.0499902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59300
2022-05-18T05:11:22.2099635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:22.2100194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:22.2100994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:22.2101697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:22.2209681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:22.3115127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:23.5353644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbt3v055c
2022-05-18T05:11:23.5354251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbt3v055c/_remote_module_non_scriptable.py
2022-05-18T05:11:23.5994013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81354jaw
2022-05-18T05:11:23.5995105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81354jaw/_remote_module_non_scriptable.py
2022-05-18T05:11:23.8572770Z ok (4.501s)
2022-05-18T05:11:23.8572978Z 
2022-05-18T05:11:23.8573389Z ----------------------------------------------------------------------
2022-05-18T05:11:23.8573718Z Ran 1 test in 4.501s
2022-05-18T05:11:23.8573880Z 
2022-05-18T05:11:23.8575189Z OK
2022-05-18T05:11:23.8575393Z 
2022-05-18T05:11:23.8575739Z Generating XML reports...
2022-05-18T05:11:23.8630400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051119.xml
2022-05-18T05:11:25.2976859Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:25.2992476Z 
2022-05-18T05:11:25.2992903Z Running tests...
2022-05-18T05:11:25.2993402Z ----------------------------------------------------------------------
2022-05-18T05:11:26.9595927Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:26.9975233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59416
2022-05-18T05:11:27.0087574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59417
2022-05-18T05:11:28.2162759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:28.2163774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:28.2165210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:28.2166612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:28.2173544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:28.2174388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:29.5432726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7kd11xv
2022-05-18T05:11:29.5433838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7kd11xv/_remote_module_non_scriptable.py
2022-05-18T05:11:29.5691564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rhj0qlu
2022-05-18T05:11:29.5694036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rhj0qlu/_remote_module_non_scriptable.py
2022-05-18T05:11:29.5865747Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:11:29.5867304Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T05:11:29.5869419Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:11:29.5870790Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T05:11:29.8764402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:29.8765129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:29.8840815Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:11:29.8842728Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:11:29.8995533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:29.8996492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:29.9087895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:29.9088825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:30.2166894Z ok (4.917s)
2022-05-18T05:11:30.2167122Z 
2022-05-18T05:11:30.2167512Z ----------------------------------------------------------------------
2022-05-18T05:11:30.2167858Z Ran 1 test in 4.917s
2022-05-18T05:11:30.2168025Z 
2022-05-18T05:11:30.2168107Z OK
2022-05-18T05:11:30.2168242Z 
2022-05-18T05:11:30.2168392Z Generating XML reports...
2022-05-18T05:11:30.2224297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051125.xml
2022-05-18T05:11:31.6511846Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:31.6526214Z 
2022-05-18T05:11:31.6526520Z Running tests...
2022-05-18T05:11:31.6526972Z ----------------------------------------------------------------------
2022-05-18T05:11:33.2973608Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:33.3349396Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59537
2022-05-18T05:11:33.3460608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59538
2022-05-18T05:11:34.5405631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:34.5406432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:34.5407223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:34.5407931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:34.5517567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:34.6421198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:35.8505542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpamfm3___
2022-05-18T05:11:35.8506217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpamfm3___/_remote_module_non_scriptable.py
2022-05-18T05:11:35.9478662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpscse5xk7
2022-05-18T05:11:35.9479652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpscse5xk7/_remote_module_non_scriptable.py
2022-05-18T05:11:35.9653004Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:11:35.9654194Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T05:11:35.9655360Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:11:35.9656180Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T05:11:36.2501577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:36.2502152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:36.5540911Z ok (4.901s)
2022-05-18T05:11:36.5541108Z 
2022-05-18T05:11:36.5541810Z ----------------------------------------------------------------------
2022-05-18T05:11:36.5542174Z Ran 1 test in 4.901s
2022-05-18T05:11:36.5542342Z 
2022-05-18T05:11:36.5542440Z OK
2022-05-18T05:11:36.5542586Z 
2022-05-18T05:11:36.5542719Z Generating XML reports...
2022-05-18T05:11:36.5599956Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051131.xml
2022-05-18T05:11:37.9850858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:37.9865563Z 
2022-05-18T05:11:37.9865720Z Running tests...
2022-05-18T05:11:37.9867086Z ----------------------------------------------------------------------
2022-05-18T05:11:39.6008456Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:39.6126018Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.626s)
2022-05-18T05:11:39.6126598Z 
2022-05-18T05:11:39.6126882Z ----------------------------------------------------------------------
2022-05-18T05:11:39.6127220Z Ran 1 test in 1.626s
2022-05-18T05:11:39.6127385Z 
2022-05-18T05:11:39.6127496Z OK (skipped=1)
2022-05-18T05:11:39.6127633Z 
2022-05-18T05:11:39.6127764Z Generating XML reports...
2022-05-18T05:11:39.6164973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051137.xml
2022-05-18T05:11:40.9759114Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:40.9774376Z 
2022-05-18T05:11:40.9774719Z Running tests...
2022-05-18T05:11:40.9775141Z ----------------------------------------------------------------------
2022-05-18T05:11:42.5918721Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:42.6287551Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59694
2022-05-18T05:11:42.6400983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59695
2022-05-18T05:11:43.8129474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:43.8130275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:43.8131105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:43.8131806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:43.8138873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:43.8139375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:45.1356201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp084h6vds
2022-05-18T05:11:45.1356820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp084h6vds/_remote_module_non_scriptable.py
2022-05-18T05:11:45.1750278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kt9i9j5
2022-05-18T05:11:45.1751923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kt9i9j5/_remote_module_non_scriptable.py
2022-05-18T05:11:45.5506539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:45.5507096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:46.6493483Z ok (5.672s)
2022-05-18T05:11:46.6493701Z 
2022-05-18T05:11:46.6494091Z ----------------------------------------------------------------------
2022-05-18T05:11:46.6494434Z Ran 1 test in 5.672s
2022-05-18T05:11:46.6494612Z 
2022-05-18T05:11:46.6494710Z OK
2022-05-18T05:11:46.6494830Z 
2022-05-18T05:11:46.6494967Z Generating XML reports...
2022-05-18T05:11:46.6551314Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051140.xml
2022-05-18T05:11:48.0578884Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:48.0594539Z 
2022-05-18T05:11:48.0594879Z Running tests...
2022-05-18T05:11:48.0595342Z ----------------------------------------------------------------------
2022-05-18T05:11:49.7264273Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:49.7644865Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59819
2022-05-18T05:11:49.7756945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59820
2022-05-18T05:11:50.9850057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:50.9850624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:50.9851428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:50.9852135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:50.9959613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:51.0865070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:52.3312515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk5moo8zr
2022-05-18T05:11:52.3313662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk5moo8zr/_remote_module_non_scriptable.py
2022-05-18T05:11:52.3695130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp9renndl
2022-05-18T05:11:52.3697788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp9renndl/_remote_module_non_scriptable.py
2022-05-18T05:11:52.6834825Z ok (4.624s)
2022-05-18T05:11:52.6836546Z 
2022-05-18T05:11:52.6837146Z ----------------------------------------------------------------------
2022-05-18T05:11:52.6837505Z Ran 1 test in 4.624s
2022-05-18T05:11:52.6837682Z 
2022-05-18T05:11:52.6837778Z OK
2022-05-18T05:11:52.6837920Z 
2022-05-18T05:11:52.6838071Z Generating XML reports...
2022-05-18T05:11:52.6892938Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051148.xml
2022-05-18T05:11:54.1092348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:54.1108394Z 
2022-05-18T05:11:54.1108768Z Running tests...
2022-05-18T05:11:54.1109205Z ----------------------------------------------------------------------
2022-05-18T05:11:55.7243637Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:55.7614131Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59936
2022-05-18T05:11:55.7727837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59937
2022-05-18T05:11:56.9662865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:56.9663443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:56.9664241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:56.9664924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:56.9671905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:56.9672647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:58.3052262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5y6mgg5
2022-05-18T05:11:58.3053396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5y6mgg5/_remote_module_non_scriptable.py
2022-05-18T05:11:58.3203299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnryu3uvp
2022-05-18T05:11:58.3205918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnryu3uvp/_remote_module_non_scriptable.py
2022-05-18T05:11:58.5811896Z ok (4.470s)
2022-05-18T05:11:58.5812267Z 
2022-05-18T05:11:58.5812778Z ----------------------------------------------------------------------
2022-05-18T05:11:58.5813158Z Ran 1 test in 4.470s
2022-05-18T05:11:58.5813333Z 
2022-05-18T05:11:58.5813409Z OK
2022-05-18T05:11:58.5813545Z 
2022-05-18T05:11:58.5813676Z Generating XML reports...
2022-05-18T05:11:58.5869363Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051154.xml
2022-05-18T05:12:00.0242495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:00.0257797Z 
2022-05-18T05:12:00.0258161Z Running tests...
2022-05-18T05:12:00.0258599Z ----------------------------------------------------------------------
2022-05-18T05:12:01.6939055Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:01.7319349Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60053
2022-05-18T05:12:01.7432000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60054
2022-05-18T05:12:02.9952235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:02.9953201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:02.9954301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:03.0053564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:03.0061242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:03.0969711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:04.2875927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnggg2kq_
2022-05-18T05:12:04.2876804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnggg2kq_/_remote_module_non_scriptable.py
2022-05-18T05:12:04.3965583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8unt8cn
2022-05-18T05:12:04.3966660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8unt8cn/_remote_module_non_scriptable.py
2022-05-18T05:12:04.6977215Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:12:04.6977749Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:12:05.0515279Z ok (5.025s)
2022-05-18T05:12:05.0515581Z 
2022-05-18T05:12:05.0516025Z ----------------------------------------------------------------------
2022-05-18T05:12:05.0516340Z Ran 1 test in 5.026s
2022-05-18T05:12:05.0516507Z 
2022-05-18T05:12:05.0516616Z OK
2022-05-18T05:12:05.0516750Z 
2022-05-18T05:12:05.0516878Z Generating XML reports...
2022-05-18T05:12:05.0574417Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051200.xml
2022-05-18T05:12:06.5019408Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:06.5035018Z 
2022-05-18T05:12:06.5035718Z Running tests...
2022-05-18T05:12:06.5036204Z ----------------------------------------------------------------------
2022-05-18T05:12:08.1524418Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:08.1646729Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.661s)
2022-05-18T05:12:08.1647310Z 
2022-05-18T05:12:08.1647597Z ----------------------------------------------------------------------
2022-05-18T05:12:08.1647924Z Ran 1 test in 1.661s
2022-05-18T05:12:08.1648092Z 
2022-05-18T05:12:08.1648201Z OK (skipped=1)
2022-05-18T05:12:08.1648356Z 
2022-05-18T05:12:08.1648463Z Generating XML reports...
2022-05-18T05:12:08.1687592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051206.xml
2022-05-18T05:12:09.5818867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:09.5834025Z 
2022-05-18T05:12:09.5834459Z Running tests...
2022-05-18T05:12:09.5834962Z ----------------------------------------------------------------------
2022-05-18T05:12:11.2646898Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:11.3020576Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60210
2022-05-18T05:12:11.3131780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60211
2022-05-18T05:12:12.4794884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:12.4795455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:12.4796264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:12.4799329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:12.4905040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:12.5810993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:13.7826236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62mrrngl
2022-05-18T05:12:13.7826841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62mrrngl/_remote_module_non_scriptable.py
2022-05-18T05:12:13.8787889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9_gumii
2022-05-18T05:12:13.8788766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9_gumii/_remote_module_non_scriptable.py
2022-05-18T05:12:14.1891776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:12:14.1892341Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:12:14.2234549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:12:14.2235264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:12:14.2396667Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp62mrrngl'>
2022-05-18T05:12:14.2397171Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:12:14.5211377Z ok (4.937s)
2022-05-18T05:12:14.5211680Z 
2022-05-18T05:12:14.5212162Z ----------------------------------------------------------------------
2022-05-18T05:12:14.5212533Z Ran 1 test in 4.938s
2022-05-18T05:12:14.5212697Z 
2022-05-18T05:12:14.5212771Z OK
2022-05-18T05:12:14.5212906Z 
2022-05-18T05:12:14.5213036Z Generating XML reports...
2022-05-18T05:12:14.5271078Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051209.xml
2022-05-18T05:12:15.9703649Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:15.9719034Z 
2022-05-18T05:12:15.9719337Z Running tests...
2022-05-18T05:12:15.9719769Z ----------------------------------------------------------------------
2022-05-18T05:12:17.6286141Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:17.6657327Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60331
2022-05-18T05:12:17.6766366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60332
2022-05-18T05:12:18.8426638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:18.8427393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:18.8428187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:18.8428861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:18.8435640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:18.8436655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:20.1707237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpig8m7gic
2022-05-18T05:12:20.1707865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpig8m7gic/_remote_module_non_scriptable.py
2022-05-18T05:12:20.1834128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs9ce0dn
2022-05-18T05:12:20.1836786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs9ce0dn/_remote_module_non_scriptable.py
2022-05-18T05:12:20.7846053Z ok (4.812s)
2022-05-18T05:12:20.7846257Z 
2022-05-18T05:12:20.7846654Z ----------------------------------------------------------------------
2022-05-18T05:12:20.7846988Z Ran 1 test in 4.813s
2022-05-18T05:12:20.7847162Z 
2022-05-18T05:12:20.7847252Z OK
2022-05-18T05:12:20.7847390Z 
2022-05-18T05:12:20.7847507Z Generating XML reports...
2022-05-18T05:12:20.7904965Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051215.xml
2022-05-18T05:12:22.2152805Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:22.2167984Z 
2022-05-18T05:12:22.2168245Z Running tests...
2022-05-18T05:12:22.2168671Z ----------------------------------------------------------------------
2022-05-18T05:12:23.8417585Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:23.8790825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60452
2022-05-18T05:12:23.8899985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60453
2022-05-18T05:12:25.0579164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:25.0579726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:25.0580525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:25.0581245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:25.0688197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:25.1592134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:25.1705207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:12:25.1705732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:12:25.1706457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:12:25.1707163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:12:25.3953106Z ok (3.178s)
2022-05-18T05:12:25.3953344Z 
2022-05-18T05:12:25.3953727Z ----------------------------------------------------------------------
2022-05-18T05:12:25.3954084Z Ran 1 test in 3.178s
2022-05-18T05:12:25.3954231Z 
2022-05-18T05:12:25.3954323Z OK
2022-05-18T05:12:25.3954460Z 
2022-05-18T05:12:25.3954594Z Generating XML reports...
2022-05-18T05:12:25.4012155Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051222.xml
2022-05-18T05:12:26.8438981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:26.8454940Z 
2022-05-18T05:12:26.8455198Z Running tests...
2022-05-18T05:12:26.8455633Z ----------------------------------------------------------------------
2022-05-18T05:12:28.4999361Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:28.5394965Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60573
2022-05-18T05:12:28.5508234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60574
2022-05-18T05:12:29.7314692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:29.7315283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:29.7316090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:29.7316782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:29.7424589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:29.8327174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:29.8535853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:12:29.8536367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:12:29.8537309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:12:29.8538042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:12:30.1560690Z ok (3.310s)
2022-05-18T05:12:30.1560919Z 
2022-05-18T05:12:30.1561304Z ----------------------------------------------------------------------
2022-05-18T05:12:30.1561647Z Ran 1 test in 3.311s
2022-05-18T05:12:30.1561813Z 
2022-05-18T05:12:30.1561895Z OK
2022-05-18T05:12:30.1562037Z 
2022-05-18T05:12:30.1562174Z Generating XML reports...
2022-05-18T05:12:30.1619934Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051226.xml
2022-05-18T05:12:31.6132678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:31.6148569Z 
2022-05-18T05:12:31.6148931Z Running tests...
2022-05-18T05:12:31.6149461Z ----------------------------------------------------------------------
2022-05-18T05:12:33.2757852Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:33.3126906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60694
2022-05-18T05:12:33.3237803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60695
2022-05-18T05:12:34.4973270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:34.4973861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:34.4974689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:34.4975392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:34.5083676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:34.5989512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:35.8339993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjyq0knmi
2022-05-18T05:12:35.8340613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjyq0knmi/_remote_module_non_scriptable.py
2022-05-18T05:12:35.8903816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfkfs03b
2022-05-18T05:12:35.8905272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfkfs03b/_remote_module_non_scriptable.py
2022-05-18T05:12:36.2087835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:12:36.2088415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:12:36.2172260Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:12:36.2173950Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:12:36.5316695Z ok (4.916s)
2022-05-18T05:12:36.5316884Z 
2022-05-18T05:12:36.5317421Z ----------------------------------------------------------------------
2022-05-18T05:12:36.5317756Z Ran 1 test in 4.917s
2022-05-18T05:12:36.5317927Z 
2022-05-18T05:12:36.5318020Z OK
2022-05-18T05:12:36.5318137Z 
2022-05-18T05:12:36.5318267Z Generating XML reports...
2022-05-18T05:12:36.5375221Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051231.xml
2022-05-18T05:12:37.9811913Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:37.9827080Z 
2022-05-18T05:12:37.9827612Z Running tests...
2022-05-18T05:12:37.9828305Z ----------------------------------------------------------------------
2022-05-18T05:12:39.6370278Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:39.6748516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60815
2022-05-18T05:12:39.6861042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60816
2022-05-18T05:12:40.8587680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:40.8588248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:40.8589020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:40.8589720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:40.8699039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:40.9604022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:42.1532172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ct4vi5e
2022-05-18T05:12:42.1533002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ct4vi5e/_remote_module_non_scriptable.py
2022-05-18T05:12:42.2597422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi3lj8p66
2022-05-18T05:12:42.2598760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi3lj8p66/_remote_module_non_scriptable.py
2022-05-18T05:12:42.5621777Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:12:42.5834771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:12:42.5835280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:12:42.8942197Z ok (4.911s)
2022-05-18T05:12:42.8942437Z 
2022-05-18T05:12:42.8943056Z ----------------------------------------------------------------------
2022-05-18T05:12:42.8943479Z Ran 1 test in 4.911s
2022-05-18T05:12:42.8943652Z 
2022-05-18T05:12:42.8943746Z OK
2022-05-18T05:12:42.8943861Z 
2022-05-18T05:12:42.8944005Z Generating XML reports...
2022-05-18T05:12:42.9001640Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051237.xml
2022-05-18T05:12:44.3171274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:44.3186437Z 
2022-05-18T05:12:44.3186906Z Running tests...
2022-05-18T05:12:44.3187413Z ----------------------------------------------------------------------
2022-05-18T05:12:45.9326191Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:45.9696595Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60936
2022-05-18T05:12:45.9808817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60937
2022-05-18T05:12:47.1491229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:47.1491842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:47.1492651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:47.1493330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:47.1500737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:47.1501243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:47.3857457Z ok (3.067s)
2022-05-18T05:12:47.3857673Z 
2022-05-18T05:12:47.3858068Z ----------------------------------------------------------------------
2022-05-18T05:12:47.3858406Z Ran 1 test in 3.067s
2022-05-18T05:12:47.3858571Z 
2022-05-18T05:12:47.3858666Z OK
2022-05-18T05:12:47.3858802Z 
2022-05-18T05:12:47.3858918Z Generating XML reports...
2022-05-18T05:12:47.3916902Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051244.xml
2022-05-18T05:12:48.7970338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:48.7984982Z 
2022-05-18T05:12:48.8037134Z Running tests...
2022-05-18T05:12:48.8037724Z ----------------------------------------------------------------------
2022-05-18T05:12:50.4189436Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:50.4560870Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61051
2022-05-18T05:12:50.4675487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61052
2022-05-18T05:12:51.6718062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:51.6718711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:51.6719480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:51.6720443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:51.6829505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:51.7729761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:51.9726809Z ok (3.174s)
2022-05-18T05:12:51.9727051Z 
2022-05-18T05:12:51.9727633Z ----------------------------------------------------------------------
2022-05-18T05:12:51.9727984Z Ran 1 test in 3.174s
2022-05-18T05:12:51.9728147Z 
2022-05-18T05:12:51.9728240Z OK
2022-05-18T05:12:51.9728372Z 
2022-05-18T05:12:51.9728512Z Generating XML reports...
2022-05-18T05:12:51.9785233Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051248.xml
2022-05-18T05:12:53.3696072Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:53.3715053Z 
2022-05-18T05:12:53.3715480Z Running tests...
2022-05-18T05:12:53.3715964Z ----------------------------------------------------------------------
2022-05-18T05:12:54.9799539Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:55.0171028Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61170
2022-05-18T05:12:55.0281282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61171
2022-05-18T05:12:56.2321826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:56.2322481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:56.2323286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:56.2323994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:56.2330778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:56.2331628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:56.4330216Z ok (3.061s)
2022-05-18T05:12:56.4330865Z 
2022-05-18T05:12:56.4331612Z ----------------------------------------------------------------------
2022-05-18T05:12:56.4332261Z Ran 1 test in 3.061s
2022-05-18T05:12:56.4332415Z 
2022-05-18T05:12:56.4332508Z OK
2022-05-18T05:12:56.4332649Z 
2022-05-18T05:12:56.4332784Z Generating XML reports...
2022-05-18T05:12:56.4387990Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051253.xml
2022-05-18T05:12:57.8247702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:57.8264233Z 
2022-05-18T05:12:57.8264630Z Running tests...
2022-05-18T05:12:57.8265166Z ----------------------------------------------------------------------
2022-05-18T05:12:57.8286645Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T05:12:57.8286949Z 
2022-05-18T05:12:57.8287244Z ----------------------------------------------------------------------
2022-05-18T05:12:57.8287555Z Ran 1 test in 0.002s
2022-05-18T05:12:57.8287731Z 
2022-05-18T05:12:57.8287842Z OK (skipped=1)
2022-05-18T05:12:57.8287999Z 
2022-05-18T05:12:57.8288124Z Generating XML reports...
2022-05-18T05:12:57.8331349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051257.xml
2022-05-18T05:12:59.1030741Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:59.1046371Z 
2022-05-18T05:12:59.1046614Z Running tests...
2022-05-18T05:12:59.1047043Z ----------------------------------------------------------------------
2022-05-18T05:13:00.7527103Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:00.7906735Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61320
2022-05-18T05:13:00.8019025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61321
2022-05-18T05:13:02.0081148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:02.0081710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:02.0082494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:02.0083175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:02.0090834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:02.0091345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:02.0198770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:13:02.0199498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:13:02.0200191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:13:02.0200880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:13:02.3069948Z ok (3.202s)
2022-05-18T05:13:02.3070160Z 
2022-05-18T05:13:02.3070509Z ----------------------------------------------------------------------
2022-05-18T05:13:02.3070847Z Ran 1 test in 3.202s
2022-05-18T05:13:02.3071011Z 
2022-05-18T05:13:02.3071111Z OK
2022-05-18T05:13:02.3071247Z 
2022-05-18T05:13:02.3071384Z Generating XML reports...
2022-05-18T05:13:02.3128676Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051259.xml
2022-05-18T05:13:03.7426634Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:03.7442513Z 
2022-05-18T05:13:03.7442932Z Running tests...
2022-05-18T05:13:03.7443396Z ----------------------------------------------------------------------
2022-05-18T05:13:05.3919403Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:05.4301924Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61445
2022-05-18T05:13:05.4413528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61446
2022-05-18T05:13:06.6026090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:06.6026641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:06.6027446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:06.6028149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:06.6135175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:06.7041598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:06.8463155Z skip: Skipped due to small world size. (3.102s)
2022-05-18T05:13:06.8463416Z 
2022-05-18T05:13:06.8463880Z ----------------------------------------------------------------------
2022-05-18T05:13:06.8464332Z Ran 1 test in 3.102s
2022-05-18T05:13:06.8464498Z 
2022-05-18T05:13:06.8464590Z OK (skipped=1)
2022-05-18T05:13:06.8465561Z 
2022-05-18T05:13:06.8465912Z Generating XML reports...
2022-05-18T05:13:06.8523982Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051303.xml
2022-05-18T05:13:08.2714282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:08.2730011Z 
2022-05-18T05:13:08.2730495Z Running tests...
2022-05-18T05:13:08.2730985Z ----------------------------------------------------------------------
2022-05-18T05:13:09.9186215Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:09.9565546Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61560
2022-05-18T05:13:09.9678389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61561
2022-05-18T05:13:11.1411057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:11.1411664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:11.1413070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:11.1414377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:11.1421093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:11.1422456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:11.3729574Z ok (3.100s)
2022-05-18T05:13:11.3730127Z 
2022-05-18T05:13:11.3730656Z ----------------------------------------------------------------------
2022-05-18T05:13:11.3730981Z Ran 1 test in 3.100s
2022-05-18T05:13:11.3731150Z 
2022-05-18T05:13:11.3731245Z OK
2022-05-18T05:13:11.3731383Z 
2022-05-18T05:13:11.3731520Z Generating XML reports...
2022-05-18T05:13:11.3788097Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051308.xml
2022-05-18T05:13:12.7814166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:12.7830889Z 
2022-05-18T05:13:12.7831249Z Running tests...
2022-05-18T05:13:12.7831780Z ----------------------------------------------------------------------
2022-05-18T05:13:14.4496719Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:14.4878553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61675
2022-05-18T05:13:14.4991030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61676
2022-05-18T05:13:15.6546902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:15.6547743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:15.6549225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:15.6550542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:15.6556468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:15.6558419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:15.6971833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:13:15.6972844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:13:15.6974157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:13:15.6975520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:13:15.7118797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:13:15.7119353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:13:15.7120652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:13:15.7221305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:13:15.7341986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T05:13:15.7342496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T05:13:15.7343688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T05:13:15.7344847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T05:13:16.0041793Z ok (3.221s)
2022-05-18T05:13:16.0041968Z 
2022-05-18T05:13:16.0043230Z ----------------------------------------------------------------------
2022-05-18T05:13:16.0044184Z Ran 1 test in 3.221s
2022-05-18T05:13:16.0044498Z 
2022-05-18T05:13:16.0044666Z OK
2022-05-18T05:13:16.0044913Z 
2022-05-18T05:13:16.0045137Z Generating XML reports...
2022-05-18T05:13:16.0103189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051312.xml
2022-05-18T05:13:17.4485555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:17.4500671Z 
2022-05-18T05:13:17.4500820Z Running tests...
2022-05-18T05:13:17.4501949Z ----------------------------------------------------------------------
2022-05-18T05:13:19.1100486Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:19.1471305Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61814
2022-05-18T05:13:19.1582480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61815
2022-05-18T05:13:20.3516969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:20.3517514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:20.3518316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:20.3519011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:20.3525536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:20.3526508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:20.3634690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:13:20.3635217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:13:20.3636206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:13:20.3636904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:13:20.6632485Z ok (3.213s)
2022-05-18T05:13:20.6632712Z 
2022-05-18T05:13:20.6633088Z ----------------------------------------------------------------------
2022-05-18T05:13:20.6633431Z Ran 1 test in 3.213s
2022-05-18T05:13:20.6633595Z 
2022-05-18T05:13:20.6633694Z OK
2022-05-18T05:13:20.6633830Z 
2022-05-18T05:13:20.6633961Z Generating XML reports...
2022-05-18T05:13:20.6691190Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051317.xml
2022-05-18T05:13:22.0932273Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:22.0947728Z 
2022-05-18T05:13:22.0948089Z Running tests...
2022-05-18T05:13:22.0948533Z ----------------------------------------------------------------------
2022-05-18T05:13:23.7520677Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:23.7893222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61935
2022-05-18T05:13:23.8004205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61936
2022-05-18T05:13:24.9717272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:24.9717840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:24.9718663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:24.9719366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:24.9726904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:24.9728028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:25.2053718Z ok (3.110s)
2022-05-18T05:13:25.2054078Z 
2022-05-18T05:13:25.2054645Z ----------------------------------------------------------------------
2022-05-18T05:13:25.2054990Z Ran 1 test in 3.111s
2022-05-18T05:13:25.2055156Z 
2022-05-18T05:13:25.2055251Z OK
2022-05-18T05:13:25.2055393Z 
2022-05-18T05:13:25.2055530Z Generating XML reports...
2022-05-18T05:13:25.2112268Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051322.xml
2022-05-18T05:13:26.6372543Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:26.6388349Z 
2022-05-18T05:13:26.6388841Z Running tests...
2022-05-18T05:13:26.6389353Z ----------------------------------------------------------------------
2022-05-18T05:13:28.3243439Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:28.3627311Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62050
2022-05-18T05:13:28.3741180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62051
2022-05-18T05:13:29.5593372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:29.5593934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:29.5594710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:29.5595401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:29.5702232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:29.6606595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:30.0796159Z ok (3.440s)
2022-05-18T05:13:30.0796348Z 
2022-05-18T05:13:30.0797003Z ----------------------------------------------------------------------
2022-05-18T05:13:30.0797360Z Ran 1 test in 3.441s
2022-05-18T05:13:30.0797522Z 
2022-05-18T05:13:30.0797623Z OK
2022-05-18T05:13:30.0797762Z 
2022-05-18T05:13:30.0797877Z Generating XML reports...
2022-05-18T05:13:30.0854132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051326.xml
2022-05-18T05:13:31.5031940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:31.5046122Z 
2022-05-18T05:13:31.5046629Z Running tests...
2022-05-18T05:13:31.5047130Z ----------------------------------------------------------------------
2022-05-18T05:13:33.1244515Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:33.1617133Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62165
2022-05-18T05:13:33.1731532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62166
2022-05-18T05:13:34.3676846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:34.3677400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:34.3678158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:34.3678911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:34.3686026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:34.3686751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:34.3893419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:13:34.3894098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:13:34.3894793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:13:34.3895681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:13:34.5780244Z ok (3.073s)
2022-05-18T05:13:34.5780477Z 
2022-05-18T05:13:34.5780852Z ----------------------------------------------------------------------
2022-05-18T05:13:34.5781193Z Ran 1 test in 3.073s
2022-05-18T05:13:34.5781359Z 
2022-05-18T05:13:34.5781451Z OK
2022-05-18T05:13:34.5781584Z 
2022-05-18T05:13:34.5781729Z Generating XML reports...
2022-05-18T05:13:34.5839395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051331.xml
2022-05-18T05:13:36.0048355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:36.0063926Z 
2022-05-18T05:13:36.0064413Z Running tests...
2022-05-18T05:13:36.0064909Z ----------------------------------------------------------------------
2022-05-18T05:13:37.6629654Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:37.7011161Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62286
2022-05-18T05:13:37.7123854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62287
2022-05-18T05:13:38.8294031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:38.8294614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:38.8295428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:38.8296124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:38.8404962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:38.9309164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:38.9517083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:13:38.9517774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:13:38.9518487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:13:38.9519182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:13:39.2174514Z ok (3.211s)
2022-05-18T05:13:39.2174738Z 
2022-05-18T05:13:39.2175102Z ----------------------------------------------------------------------
2022-05-18T05:13:39.2175440Z Ran 1 test in 3.211s
2022-05-18T05:13:39.2175605Z 
2022-05-18T05:13:39.2175700Z OK
2022-05-18T05:13:39.2175841Z 
2022-05-18T05:13:39.2175974Z Generating XML reports...
2022-05-18T05:13:39.2232989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051335.xml
2022-05-18T05:13:40.6333528Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:40.6348781Z 
2022-05-18T05:13:40.6349078Z Running tests...
2022-05-18T05:13:40.6349499Z ----------------------------------------------------------------------
2022-05-18T05:13:42.2848842Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:42.3218996Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62407
2022-05-18T05:13:42.3330375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62408
2022-05-18T05:13:43.5432696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:43.5433419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:43.5434222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:43.5434943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:43.5442732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:43.5443534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:44.8782421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6hknn_z
2022-05-18T05:13:44.8783032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6hknn_z/_remote_module_non_scriptable.py
2022-05-18T05:13:44.9170075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8sjrl_zt
2022-05-18T05:13:44.9172773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8sjrl_zt/_remote_module_non_scriptable.py
2022-05-18T05:13:45.5410546Z ok (4.906s)
2022-05-18T05:13:45.5410792Z 
2022-05-18T05:13:45.5411180Z ----------------------------------------------------------------------
2022-05-18T05:13:45.5411519Z Ran 1 test in 4.906s
2022-05-18T05:13:45.5411705Z 
2022-05-18T05:13:45.5411803Z OK
2022-05-18T05:13:45.5411946Z 
2022-05-18T05:13:45.5412066Z Generating XML reports...
2022-05-18T05:13:45.5468343Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051340.xml
2022-05-18T05:13:46.9779179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:46.9794262Z 
2022-05-18T05:13:46.9794511Z Running tests...
2022-05-18T05:13:46.9794959Z ----------------------------------------------------------------------
2022-05-18T05:13:48.6490403Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:48.6868238Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62528
2022-05-18T05:13:48.6979610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62529
2022-05-18T05:13:49.9119450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:49.9120032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:49.9120802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:49.9121503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:49.9128382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:49.9129878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:50.1028853Z ok (3.123s)
2022-05-18T05:13:50.1029086Z 
2022-05-18T05:13:50.1029659Z ----------------------------------------------------------------------
2022-05-18T05:13:50.1030007Z Ran 1 test in 3.123s
2022-05-18T05:13:50.1030175Z 
2022-05-18T05:13:50.1030260Z OK
2022-05-18T05:13:50.1030397Z 
2022-05-18T05:13:50.1030557Z Generating XML reports...
2022-05-18T05:13:50.1086450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051346.xml
2022-05-18T05:13:51.5287051Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:51.5303051Z 
2022-05-18T05:13:51.5303295Z Running tests...
2022-05-18T05:13:51.5303725Z ----------------------------------------------------------------------
2022-05-18T05:13:53.1941696Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:53.2321218Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62643
2022-05-18T05:13:53.2433486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62644
2022-05-18T05:13:54.4085843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:54.4086430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:54.4087202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:54.4087889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:54.4094790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:54.4096258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:54.6481609Z ok (3.117s)
2022-05-18T05:13:54.6481836Z 
2022-05-18T05:13:54.6482187Z ----------------------------------------------------------------------
2022-05-18T05:13:54.6482529Z Ran 1 test in 3.118s
2022-05-18T05:13:54.6482693Z 
2022-05-18T05:13:54.6482795Z OK
2022-05-18T05:13:54.6482929Z 
2022-05-18T05:13:54.6483063Z Generating XML reports...
2022-05-18T05:13:54.6540048Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051351.xml
2022-05-18T05:13:56.0723275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:56.0739502Z 
2022-05-18T05:13:56.0739797Z Running tests...
2022-05-18T05:13:56.0740217Z ----------------------------------------------------------------------
2022-05-18T05:13:57.7440104Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:57.7820415Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62758
2022-05-18T05:13:57.7931879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62759
2022-05-18T05:13:58.9666348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:58.9666868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:58.9667843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:58.9668563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:58.9777385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:59.0678134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:59.3984459Z ok (3.324s)
2022-05-18T05:13:59.3986121Z 
2022-05-18T05:13:59.3986809Z ----------------------------------------------------------------------
2022-05-18T05:13:59.3987178Z Ran 1 test in 3.325s
2022-05-18T05:13:59.3987350Z 
2022-05-18T05:13:59.3987456Z OK
2022-05-18T05:13:59.3987574Z 
2022-05-18T05:13:59.3987707Z Generating XML reports...
2022-05-18T05:13:59.4042550Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051356.xml
2022-05-18T05:14:00.8255369Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:00.8270842Z 
2022-05-18T05:14:00.8271318Z Running tests...
2022-05-18T05:14:00.8271770Z ----------------------------------------------------------------------
2022-05-18T05:14:02.4736958Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:02.5119248Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62877
2022-05-18T05:14:02.5232761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62878
2022-05-18T05:14:03.6898783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:03.6899350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:03.6900136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:03.6900840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:03.7008019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:03.7912984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:04.1285119Z ok (3.301s)
2022-05-18T05:14:04.1285342Z 
2022-05-18T05:14:04.1285747Z ----------------------------------------------------------------------
2022-05-18T05:14:04.1286091Z Ran 1 test in 3.301s
2022-05-18T05:14:04.1286236Z 
2022-05-18T05:14:04.1286339Z OK
2022-05-18T05:14:04.1286481Z 
2022-05-18T05:14:04.1286619Z Generating XML reports...
2022-05-18T05:14:04.1344576Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051400.xml
2022-05-18T05:14:05.5638656Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:05.5654710Z 
2022-05-18T05:14:05.5654987Z Running tests...
2022-05-18T05:14:05.5655423Z ----------------------------------------------------------------------
2022-05-18T05:14:05.5676881Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T05:14:05.5677247Z 
2022-05-18T05:14:05.5677768Z ----------------------------------------------------------------------
2022-05-18T05:14:05.5678132Z Ran 1 test in 0.002s
2022-05-18T05:14:05.5678304Z 
2022-05-18T05:14:05.5678431Z OK (skipped=1)
2022-05-18T05:14:05.5678720Z 
2022-05-18T05:14:05.5678903Z Generating XML reports...
2022-05-18T05:14:05.5722226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051405.xml
2022-05-18T05:14:06.8462778Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:06.8477261Z 
2022-05-18T05:14:06.8477534Z Running tests...
2022-05-18T05:14:06.8478000Z ----------------------------------------------------------------------
2022-05-18T05:14:06.8499195Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T05:14:06.8499582Z 
2022-05-18T05:14:06.8499874Z ----------------------------------------------------------------------
2022-05-18T05:14:06.8500205Z Ran 1 test in 0.002s
2022-05-18T05:14:06.8500351Z 
2022-05-18T05:14:06.8500461Z OK (skipped=1)
2022-05-18T05:14:06.8500616Z 
2022-05-18T05:14:06.8500744Z Generating XML reports...
2022-05-18T05:14:06.8543154Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051406.xml
2022-05-18T05:14:08.1425384Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:08.1440320Z 
2022-05-18T05:14:08.1440706Z Running tests...
2022-05-18T05:14:08.1441218Z ----------------------------------------------------------------------
2022-05-18T05:14:09.8057265Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:09.8436269Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63066
2022-05-18T05:14:09.8548566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63067
2022-05-18T05:14:11.0420754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:11.0421734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:11.0422531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:11.0423230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:11.0531926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:11.1436124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:11.3602930Z skip: Skipped due to small world size. (3.216s)
2022-05-18T05:14:11.3603621Z 
2022-05-18T05:14:11.3604256Z ----------------------------------------------------------------------
2022-05-18T05:14:11.3604908Z Ran 1 test in 3.216s
2022-05-18T05:14:11.3605260Z 
2022-05-18T05:14:11.3605492Z OK (skipped=1)
2022-05-18T05:14:11.3605837Z 
2022-05-18T05:14:11.3606061Z Generating XML reports...
2022-05-18T05:14:11.3661957Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051408.xml
2022-05-18T05:14:12.7646462Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:12.7664807Z 
2022-05-18T05:14:12.7665307Z Running tests...
2022-05-18T05:14:12.7666021Z ----------------------------------------------------------------------
2022-05-18T05:14:14.3763760Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:14.4132734Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63181
2022-05-18T05:14:14.4242692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63182
2022-05-18T05:14:15.6406331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:15.6406903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:15.6407702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:15.6408598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:15.6514998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:15.7418916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:17.7570524Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T05:14:18.0329601Z ok (5.266s)
2022-05-18T05:14:18.0329897Z 
2022-05-18T05:14:18.0330571Z ----------------------------------------------------------------------
2022-05-18T05:14:18.0330951Z Ran 1 test in 5.267s
2022-05-18T05:14:18.0331118Z 
2022-05-18T05:14:18.0331214Z OK
2022-05-18T05:14:18.0331351Z 
2022-05-18T05:14:18.0331485Z Generating XML reports...
2022-05-18T05:14:18.0388096Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051412.xml
2022-05-18T05:14:19.4724437Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:19.4740050Z 
2022-05-18T05:14:19.4740583Z Running tests...
2022-05-18T05:14:19.4741002Z ----------------------------------------------------------------------
2022-05-18T05:14:21.1406359Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:21.1786295Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63296
2022-05-18T05:14:21.1898627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63297
2022-05-18T05:14:22.3920406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:22.3920972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:22.3921761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:22.3922463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:22.4030198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:22.4932559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:22.5140429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:14:22.5140924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:14:22.5141623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:14:22.5142320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:14:22.5144737Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-05-18T05:14:22.5145140Z No ranks successfully processed in monitoredBarrier.
2022-05-18T05:14:22.5174654Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-05-18T05:14:22.7951463Z ok (3.321s)
2022-05-18T05:14:22.7951674Z 
2022-05-18T05:14:22.7952047Z ----------------------------------------------------------------------
2022-05-18T05:14:22.7952366Z Ran 1 test in 3.321s
2022-05-18T05:14:22.7952533Z 
2022-05-18T05:14:22.7952635Z OK
2022-05-18T05:14:22.7952770Z 
2022-05-18T05:14:22.7952904Z Generating XML reports...
2022-05-18T05:14:22.8009339Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051419.xml
2022-05-18T05:14:24.2431434Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:24.2446528Z 
2022-05-18T05:14:24.2446985Z Running tests...
2022-05-18T05:14:24.2447494Z ----------------------------------------------------------------------
2022-05-18T05:14:25.8937913Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:25.9306545Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63417
2022-05-18T05:14:25.9417445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63418
2022-05-18T05:14:27.1288483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:27.1289065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:27.1290113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:27.1290795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:27.1297484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:27.1298580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:27.1405433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:14:27.1406439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:14:27.1407135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:14:27.1407820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:14:27.2413522Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-05-18T05:14:27.5469512Z ok (3.302s)
2022-05-18T05:14:27.5469712Z 
2022-05-18T05:14:27.5470091Z ----------------------------------------------------------------------
2022-05-18T05:14:27.5470444Z Ran 1 test in 3.302s
2022-05-18T05:14:27.5470609Z 
2022-05-18T05:14:27.5470702Z OK
2022-05-18T05:14:27.5470816Z 
2022-05-18T05:14:27.5470945Z Generating XML reports...
2022-05-18T05:14:27.5527712Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051424.xml
2022-05-18T05:14:28.9954268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:28.9970180Z 
2022-05-18T05:14:28.9970431Z Running tests...
2022-05-18T05:14:28.9970851Z ----------------------------------------------------------------------
2022-05-18T05:14:30.6498067Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:30.6879354Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63538
2022-05-18T05:14:30.6992739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63539
2022-05-18T05:14:31.8982064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:31.8982771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:31.8983558Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:31.8984245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:31.9091057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:31.9997080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:32.2043535Z skip: Skipped due to small world size. (3.207s)
2022-05-18T05:14:32.2044037Z 
2022-05-18T05:14:32.2044748Z ----------------------------------------------------------------------
2022-05-18T05:14:32.2045086Z Ran 1 test in 3.207s
2022-05-18T05:14:32.2045497Z 
2022-05-18T05:14:32.2045623Z OK (skipped=1)
2022-05-18T05:14:32.2045787Z 
2022-05-18T05:14:32.2045912Z Generating XML reports...
2022-05-18T05:14:32.2102160Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051428.xml
2022-05-18T05:14:33.6310076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:33.6325553Z 
2022-05-18T05:14:33.6326143Z Running tests...
2022-05-18T05:14:33.6326584Z ----------------------------------------------------------------------
2022-05-18T05:14:33.6354468Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T05:14:33.6355202Z 
2022-05-18T05:14:33.6355470Z ----------------------------------------------------------------------
2022-05-18T05:14:33.6355831Z Ran 1 test in 0.003s
2022-05-18T05:14:33.6355998Z 
2022-05-18T05:14:33.6356112Z OK (skipped=1)
2022-05-18T05:14:33.6356292Z 
2022-05-18T05:14:33.6356418Z Generating XML reports...
2022-05-18T05:14:33.6399453Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051433.xml
2022-05-18T05:14:34.9004688Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:34.9020211Z 
2022-05-18T05:14:34.9020620Z Running tests...
2022-05-18T05:14:34.9021120Z ----------------------------------------------------------------------
2022-05-18T05:14:34.9050737Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T05:14:34.9051079Z 
2022-05-18T05:14:34.9051347Z ----------------------------------------------------------------------
2022-05-18T05:14:34.9051675Z Ran 1 test in 0.003s
2022-05-18T05:14:34.9051841Z 
2022-05-18T05:14:34.9051952Z OK (skipped=1)
2022-05-18T05:14:34.9052115Z 
2022-05-18T05:14:34.9052239Z Generating XML reports...
2022-05-18T05:14:34.9094284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051434.xml
2022-05-18T05:14:36.1820425Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:36.1835412Z 
2022-05-18T05:14:36.1835658Z Running tests...
2022-05-18T05:14:36.1836368Z ----------------------------------------------------------------------
2022-05-18T05:14:36.1863961Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T05:14:36.1864300Z 
2022-05-18T05:14:36.1864799Z ----------------------------------------------------------------------
2022-05-18T05:14:36.1865156Z Ran 1 test in 0.003s
2022-05-18T05:14:36.1865319Z 
2022-05-18T05:14:36.1865430Z OK (skipped=1)
2022-05-18T05:14:36.1865569Z 
2022-05-18T05:14:36.1865735Z Generating XML reports...
2022-05-18T05:14:36.1908419Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051436.xml
2022-05-18T05:14:37.4490509Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:37.4506078Z 
2022-05-18T05:14:37.4506216Z Running tests...
2022-05-18T05:14:37.4507290Z ----------------------------------------------------------------------
2022-05-18T05:14:37.4541073Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T05:14:37.4541629Z 
2022-05-18T05:14:37.4542023Z ----------------------------------------------------------------------
2022-05-18T05:14:37.4542405Z Ran 1 test in 0.004s
2022-05-18T05:14:37.4542573Z 
2022-05-18T05:14:37.4542697Z OK (skipped=1)
2022-05-18T05:14:37.4542853Z 
2022-05-18T05:14:37.4542983Z Generating XML reports...
2022-05-18T05:14:37.4585785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051437.xml
2022-05-18T05:14:38.7534163Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:38.7550044Z 
2022-05-18T05:14:38.7550334Z Running tests...
2022-05-18T05:14:38.7550793Z ----------------------------------------------------------------------
2022-05-18T05:14:38.7579186Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.003s)
2022-05-18T05:14:38.7579771Z 
2022-05-18T05:14:38.7580111Z ----------------------------------------------------------------------
2022-05-18T05:14:38.7580438Z Ran 1 test in 0.003s
2022-05-18T05:14:38.7580603Z 
2022-05-18T05:14:38.7580713Z OK (skipped=1)
2022-05-18T05:14:38.7580868Z 
2022-05-18T05:14:38.7580994Z Generating XML reports...
2022-05-18T05:14:38.7624572Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051438.xml
2022-05-18T05:14:40.0429954Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:40.0446269Z 
2022-05-18T05:14:40.0446408Z Running tests...
2022-05-18T05:14:40.0447330Z ----------------------------------------------------------------------
2022-05-18T05:14:40.0472757Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T05:14:40.0473236Z 
2022-05-18T05:14:40.0474135Z ----------------------------------------------------------------------
2022-05-18T05:14:40.0474813Z Ran 1 test in 0.003s
2022-05-18T05:14:40.0475132Z 
2022-05-18T05:14:40.0475337Z OK (skipped=1)
2022-05-18T05:14:40.0475650Z 
2022-05-18T05:14:40.0475858Z Generating XML reports...
2022-05-18T05:14:40.0520322Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051440.xml
2022-05-18T05:14:41.3287929Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:41.3303173Z 
2022-05-18T05:14:41.3303482Z Running tests...
2022-05-18T05:14:41.3304206Z ----------------------------------------------------------------------
2022-05-18T05:14:41.3332182Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-05-18T05:14:41.3332524Z 
2022-05-18T05:14:41.3332823Z ----------------------------------------------------------------------
2022-05-18T05:14:41.3333432Z Ran 1 test in 0.003s
2022-05-18T05:14:41.3333609Z 
2022-05-18T05:14:41.3333700Z OK (skipped=1)
2022-05-18T05:14:41.3333856Z 
2022-05-18T05:14:41.3333980Z Generating XML reports...
2022-05-18T05:14:41.3376366Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051441.xml
2022-05-18T05:14:42.6017023Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:42.6031859Z 
2022-05-18T05:14:42.6032265Z Running tests...
2022-05-18T05:14:42.6032780Z ----------------------------------------------------------------------
2022-05-18T05:14:42.6056004Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T05:14:42.6057224Z 
2022-05-18T05:14:42.6057519Z ----------------------------------------------------------------------
2022-05-18T05:14:42.6057842Z Ran 1 test in 0.003s
2022-05-18T05:14:42.6058005Z 
2022-05-18T05:14:42.6058113Z OK (skipped=1)
2022-05-18T05:14:42.6058268Z 
2022-05-18T05:14:42.6058398Z Generating XML reports...
2022-05-18T05:14:42.6100857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051442.xml
2022-05-18T05:14:43.8834248Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:43.8849538Z 
2022-05-18T05:14:43.8850046Z Running tests...
2022-05-18T05:14:43.8851086Z ----------------------------------------------------------------------
2022-05-18T05:14:45.5838001Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:45.6219517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63933
2022-05-18T05:14:45.6332827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63934
2022-05-18T05:14:46.8326557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:46.8327095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:46.8327901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:46.8328599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:46.8435255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:46.9338960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:47.1383404Z ok (3.253s)
2022-05-18T05:14:47.1383624Z 
2022-05-18T05:14:47.1384005Z ----------------------------------------------------------------------
2022-05-18T05:14:47.1384322Z Ran 1 test in 3.253s
2022-05-18T05:14:47.1384488Z 
2022-05-18T05:14:47.1384583Z OK
2022-05-18T05:14:47.1384725Z 
2022-05-18T05:14:47.1384863Z Generating XML reports...
2022-05-18T05:14:47.1442664Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051443.xml
2022-05-18T05:14:48.5603892Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:48.5619969Z 
2022-05-18T05:14:48.5620119Z Running tests...
2022-05-18T05:14:48.5620567Z ----------------------------------------------------------------------
2022-05-18T05:14:50.2191949Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:50.2569056Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64048
2022-05-18T05:14:50.2682465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64049
2022-05-18T05:14:51.4369317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:51.4370148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:51.4370939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:51.4371636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:51.4478055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:51.5381474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:51.7732464Z ok (3.211s)
2022-05-18T05:14:51.7732742Z 
2022-05-18T05:14:51.7733160Z ----------------------------------------------------------------------
2022-05-18T05:14:51.7733498Z Ran 1 test in 3.211s
2022-05-18T05:14:51.7733665Z 
2022-05-18T05:14:51.7733756Z OK
2022-05-18T05:14:51.7733899Z 
2022-05-18T05:14:51.7734017Z Generating XML reports...
2022-05-18T05:14:51.7791634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051448.xml
2022-05-18T05:14:53.2102191Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:53.2116912Z 
2022-05-18T05:14:53.2117245Z Running tests...
2022-05-18T05:14:53.2117687Z ----------------------------------------------------------------------
2022-05-18T05:14:53.2139412Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T05:14:53.2139780Z 
2022-05-18T05:14:53.2140079Z ----------------------------------------------------------------------
2022-05-18T05:14:53.2140429Z Ran 1 test in 0.002s
2022-05-18T05:14:53.2140593Z 
2022-05-18T05:14:53.2140703Z OK (skipped=1)
2022-05-18T05:14:53.2140860Z 
2022-05-18T05:14:53.2140966Z Generating XML reports...
2022-05-18T05:14:53.2182046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051453.xml
2022-05-18T05:14:54.4723210Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:54.4737921Z 
2022-05-18T05:14:54.4738145Z Running tests...
2022-05-18T05:14:54.4738880Z ----------------------------------------------------------------------
2022-05-18T05:14:54.4759156Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T05:14:54.4759659Z 
2022-05-18T05:14:54.4759956Z ----------------------------------------------------------------------
2022-05-18T05:14:54.4760290Z Ran 1 test in 0.002s
2022-05-18T05:14:54.4760960Z 
2022-05-18T05:14:54.4761074Z OK (skipped=1)
2022-05-18T05:14:54.4761234Z 
2022-05-18T05:14:54.4761338Z Generating XML reports...
2022-05-18T05:14:54.4801807Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051454.xml
2022-05-18T05:14:55.7415485Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:55.7430189Z 
2022-05-18T05:14:55.7430798Z Running tests...
2022-05-18T05:14:55.7431441Z ----------------------------------------------------------------------
2022-05-18T05:14:57.3929872Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:57.4299535Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64233
2022-05-18T05:14:57.4409211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64234
2022-05-18T05:14:58.6168922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:58.6169478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:58.6170582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:58.6171290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:58.6277548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:58.7185570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:59.9255389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xuxr46k
2022-05-18T05:14:59.9255989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xuxr46k/_remote_module_non_scriptable.py
2022-05-18T05:15:00.0081339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ym19lnp
2022-05-18T05:15:00.0082583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ym19lnp/_remote_module_non_scriptable.py
2022-05-18T05:15:00.6488087Z ok (4.905s)
2022-05-18T05:15:00.6488304Z 
2022-05-18T05:15:00.6488667Z ----------------------------------------------------------------------
2022-05-18T05:15:00.6489008Z Ran 1 test in 4.906s
2022-05-18T05:15:00.6489172Z 
2022-05-18T05:15:00.6489274Z OK
2022-05-18T05:15:00.6489410Z 
2022-05-18T05:15:00.6489785Z Generating XML reports...
2022-05-18T05:15:00.6546780Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051455.xml
2022-05-18T05:15:02.1041808Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:02.1057478Z 
2022-05-18T05:15:02.1057903Z Running tests...
2022-05-18T05:15:02.1058391Z ----------------------------------------------------------------------
2022-05-18T05:15:03.7687698Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:03.8067706Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64354
2022-05-18T05:15:03.8179487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64355
2022-05-18T05:15:05.0124664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:05.0125249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:05.0126045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:05.0126765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:05.0134810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:05.0135784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:06.3553250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3t25yu6
2022-05-18T05:15:06.3554381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3t25yu6/_remote_module_non_scriptable.py
2022-05-18T05:15:06.3856536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_0ubu3a
2022-05-18T05:15:06.3859115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_0ubu3a/_remote_module_non_scriptable.py
2022-05-18T05:15:07.0264153Z ok (4.920s)
2022-05-18T05:15:07.0264401Z 
2022-05-18T05:15:07.0264820Z ----------------------------------------------------------------------
2022-05-18T05:15:07.0265144Z Ran 1 test in 4.921s
2022-05-18T05:15:07.0265309Z 
2022-05-18T05:15:07.0265405Z OK
2022-05-18T05:15:07.0265541Z 
2022-05-18T05:15:07.0265682Z Generating XML reports...
2022-05-18T05:15:07.0323248Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051502.xml
2022-05-18T05:15:08.4714604Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:08.4729958Z 
2022-05-18T05:15:08.4730113Z Running tests...
2022-05-18T05:15:08.4730735Z ----------------------------------------------------------------------
2022-05-18T05:15:10.1386044Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:10.1765723Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64475
2022-05-18T05:15:10.1877642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64476
2022-05-18T05:15:11.3557958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:11.3558515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:11.3559322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:11.3560020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:11.3668689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:11.4572803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:14.0972194Z ok (5.624s)
2022-05-18T05:15:14.0972423Z 
2022-05-18T05:15:14.0972795Z ----------------------------------------------------------------------
2022-05-18T05:15:14.0973403Z Ran 1 test in 5.624s
2022-05-18T05:15:14.0973591Z 
2022-05-18T05:15:14.0977131Z OK
2022-05-18T05:15:14.0977616Z 
2022-05-18T05:15:14.0977908Z Generating XML reports...
2022-05-18T05:15:14.1030946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051508.xml
2022-05-18T05:15:15.5244261Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:15.5259596Z 
2022-05-18T05:15:15.5259960Z Running tests...
2022-05-18T05:15:15.5260473Z ----------------------------------------------------------------------
2022-05-18T05:15:17.1533772Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:17.1908204Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64593
2022-05-18T05:15:17.2018977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64594
2022-05-18T05:15:18.4040532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:18.4041098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:18.4042111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:18.4042788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:18.4049380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:18.4050593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:21.1111585Z ok (5.585s)
2022-05-18T05:15:21.1111811Z 
2022-05-18T05:15:21.1112188Z ----------------------------------------------------------------------
2022-05-18T05:15:21.1112532Z Ran 1 test in 5.585s
2022-05-18T05:15:21.1112676Z 
2022-05-18T05:15:21.1112785Z OK
2022-05-18T05:15:21.1112921Z 
2022-05-18T05:15:21.1113059Z Generating XML reports...
2022-05-18T05:15:21.1169399Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051515.xml
2022-05-18T05:15:22.5558540Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:22.5574484Z 
2022-05-18T05:15:22.5574875Z Running tests...
2022-05-18T05:15:22.5575383Z ----------------------------------------------------------------------
2022-05-18T05:15:24.1986125Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:24.2106048Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.653s)
2022-05-18T05:15:24.2106650Z 
2022-05-18T05:15:24.2106903Z ----------------------------------------------------------------------
2022-05-18T05:15:24.2107237Z Ran 1 test in 1.653s
2022-05-18T05:15:24.2107411Z 
2022-05-18T05:15:24.2107522Z OK (skipped=1)
2022-05-18T05:15:24.2107683Z 
2022-05-18T05:15:24.2107808Z Generating XML reports...
2022-05-18T05:15:24.2147414Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051522.xml
2022-05-18T05:15:25.5923804Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:25.5938573Z 
2022-05-18T05:15:25.5938994Z Running tests...
2022-05-18T05:15:25.5939490Z ----------------------------------------------------------------------
2022-05-18T05:15:27.2266813Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:27.2383163Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.644s)
2022-05-18T05:15:27.2383756Z 
2022-05-18T05:15:27.2384018Z ----------------------------------------------------------------------
2022-05-18T05:15:27.2384352Z Ran 1 test in 1.644s
2022-05-18T05:15:27.2384517Z 
2022-05-18T05:15:27.2384626Z OK (skipped=1)
2022-05-18T05:15:27.2384780Z 
2022-05-18T05:15:27.2384906Z Generating XML reports...
2022-05-18T05:15:27.2421514Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051525.xml
2022-05-18T05:15:28.6280027Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:28.6295787Z 
2022-05-18T05:15:28.6296041Z Running tests...
2022-05-18T05:15:28.6296483Z ----------------------------------------------------------------------
2022-05-18T05:15:30.2806770Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:30.3178760Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64783
2022-05-18T05:15:30.3290466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64784
2022-05-18T05:15:31.5005350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:31.5005917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:31.5006707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:31.5007393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:31.5014380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:31.5015645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:31.7339644Z skip: Need at least 4 CUDA devices (3.104s)
2022-05-18T05:15:31.7339896Z 
2022-05-18T05:15:31.7340284Z ----------------------------------------------------------------------
2022-05-18T05:15:31.7340623Z Ran 1 test in 3.104s
2022-05-18T05:15:31.7340792Z 
2022-05-18T05:15:31.7340892Z OK (skipped=1)
2022-05-18T05:15:31.7341051Z 
2022-05-18T05:15:31.7341180Z Generating XML reports...
2022-05-18T05:15:31.7398413Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051528.xml
2022-05-18T05:15:33.1413435Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:33.1427611Z 
2022-05-18T05:15:33.1428026Z Running tests...
2022-05-18T05:15:33.1428533Z ----------------------------------------------------------------------
2022-05-18T05:15:34.7428767Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:34.7798090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64898
2022-05-18T05:15:34.7908782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64899
2022-05-18T05:15:35.9732507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:35.9733082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:35.9733840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:35.9734539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:35.9841225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:36.0748383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:36.2957897Z skip: Need at least 4 CUDA devices (3.153s)
2022-05-18T05:15:36.2958163Z 
2022-05-18T05:15:36.2958556Z ----------------------------------------------------------------------
2022-05-18T05:15:36.2958896Z Ran 1 test in 3.153s
2022-05-18T05:15:36.2959061Z 
2022-05-18T05:15:36.2959156Z OK (skipped=1)
2022-05-18T05:15:36.2959321Z 
2022-05-18T05:15:36.2959451Z Generating XML reports...
2022-05-18T05:15:36.3016354Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051533.xml
2022-05-18T05:15:37.7137356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:37.7151574Z 
2022-05-18T05:15:37.7151816Z Running tests...
2022-05-18T05:15:37.7152265Z ----------------------------------------------------------------------
2022-05-18T05:15:39.3271923Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:39.3641447Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65013
2022-05-18T05:15:39.3754707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65014
2022-05-18T05:15:40.5707153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:40.5707709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:40.5708487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:40.5709189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:40.5816686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:40.6719227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:40.6832389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:15:40.6832903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:15:40.6833587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:15:40.6834281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:15:40.8803869Z ok (3.165s)
2022-05-18T05:15:40.8804121Z 
2022-05-18T05:15:40.8804507Z ----------------------------------------------------------------------
2022-05-18T05:15:40.8804843Z Ran 1 test in 3.165s
2022-05-18T05:15:40.8805007Z 
2022-05-18T05:15:40.8805118Z OK
2022-05-18T05:15:40.8805254Z 
2022-05-18T05:15:40.8805368Z Generating XML reports...
2022-05-18T05:15:40.8863936Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051537.xml
2022-05-18T05:15:42.2999256Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:42.3014357Z 
2022-05-18T05:15:42.3014830Z Running tests...
2022-05-18T05:15:42.3015336Z ----------------------------------------------------------------------
2022-05-18T05:15:43.9705450Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:44.0075150Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65138
2022-05-18T05:15:44.0187271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65139
2022-05-18T05:15:45.2298972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:45.2299550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:45.2300360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:45.2301068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:45.2407757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:45.3310354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:45.3518297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:15:45.3518811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:15:45.3519482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:15:45.3520173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:15:45.6237478Z ok (3.322s)
2022-05-18T05:15:45.6237709Z 
2022-05-18T05:15:45.6238105Z ----------------------------------------------------------------------
2022-05-18T05:15:45.6238450Z Ran 1 test in 3.322s
2022-05-18T05:15:45.6238616Z 
2022-05-18T05:15:45.6238691Z OK
2022-05-18T05:15:45.6238825Z 
2022-05-18T05:15:45.6238958Z Generating XML reports...
2022-05-18T05:15:45.6295876Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051542.xml
2022-05-18T05:15:47.0260790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:47.0274954Z 
2022-05-18T05:15:47.0275093Z Running tests...
2022-05-18T05:15:47.0275614Z ----------------------------------------------------------------------
2022-05-18T05:15:48.6338690Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:48.6708640Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65263
2022-05-18T05:15:48.6820708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65264
2022-05-18T05:15:49.8643645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:49.8644198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:49.8645015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:49.8645689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:49.8753499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:49.9658383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:49.9868046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:15:49.9868566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:15:49.9869235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:15:49.9869927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:15:50.1872986Z ok (3.159s)
2022-05-18T05:15:50.1873210Z 
2022-05-18T05:15:50.1873606Z ----------------------------------------------------------------------
2022-05-18T05:15:50.1873946Z Ran 1 test in 3.160s
2022-05-18T05:15:50.1874091Z 
2022-05-18T05:15:50.1874193Z OK
2022-05-18T05:15:50.1874643Z 
2022-05-18T05:15:50.1874840Z Generating XML reports...
2022-05-18T05:15:50.1932177Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051547.xml
2022-05-18T05:15:51.6205326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:51.6220937Z 
2022-05-18T05:15:51.6221329Z Running tests...
2022-05-18T05:15:51.6221777Z ----------------------------------------------------------------------
2022-05-18T05:15:53.2674120Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:53.3055848Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65388
2022-05-18T05:15:53.3170123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65389
2022-05-18T05:15:54.4749529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:54.4750118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:54.4750912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:54.4751847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:54.4861476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:54.5763464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:54.5879581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:15:54.5880268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:15:54.5881170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:15:54.5881935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:15:54.8221684Z ok (3.200s)
2022-05-18T05:15:54.8221928Z 
2022-05-18T05:15:54.8222417Z ----------------------------------------------------------------------
2022-05-18T05:15:54.8222905Z Ran 1 test in 3.200s
2022-05-18T05:15:54.8223073Z 
2022-05-18T05:15:54.8223175Z OK
2022-05-18T05:15:54.8223313Z 
2022-05-18T05:15:54.8223431Z Generating XML reports...
2022-05-18T05:15:54.8279768Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051551.xml
2022-05-18T05:15:56.2498330Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:56.2513610Z 
2022-05-18T05:15:56.2513991Z Running tests...
2022-05-18T05:15:56.2514431Z ----------------------------------------------------------------------
2022-05-18T05:15:57.8955013Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:57.9325902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65513
2022-05-18T05:15:57.9436927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65514
2022-05-18T05:15:59.1578690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:59.1579202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:59.1579996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:59.1580694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:59.1687729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:59.2594265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:59.4486507Z skip: Skipped due to small world size. (3.197s)
2022-05-18T05:15:59.4486777Z 
2022-05-18T05:15:59.4487145Z ----------------------------------------------------------------------
2022-05-18T05:15:59.4487489Z Ran 1 test in 3.197s
2022-05-18T05:15:59.4487651Z 
2022-05-18T05:15:59.4487752Z OK (skipped=1)
2022-05-18T05:15:59.4487911Z 
2022-05-18T05:15:59.4488036Z Generating XML reports...
2022-05-18T05:15:59.4548480Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051556.xml
2022-05-18T05:16:00.8695815Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:00.8710994Z 
2022-05-18T05:16:00.8711454Z Running tests...
2022-05-18T05:16:00.8711873Z ----------------------------------------------------------------------
2022-05-18T05:16:02.5138920Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:02.5522232Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65628
2022-05-18T05:16:02.5639754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65629
2022-05-18T05:16:03.7312412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:03.7312991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:03.7313765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:03.7314463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:03.7421973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:03.8327233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:03.9686156Z skip: Skipped due to small world size. (3.097s)
2022-05-18T05:16:03.9686419Z 
2022-05-18T05:16:03.9686826Z ----------------------------------------------------------------------
2022-05-18T05:16:03.9687161Z Ran 1 test in 3.097s
2022-05-18T05:16:03.9687322Z 
2022-05-18T05:16:03.9687412Z OK (skipped=1)
2022-05-18T05:16:03.9687568Z 
2022-05-18T05:16:03.9687694Z Generating XML reports...
2022-05-18T05:16:03.9754539Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051600.xml
2022-05-18T05:16:05.4064643Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:05.4079393Z 
2022-05-18T05:16:05.4079592Z Running tests...
2022-05-18T05:16:05.4080195Z ----------------------------------------------------------------------
2022-05-18T05:16:07.1040633Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:07.1420507Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65743
2022-05-18T05:16:07.1533218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65744
2022-05-18T05:16:08.3611099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:08.3611656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:08.3612434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:08.3613136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:08.3619928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:08.3621327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:08.5582222Z skip: Skipped due to small world size. (3.150s)
2022-05-18T05:16:08.5582464Z 
2022-05-18T05:16:08.5582836Z ----------------------------------------------------------------------
2022-05-18T05:16:08.5583167Z Ran 1 test in 3.150s
2022-05-18T05:16:08.5583329Z 
2022-05-18T05:16:08.5583446Z OK (skipped=1)
2022-05-18T05:16:08.5583602Z 
2022-05-18T05:16:08.5583725Z Generating XML reports...
2022-05-18T05:16:08.5641108Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051605.xml
2022-05-18T05:16:09.9946281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:09.9961731Z 
2022-05-18T05:16:09.9962003Z Running tests...
2022-05-18T05:16:09.9962717Z ----------------------------------------------------------------------
2022-05-18T05:16:11.6433709Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:11.6807953Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65858
2022-05-18T05:16:11.6921123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65859
2022-05-18T05:16:12.8764730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:12.8765781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:12.8767215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:12.8768574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:12.8876483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:12.9780828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:13.1970665Z skip: Skipped due to small world size. (3.201s)
2022-05-18T05:16:13.1970970Z 
2022-05-18T05:16:13.1971454Z ----------------------------------------------------------------------
2022-05-18T05:16:13.1972067Z Ran 1 test in 3.201s
2022-05-18T05:16:13.1972277Z 
2022-05-18T05:16:13.1972391Z OK (skipped=1)
2022-05-18T05:16:13.1972553Z 
2022-05-18T05:16:13.1972671Z Generating XML reports...
2022-05-18T05:16:13.2031425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051609.xml
2022-05-18T05:16:14.6495715Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:14.6510564Z 
2022-05-18T05:16:14.6511012Z Running tests...
2022-05-18T05:16:14.6511528Z ----------------------------------------------------------------------
2022-05-18T05:16:16.2968297Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:16.3348461Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65973
2022-05-18T05:16:16.3459470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65974
2022-05-18T05:16:17.5638401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:17.5638978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:17.5639778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:17.5640451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:17.5747216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:17.6650026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:17.8510712Z ok (3.200s)
2022-05-18T05:16:17.8510980Z 
2022-05-18T05:16:17.8511338Z ----------------------------------------------------------------------
2022-05-18T05:16:17.8511674Z Ran 1 test in 3.200s
2022-05-18T05:16:17.8511850Z 
2022-05-18T05:16:17.8511941Z OK
2022-05-18T05:16:17.8512073Z 
2022-05-18T05:16:17.8512208Z Generating XML reports...
2022-05-18T05:16:17.8568985Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051614.xml
2022-05-18T05:16:19.2824902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:19.2840837Z 
2022-05-18T05:16:19.2841094Z Running tests...
2022-05-18T05:16:19.2841522Z ----------------------------------------------------------------------
2022-05-18T05:16:20.9436113Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:20.9815651Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66092
2022-05-18T05:16:20.9927502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66093
2022-05-18T05:16:22.2387726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:22.2388620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:22.2389631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:22.2390454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:22.2496732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:22.3400433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:22.5980251Z ok (3.314s)
2022-05-18T05:16:22.5980441Z 
2022-05-18T05:16:22.5980845Z ----------------------------------------------------------------------
2022-05-18T05:16:22.5981434Z Ran 1 test in 3.314s
2022-05-18T05:16:22.5981606Z 
2022-05-18T05:16:22.5981701Z OK
2022-05-18T05:16:22.5981848Z 
2022-05-18T05:16:22.5981977Z Generating XML reports...
2022-05-18T05:16:22.6039356Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051619.xml
2022-05-18T05:16:24.0244343Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:24.0259755Z 
2022-05-18T05:16:24.0260168Z Running tests...
2022-05-18T05:16:24.0260659Z ----------------------------------------------------------------------
2022-05-18T05:16:24.0284623Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-05-18T05:16:24.0285072Z 
2022-05-18T05:16:24.0285488Z ----------------------------------------------------------------------
2022-05-18T05:16:24.0285837Z Ran 1 test in 0.003s
2022-05-18T05:16:24.0286003Z 
2022-05-18T05:16:24.0286116Z OK (skipped=1)
2022-05-18T05:16:24.0286273Z 
2022-05-18T05:16:24.0286397Z Generating XML reports...
2022-05-18T05:16:24.0328489Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051624.xml
2022-05-18T05:16:25.3102708Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:25.3117639Z 
2022-05-18T05:16:25.3118059Z Running tests...
2022-05-18T05:16:25.3118562Z ----------------------------------------------------------------------
2022-05-18T05:16:26.9655800Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:27.0034202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66246
2022-05-18T05:16:27.0147197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66247
2022-05-18T05:16:28.2101336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:28.2101928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:28.2102718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:28.2103406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:28.2212114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:28.3113748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:28.5200551Z ok (3.208s)
2022-05-18T05:16:28.5200739Z 
2022-05-18T05:16:28.5201111Z ----------------------------------------------------------------------
2022-05-18T05:16:28.5201455Z Ran 1 test in 3.208s
2022-05-18T05:16:28.5201630Z 
2022-05-18T05:16:28.5201718Z OK
2022-05-18T05:16:28.5201858Z 
2022-05-18T05:16:28.5201991Z Generating XML reports...
2022-05-18T05:16:28.5259031Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051625.xml
2022-05-18T05:16:29.9613601Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:29.9629018Z 
2022-05-18T05:16:29.9629422Z Running tests...
2022-05-18T05:16:29.9629890Z ----------------------------------------------------------------------
2022-05-18T05:16:31.6345666Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:31.6724000Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66365
2022-05-18T05:16:31.6835943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66366
2022-05-18T05:16:32.8060620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:32.8061182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:32.8061961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:32.8062649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:32.8169761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:32.9074221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:33.0889728Z ok (3.126s)
2022-05-18T05:16:33.0890386Z 
2022-05-18T05:16:33.0890895Z ----------------------------------------------------------------------
2022-05-18T05:16:33.0891239Z Ran 1 test in 3.126s
2022-05-18T05:16:33.0891402Z 
2022-05-18T05:16:33.0891496Z OK
2022-05-18T05:16:33.0891612Z 
2022-05-18T05:16:33.0891763Z Generating XML reports...
2022-05-18T05:16:33.0949621Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051629.xml
2022-05-18T05:16:34.5338632Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:34.5353780Z 
2022-05-18T05:16:34.5354028Z Running tests...
2022-05-18T05:16:34.5354459Z ----------------------------------------------------------------------
2022-05-18T05:16:34.5379667Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-05-18T05:16:34.5379976Z 
2022-05-18T05:16:34.5380236Z ----------------------------------------------------------------------
2022-05-18T05:16:34.5380561Z Ran 1 test in 0.003s
2022-05-18T05:16:34.5380725Z 
2022-05-18T05:16:34.5380842Z OK (skipped=1)
2022-05-18T05:16:34.5380997Z 
2022-05-18T05:16:34.5381123Z Generating XML reports...
2022-05-18T05:16:34.5424153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051634.xml
2022-05-18T05:16:35.8304535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:35.8319547Z 
2022-05-18T05:16:35.8319770Z Running tests...
2022-05-18T05:16:35.8320224Z ----------------------------------------------------------------------
2022-05-18T05:16:35.8345256Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-05-18T05:16:35.8345571Z 
2022-05-18T05:16:35.8345856Z ----------------------------------------------------------------------
2022-05-18T05:16:35.8346174Z Ran 1 test in 0.003s
2022-05-18T05:16:35.8346339Z 
2022-05-18T05:16:35.8346449Z OK (skipped=1)
2022-05-18T05:16:35.8346607Z 
2022-05-18T05:16:35.8346733Z Generating XML reports...
2022-05-18T05:16:35.8389575Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051635.xml
2022-05-18T05:16:37.1194233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:37.1209226Z 
2022-05-18T05:16:37.1209497Z Running tests...
2022-05-18T05:16:37.1210571Z ----------------------------------------------------------------------
2022-05-18T05:16:38.7680243Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:38.8062744Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66554
2022-05-18T05:16:38.8174524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66555
2022-05-18T05:16:39.9815245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:39.9815819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:39.9816589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:39.9817289Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:39.9925567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:40.0826104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:40.3227004Z ok (3.201s)
2022-05-18T05:16:40.3227227Z 
2022-05-18T05:16:40.3227619Z ----------------------------------------------------------------------
2022-05-18T05:16:40.3227966Z Ran 1 test in 3.202s
2022-05-18T05:16:40.3228114Z 
2022-05-18T05:16:40.3228211Z OK
2022-05-18T05:16:40.3228352Z 
2022-05-18T05:16:40.3228494Z Generating XML reports...
2022-05-18T05:16:40.3285398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051637.xml
2022-05-18T05:16:41.7463973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:41.7479107Z 
2022-05-18T05:16:41.7479547Z Running tests...
2022-05-18T05:16:41.7480033Z ----------------------------------------------------------------------
2022-05-18T05:16:43.4246549Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:43.4618390Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66673
2022-05-18T05:16:43.4734169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66674
2022-05-18T05:16:44.6903457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:44.6904020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:44.6904803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:44.6905731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:44.6913708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:44.6914202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:44.8781489Z ok (3.130s)
2022-05-18T05:16:44.8781940Z 
2022-05-18T05:16:44.8782352Z ----------------------------------------------------------------------
2022-05-18T05:16:44.8782692Z Ran 1 test in 3.130s
2022-05-18T05:16:44.8782855Z 
2022-05-18T05:16:44.8782950Z OK
2022-05-18T05:16:44.8783064Z 
2022-05-18T05:16:44.8783201Z Generating XML reports...
2022-05-18T05:16:44.8840376Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051641.xml
2022-05-18T05:16:46.3320179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:46.3335864Z 
2022-05-18T05:16:46.3336188Z Running tests...
2022-05-18T05:16:46.3336660Z ----------------------------------------------------------------------
2022-05-18T05:16:48.0028946Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:48.0411130Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66792
2022-05-18T05:16:48.0524650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66793
2022-05-18T05:16:49.2537767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:49.2538348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:49.2539141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:49.2539862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:49.2546683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:49.2547397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:49.4572238Z ok (3.123s)
2022-05-18T05:16:49.4574057Z 
2022-05-18T05:16:49.4574836Z ----------------------------------------------------------------------
2022-05-18T05:16:49.4575221Z Ran 1 test in 3.124s
2022-05-18T05:16:49.4575390Z 
2022-05-18T05:16:49.4575504Z OK
2022-05-18T05:16:49.4575624Z 
2022-05-18T05:16:49.4575765Z Generating XML reports...
2022-05-18T05:16:49.4630711Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051646.xml
2022-05-18T05:16:50.9054604Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:50.9069818Z 
2022-05-18T05:16:50.9070130Z Running tests...
2022-05-18T05:16:50.9070582Z ----------------------------------------------------------------------
2022-05-18T05:16:52.5709685Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:52.6091537Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66907
2022-05-18T05:16:52.6203562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66908
2022-05-18T05:16:53.7944473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:53.7945047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:53.7945847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:53.7946522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:53.8053838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:53.8955886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:54.1254402Z ok (3.218s)
2022-05-18T05:16:54.1254594Z 
2022-05-18T05:16:54.1255365Z ----------------------------------------------------------------------
2022-05-18T05:16:54.1255740Z Ran 1 test in 3.218s
2022-05-18T05:16:54.1255887Z 
2022-05-18T05:16:54.1255984Z OK
2022-05-18T05:16:54.1256130Z 
2022-05-18T05:16:54.1256266Z Generating XML reports...
2022-05-18T05:16:54.1313771Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051650.xml
2022-05-18T05:16:55.5563498Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:55.5581846Z 
2022-05-18T05:16:55.5582124Z Running tests...
2022-05-18T05:16:55.5582581Z ----------------------------------------------------------------------
2022-05-18T05:16:55.5603246Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T05:16:55.5603558Z 
2022-05-18T05:16:55.5603831Z ----------------------------------------------------------------------
2022-05-18T05:16:55.5604463Z Ran 1 test in 0.002s
2022-05-18T05:16:55.5604626Z 
2022-05-18T05:16:55.5604735Z OK (skipped=1)
2022-05-18T05:16:55.5604891Z 
2022-05-18T05:16:55.5605019Z Generating XML reports...
2022-05-18T05:16:55.5647622Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051655.xml
2022-05-18T05:16:56.8415203Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:56.8430399Z 
2022-05-18T05:16:56.8430550Z Running tests...
2022-05-18T05:16:56.8431229Z ----------------------------------------------------------------------
2022-05-18T05:16:56.8452794Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T05:16:56.8453115Z 
2022-05-18T05:16:56.8453395Z ----------------------------------------------------------------------
2022-05-18T05:16:56.8453727Z Ran 1 test in 0.002s
2022-05-18T05:16:56.8453906Z 
2022-05-18T05:16:56.8453996Z OK (skipped=1)
2022-05-18T05:16:56.8454151Z 
2022-05-18T05:16:56.8454276Z Generating XML reports...
2022-05-18T05:16:56.8496354Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051656.xml
2022-05-18T05:16:58.1224960Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:58.1239901Z 
2022-05-18T05:16:58.1240334Z Running tests...
2022-05-18T05:16:58.1240831Z ----------------------------------------------------------------------
2022-05-18T05:16:59.7748228Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:59.8127733Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67096
2022-05-18T05:16:59.8240787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67097
2022-05-18T05:17:01.0292319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:01.0292880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:01.0293685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:01.0294374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:01.0402011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:01.1305037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:01.1416326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:17:01.1416872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:17:01.1417686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:17:01.1418531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:17:01.4290814Z ok (3.305s)
2022-05-18T05:17:01.4291097Z 
2022-05-18T05:17:01.4291647Z ----------------------------------------------------------------------
2022-05-18T05:17:01.4291992Z Ran 1 test in 3.305s
2022-05-18T05:17:01.4292156Z 
2022-05-18T05:17:01.4292231Z OK
2022-05-18T05:17:01.4292375Z 
2022-05-18T05:17:01.4292512Z Generating XML reports...
2022-05-18T05:17:01.4348515Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051658.xml
2022-05-18T05:17:02.8703737Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:02.8718354Z 
2022-05-18T05:17:02.8718592Z Running tests...
2022-05-18T05:17:02.8719333Z ----------------------------------------------------------------------
2022-05-18T05:17:04.5279411Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:04.5660670Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67221
2022-05-18T05:17:04.5774610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67222
2022-05-18T05:17:05.7912576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:05.7913152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:05.7913924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:05.7914619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:05.8020331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:05.8927650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:06.0826056Z skip: Skipped due to small world size. (3.210s)
2022-05-18T05:17:06.0826418Z 
2022-05-18T05:17:06.0826999Z ----------------------------------------------------------------------
2022-05-18T05:17:06.0827345Z Ran 1 test in 3.211s
2022-05-18T05:17:06.0827490Z 
2022-05-18T05:17:06.0827603Z OK (skipped=1)
2022-05-18T05:17:06.0827761Z 
2022-05-18T05:17:06.0827891Z Generating XML reports...
2022-05-18T05:17:06.0885513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051702.xml
2022-05-18T05:17:07.5181036Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:07.5196628Z 
2022-05-18T05:17:07.5197355Z Running tests...
2022-05-18T05:17:07.5197948Z ----------------------------------------------------------------------
2022-05-18T05:17:09.1870029Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:09.2248891Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67336
2022-05-18T05:17:09.2362491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67337
2022-05-18T05:17:10.3902957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:10.3903849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:10.3904643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:10.3905553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:10.3911263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:10.3912328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:10.5408697Z ok (3.021s)
2022-05-18T05:17:10.5409083Z 
2022-05-18T05:17:10.5410291Z ----------------------------------------------------------------------
2022-05-18T05:17:10.5411030Z Ran 1 test in 3.021s
2022-05-18T05:17:10.5411388Z 
2022-05-18T05:17:10.5411568Z OK
2022-05-18T05:17:10.5411825Z 
2022-05-18T05:17:10.5411996Z Generating XML reports...
2022-05-18T05:17:10.5468430Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051707.xml
2022-05-18T05:17:11.9730350Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:11.9745588Z 
2022-05-18T05:17:11.9746033Z Running tests...
2022-05-18T05:17:11.9746472Z ----------------------------------------------------------------------
2022-05-18T05:17:13.6398210Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:13.6772082Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67451
2022-05-18T05:17:13.6884600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67452
2022-05-18T05:17:14.8821408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:14.8821968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:14.8822755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:14.8823478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:14.8931931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:14.9834538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:15.1935039Z ok (3.219s)
2022-05-18T05:17:15.1935286Z 
2022-05-18T05:17:15.1935664Z ----------------------------------------------------------------------
2022-05-18T05:17:15.1936000Z Ran 1 test in 3.219s
2022-05-18T05:17:15.1936166Z 
2022-05-18T05:17:15.1936265Z OK
2022-05-18T05:17:15.1936382Z 
2022-05-18T05:17:15.1936523Z Generating XML reports...
2022-05-18T05:17:15.1992599Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051711.xml
2022-05-18T05:17:16.6155488Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:16.6170243Z 
2022-05-18T05:17:16.6170389Z Running tests...
2022-05-18T05:17:16.6171111Z ----------------------------------------------------------------------
2022-05-18T05:17:18.2578257Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:18.2949839Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67566
2022-05-18T05:17:18.3061730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67567
2022-05-18T05:17:19.5160867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:19.5161425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:19.5162232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:19.5162928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:19.5170249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:19.5171235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:19.7111766Z ok (3.094s)
2022-05-18T05:17:19.7112147Z 
2022-05-18T05:17:19.7112579Z ----------------------------------------------------------------------
2022-05-18T05:17:19.7112904Z Ran 1 test in 3.094s
2022-05-18T05:17:19.7113072Z 
2022-05-18T05:17:19.7113174Z OK
2022-05-18T05:17:19.7113312Z 
2022-05-18T05:17:19.7113451Z Generating XML reports...
2022-05-18T05:17:19.7171832Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051716.xml
2022-05-18T05:17:21.1244546Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:21.1259648Z 
2022-05-18T05:17:21.1260130Z Running tests...
2022-05-18T05:17:21.1260657Z ----------------------------------------------------------------------
2022-05-18T05:17:22.7403655Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:22.7776574Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67681
2022-05-18T05:17:22.7889304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67682
2022-05-18T05:17:23.9628351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:23.9629417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:23.9630918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:23.9631760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:23.9738458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:24.0641330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:24.2942762Z ok (3.168s)
2022-05-18T05:17:24.2942995Z 
2022-05-18T05:17:24.2943399Z ----------------------------------------------------------------------
2022-05-18T05:17:24.2943739Z Ran 1 test in 3.168s
2022-05-18T05:17:24.2943903Z 
2022-05-18T05:17:24.2943995Z OK
2022-05-18T05:17:24.2944135Z 
2022-05-18T05:17:24.2944251Z Generating XML reports...
2022-05-18T05:17:24.3000273Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051721.xml
2022-05-18T05:17:25.7145025Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:25.7159699Z 
2022-05-18T05:17:25.7160053Z Running tests...
2022-05-18T05:17:25.7160501Z ----------------------------------------------------------------------
2022-05-18T05:17:27.3704940Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:27.4084095Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67800
2022-05-18T05:17:27.4197464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67801
2022-05-18T05:17:28.6609422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:28.6610238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:28.6611022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:28.6611712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:28.6619298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:28.6620625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:28.9248605Z ok (3.209s)
2022-05-18T05:17:28.9248937Z 
2022-05-18T05:17:28.9249953Z ----------------------------------------------------------------------
2022-05-18T05:17:28.9250561Z Ran 1 test in 3.209s
2022-05-18T05:17:28.9250864Z 
2022-05-18T05:17:28.9251030Z OK
2022-05-18T05:17:28.9251285Z 
2022-05-18T05:17:28.9251524Z Generating XML reports...
2022-05-18T05:17:28.9309139Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051725.xml
2022-05-18T05:17:30.3555886Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:30.3570947Z 
2022-05-18T05:17:30.3571173Z Running tests...
2022-05-18T05:17:30.3572088Z ----------------------------------------------------------------------
2022-05-18T05:17:32.0108350Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:32.0479012Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67919
2022-05-18T05:17:32.0589999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67920
2022-05-18T05:17:33.2230219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:33.2230926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:33.2231948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:33.2232640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:33.2339504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:33.3243138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:33.5640210Z ok (3.207s)
2022-05-18T05:17:33.5640532Z 
2022-05-18T05:17:33.5641058Z ----------------------------------------------------------------------
2022-05-18T05:17:33.5641427Z Ran 1 test in 3.207s
2022-05-18T05:17:33.5641600Z 
2022-05-18T05:17:33.5641694Z OK
2022-05-18T05:17:33.5641830Z 
2022-05-18T05:17:33.5641961Z Generating XML reports...
2022-05-18T05:17:33.5699793Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051730.xml
2022-05-18T05:17:34.9916608Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:34.9932172Z 
2022-05-18T05:17:34.9932393Z Running tests...
2022-05-18T05:17:34.9932810Z ----------------------------------------------------------------------
2022-05-18T05:17:34.9953449Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T05:17:34.9953743Z 
2022-05-18T05:17:34.9954057Z ----------------------------------------------------------------------
2022-05-18T05:17:34.9954385Z Ran 1 test in 0.002s
2022-05-18T05:17:34.9954556Z 
2022-05-18T05:17:34.9954673Z OK (skipped=1)
2022-05-18T05:17:34.9954829Z 
2022-05-18T05:17:34.9954935Z Generating XML reports...
2022-05-18T05:17:34.9997251Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051734.xml
2022-05-18T05:17:36.2755458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:36.2770369Z 
2022-05-18T05:17:36.2770761Z Running tests...
2022-05-18T05:17:36.2771230Z ----------------------------------------------------------------------
2022-05-18T05:17:36.2791689Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T05:17:36.2792003Z 
2022-05-18T05:17:36.2792548Z ----------------------------------------------------------------------
2022-05-18T05:17:36.2792889Z Ran 1 test in 0.002s
2022-05-18T05:17:36.2793033Z 
2022-05-18T05:17:36.2793141Z OK (skipped=1)
2022-05-18T05:17:36.2793293Z 
2022-05-18T05:17:36.2793418Z Generating XML reports...
2022-05-18T05:17:36.2835743Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051736.xml
2022-05-18T05:17:37.5553326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:37.5568194Z 
2022-05-18T05:17:37.5568447Z Running tests...
2022-05-18T05:17:37.5568891Z ----------------------------------------------------------------------
2022-05-18T05:17:37.5592313Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T05:17:37.5592673Z 
2022-05-18T05:17:37.5592968Z ----------------------------------------------------------------------
2022-05-18T05:17:37.5593299Z Ran 1 test in 0.002s
2022-05-18T05:17:37.5593462Z 
2022-05-18T05:17:37.5593589Z OK (skipped=1)
2022-05-18T05:17:37.5593747Z 
2022-05-18T05:17:37.5593852Z Generating XML reports...
2022-05-18T05:17:37.5636009Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051737.xml
2022-05-18T05:17:38.8408003Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:38.8423441Z 
2022-05-18T05:17:38.8423670Z Running tests...
2022-05-18T05:17:38.8424116Z ----------------------------------------------------------------------
2022-05-18T05:17:40.5126918Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:40.5504712Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68143
2022-05-18T05:17:40.5616827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68144
2022-05-18T05:17:41.7223867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:41.7224415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:41.7225199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:41.7225905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:41.7332661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:41.8235583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:42.0666690Z ok (3.224s)
2022-05-18T05:17:42.0666870Z 
2022-05-18T05:17:42.0667254Z ----------------------------------------------------------------------
2022-05-18T05:17:42.0667579Z Ran 1 test in 3.224s
2022-05-18T05:17:42.0667745Z 
2022-05-18T05:17:42.0667840Z OK
2022-05-18T05:17:42.0667975Z 
2022-05-18T05:17:42.0668117Z Generating XML reports...
2022-05-18T05:17:42.0725291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051738.xml
2022-05-18T05:17:43.4971826Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:43.4987378Z 
2022-05-18T05:17:43.4987598Z Running tests...
2022-05-18T05:17:43.4988031Z ----------------------------------------------------------------------
2022-05-18T05:17:45.1548900Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:45.1922226Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68262
2022-05-18T05:17:45.2033422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68263
2022-05-18T05:17:46.4054610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:46.4055388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:46.4056178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:46.4056874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:46.4115688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:46.4116254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:46.6081066Z ok (3.109s)
2022-05-18T05:17:46.6081286Z 
2022-05-18T05:17:46.6081663Z ----------------------------------------------------------------------
2022-05-18T05:17:46.6082002Z Ran 1 test in 3.109s
2022-05-18T05:17:46.6082166Z 
2022-05-18T05:17:46.6082265Z OK
2022-05-18T05:17:46.6082380Z 
2022-05-18T05:17:46.6082513Z Generating XML reports...
2022-05-18T05:17:46.6139229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051743.xml
2022-05-18T05:17:48.0421654Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:48.0436567Z 
2022-05-18T05:17:48.0437028Z Running tests...
2022-05-18T05:17:48.0437548Z ----------------------------------------------------------------------
2022-05-18T05:17:49.6887656Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:49.7259972Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68377
2022-05-18T05:17:49.7370038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68378
2022-05-18T05:17:50.9101436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:50.9102035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:50.9102839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:50.9103539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:50.9212096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:51.0113641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:51.2421625Z ok (3.198s)
2022-05-18T05:17:51.2421874Z 
2022-05-18T05:17:51.2422262Z ----------------------------------------------------------------------
2022-05-18T05:17:51.2422599Z Ran 1 test in 3.198s
2022-05-18T05:17:51.2422770Z 
2022-05-18T05:17:51.2422867Z OK
2022-05-18T05:17:51.2422984Z 
2022-05-18T05:17:51.2423120Z Generating XML reports...
2022-05-18T05:17:51.2479399Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051748.xml
2022-05-18T05:17:52.6767274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:52.6783004Z 
2022-05-18T05:17:52.6783215Z Running tests...
2022-05-18T05:17:52.6783666Z ----------------------------------------------------------------------
2022-05-18T05:17:54.3383362Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:54.3763667Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68496
2022-05-18T05:17:54.3876251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68497
2022-05-18T05:17:55.5605034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:55.5605613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:55.5606634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:55.5607369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:55.5715742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:55.6618693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:55.8927642Z ok (3.214s)
2022-05-18T05:17:55.8927857Z 
2022-05-18T05:17:55.8928237Z ----------------------------------------------------------------------
2022-05-18T05:17:55.8928578Z Ran 1 test in 3.214s
2022-05-18T05:17:55.8928751Z 
2022-05-18T05:17:55.8928846Z OK
2022-05-18T05:17:55.8928965Z 
2022-05-18T05:17:55.8929102Z Generating XML reports...
2022-05-18T05:17:55.8986311Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051752.xml
2022-05-18T05:17:57.3194860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:57.3210380Z 
2022-05-18T05:17:57.3210733Z Running tests...
2022-05-18T05:17:57.3211503Z ----------------------------------------------------------------------
2022-05-18T05:17:58.9647745Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:59.0021934Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68615
2022-05-18T05:17:59.0134136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68616
2022-05-18T05:18:00.2538433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:00.2538992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:00.2539808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:00.2540492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:00.2647405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:00.3550134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:00.5184218Z ok (3.197s)
2022-05-18T05:18:00.5184452Z 
2022-05-18T05:18:00.5184832Z ----------------------------------------------------------------------
2022-05-18T05:18:00.5185175Z Ran 1 test in 3.197s
2022-05-18T05:18:00.5185342Z 
2022-05-18T05:18:00.5185442Z OK
2022-05-18T05:18:00.5185560Z 
2022-05-18T05:18:00.5185697Z Generating XML reports...
2022-05-18T05:18:00.5243362Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051757.xml
2022-05-18T05:18:01.9500234Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:18:01.9515313Z 
2022-05-18T05:18:01.9515617Z Running tests...
2022-05-18T05:18:01.9516050Z ----------------------------------------------------------------------
2022-05-18T05:18:03.6110345Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:18:03.6490564Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68820
2022-05-18T05:18:03.6602488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68821
2022-05-18T05:18:04.8140933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:04.8141505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:04.8142287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:04.8143243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:04.8149909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:04.8150418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:06.4679365Z ok (4.516s)
2022-05-18T05:18:06.4679603Z 
2022-05-18T05:18:06.4680001Z ----------------------------------------------------------------------
2022-05-18T05:18:06.4680341Z Ran 1 test in 4.516s
2022-05-18T05:18:06.4680507Z 
2022-05-18T05:18:06.4680583Z OK
2022-05-18T05:18:06.4684216Z 
2022-05-18T05:18:06.4684485Z Generating XML reports...
2022-05-18T05:18:06.4742849Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051801.xml
2022-05-18T05:18:07.8962198Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:18:07.8976867Z 
2022-05-18T05:18:07.8977009Z Running tests...
2022-05-18T05:18:07.8977599Z ----------------------------------------------------------------------
2022-05-18T05:18:09.5085163Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:18:09.5457852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69027
2022-05-18T05:18:09.5568946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69028
2022-05-18T05:18:10.7419291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:10.7419841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:10.7420656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:10.7421379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:10.7428029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:10.7428522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:12.6647725Z ok (4.767s)
2022-05-18T05:18:12.6647954Z 
2022-05-18T05:18:12.6648354Z ----------------------------------------------------------------------
2022-05-18T05:18:12.6648695Z Ran 1 test in 4.767s
2022-05-18T05:18:12.6648865Z 
2022-05-18T05:18:12.6648945Z OK
2022-05-18T05:18:12.6649088Z 
2022-05-18T05:18:12.6649225Z Generating XML reports...
2022-05-18T05:18:12.6708457Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051807.xml
2022-05-18T05:18:14.1133296Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:18:14.1154798Z 
2022-05-18T05:18:14.1155108Z Running tests...
2022-05-18T05:18:14.1155599Z ----------------------------------------------------------------------
2022-05-18T05:18:15.7923356Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:18:15.8304814Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69148
2022-05-18T05:18:15.8420427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69149
2022-05-18T05:18:17.0552649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:17.0553188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:17.0553991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:17.0554699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:17.0562199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:17.0562719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:17.0649898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp73ex7414
2022-05-18T05:18:17.0653752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpum2nrc_3
2022-05-18T05:18:17.0654306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp73ex7414/_remote_module_non_scriptable.py
2022-05-18T05:18:17.0656544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpum2nrc_3/_remote_module_non_scriptable.py
2022-05-18T05:18:17.2470191Z ok (3.131s)
2022-05-18T05:18:17.2470403Z 
2022-05-18T05:18:17.2470834Z ----------------------------------------------------------------------
2022-05-18T05:18:17.2471188Z Ran 1 test in 3.132s
2022-05-18T05:18:17.2471355Z 
2022-05-18T05:18:17.2471453Z OK
2022-05-18T05:18:17.2471585Z 
2022-05-18T05:18:17.2471722Z Generating XML reports...
2022-05-18T05:18:17.2528055Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051814.xml
2022-05-18T05:18:18.6781773Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:18:18.6796760Z 
2022-05-18T05:18:18.6797191Z Running tests...
2022-05-18T05:18:18.6797693Z ----------------------------------------------------------------------
2022-05-18T05:18:20.3282217Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:18:20.3655743Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69267
2022-05-18T05:18:20.3767184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69268
2022-05-18T05:18:21.5776956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:21.5777535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:21.5778334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:21.5779054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:21.5885714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:21.6791911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:22.8811141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmpcsubr
2022-05-18T05:18:22.8811920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmpcsubr/_remote_module_non_scriptable.py
2022-05-18T05:18:22.9885758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46bw6wd7
2022-05-18T05:18:22.9886645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46bw6wd7/_remote_module_non_scriptable.py
2022-05-18T05:18:23.2840411Z ok (4.604s)
2022-05-18T05:18:23.2840780Z 
2022-05-18T05:18:23.2841213Z ----------------------------------------------------------------------
2022-05-18T05:18:23.2841567Z Ran 1 test in 4.604s
2022-05-18T05:18:23.2841737Z 
2022-05-18T05:18:23.2841831Z OK
2022-05-18T05:18:23.2841978Z 
2022-05-18T05:18:23.2842094Z Generating XML reports...
2022-05-18T05:18:23.2899004Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051818.xml
2022-05-18T05:18:24.7021005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:18:24.7035752Z 
2022-05-18T05:18:24.7036230Z Running tests...
2022-05-18T05:18:24.7037079Z ----------------------------------------------------------------------
2022-05-18T05:18:26.3210302Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:18:26.3580591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69384
2022-05-18T05:18:26.3693929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69385
2022-05-18T05:18:27.5759098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:27.5759652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:27.5760455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:27.5761153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:27.5767798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:27.5768352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:28.8938877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8wp20p1
2022-05-18T05:18:28.8939745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8wp20p1/_remote_module_non_scriptable.py
2022-05-18T05:18:28.9034561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprd7ofcbi
2022-05-18T05:18:28.9037275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprd7ofcbi/_remote_module_non_scriptable.py
2022-05-18T05:18:29.1979958Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:18:29.2015698Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:18:29.4772393Z ok (4.773s)
2022-05-18T05:18:29.4772874Z 
2022-05-18T05:18:29.4773600Z ----------------------------------------------------------------------
2022-05-18T05:18:29.4773975Z Ran 1 test in 4.774s
2022-05-18T05:18:29.4774147Z 
2022-05-18T05:18:29.4774254Z OK
2022-05-18T05:18:29.4774391Z 
2022-05-18T05:18:29.4774549Z Generating XML reports...
2022-05-18T05:18:29.4830677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051824.xml
2022-05-18T05:18:30.8960517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:18:30.8976201Z 
2022-05-18T05:18:30.8976576Z Running tests...
2022-05-18T05:18:30.8976997Z ----------------------------------------------------------------------
2022-05-18T05:18:32.5528746Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:18:32.5906125Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69505
2022-05-18T05:18:32.6019752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69506
2022-05-18T05:18:33.7793884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:33.7794696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:33.7795520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:33.7796240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:33.7802994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:33.7803490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:33.8011012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:18:33.8011725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:18:33.8012479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:18:33.8013188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:18:33.8219140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:18:33.8220117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:18:33.8220821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:18:33.8221521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:18:35.1377900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynf9q1wy
2022-05-18T05:18:35.1379023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynf9q1wy/_remote_module_non_scriptable.py
2022-05-18T05:18:35.1564702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhbcau9c
2022-05-18T05:18:35.1566666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhbcau9c/_remote_module_non_scriptable.py
2022-05-18T05:18:40.5185076Z ok (9.621s)
2022-05-18T05:18:40.5185400Z 
2022-05-18T05:18:40.5185807Z ----------------------------------------------------------------------
2022-05-18T05:18:40.5186159Z Ran 1 test in 9.621s
2022-05-18T05:18:40.5186326Z 
2022-05-18T05:18:40.5186401Z OK
2022-05-18T05:18:40.5186542Z 
2022-05-18T05:18:40.5186675Z Generating XML reports...
2022-05-18T05:18:40.5243108Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051830.xml
2022-05-18T05:18:41.9350648Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:18:41.9364331Z 
2022-05-18T05:18:41.9364715Z Running tests...
2022-05-18T05:18:41.9365242Z ----------------------------------------------------------------------
2022-05-18T05:18:43.5768168Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:18:43.6142620Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69634
2022-05-18T05:18:43.6253293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69635
2022-05-18T05:18:44.8412861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:44.8413418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:44.8414220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:44.8414923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:44.8522661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:44.9424845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:44.9538443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:18:44.9539465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:18:44.9540841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:18:44.9542333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:18:44.9646563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:18:44.9647104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:18:44.9647830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:18:44.9648513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:18:46.3092539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2r3tfc_
2022-05-18T05:18:46.3093701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2r3tfc_/_remote_module_non_scriptable.py
2022-05-18T05:18:46.3120614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvz69ezp
2022-05-18T05:18:46.3123658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvz69ezp/_remote_module_non_scriptable.py
2022-05-18T05:18:51.6418735Z ok (9.705s)
2022-05-18T05:18:51.6418965Z 
2022-05-18T05:18:51.6419368Z ----------------------------------------------------------------------
2022-05-18T05:18:51.6419712Z Ran 1 test in 9.705s
2022-05-18T05:18:51.6419889Z 
2022-05-18T05:18:51.6419983Z OK
2022-05-18T05:18:51.6420124Z 
2022-05-18T05:18:51.6420261Z Generating XML reports...
2022-05-18T05:18:51.6476422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051841.xml
2022-05-18T05:18:52.0496685Z Running distributed tests for the gloo backend with file init_method
2022-05-18T05:18:52.0499332Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:18:52.049592]
2022-05-18T05:18:53.1959434Z <unittest.suite.TestSuite tests=[]>
2022-05-18T05:18:53.2001266Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-05-18T05:18:53.2036948Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2037464Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2038008Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2038410Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2038859Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2039334Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2039805Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2040302Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2040814Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2041366Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2041914Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2042444Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2042979Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2043501Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2043964Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2044456Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2044927Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2045342Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2045771Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2046221Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2046706Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2047168Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2047575Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2047973Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2048386Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2048811Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2049231Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2050235Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2050766Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2051161Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2051554Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2051945Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2052335Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2052726Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2053115Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2053535Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2053950Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2054362Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2054807Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2055258Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2055707Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2056122Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2056643Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2057077Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2057494Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2057911Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2058351Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2058791Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2059189Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2059610Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2060034Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2060444Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2060853Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2061272Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2061685Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2062070Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2062461Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2062860Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2063241Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2063620Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2063996Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2064368Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2064776Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2065184Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2065578Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2065943Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2066328Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2066720Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2067099Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2067496Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2067905Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2068329Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2068713Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2069090Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2069465Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2069902Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2070301Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2070693Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2071063Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2071467Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2071898Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2072321Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2072770Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2073225Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2073738Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2074173Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2074617Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2075056Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2075479Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2075923Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2076376Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2076843Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2077292Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2077748Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2078206Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2078614Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2079007Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2079384Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2079751Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2080097Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2080479Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2080872Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2081248Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2081631Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2082029Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2082425Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2082824Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2083223Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2083612Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2084040Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2084453Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2084868Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2085270Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2085737Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2086176Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2086588Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2087001Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2087387Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2087758Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2088129Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2088517Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2088906Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2089285Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2090375Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2090901Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2091339Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2091843Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2092261Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2092694Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2093131Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2093598Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2094041Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2094450Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2094896Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2095306Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2095682Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2096061Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2096477Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2096892Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2097311Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2097751Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2098166Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2098624Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2099117Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2099688Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2100314Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2100921Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2101516Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2102132Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2102813Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2103431Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2104021Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2104572Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2105069Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2105500Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2105893Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2106294Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2106707Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2107092Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2107514Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2108017Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2108466Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2108937Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2109359Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2109750Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2110146Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2110585Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2111015Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2111417Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2111832Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2112265Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2112679Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2113100Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2113507Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2113915Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2114321Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2114721Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2115145Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2115595Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2116023Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2116405Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2116808Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2117214Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2117632Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2118016Z test_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2118364Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2118734Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2119112Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2119475Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2119898Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2120293Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2120679Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2121029Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2121391Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2121771Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2122152Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2122542Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2122913Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2123243Z test_isend (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2123621Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2124022Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2124424Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2124881Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2125331Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2125813Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2126220Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2126655Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2127085Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2127497Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2127913Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2128329Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2128741Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2129134Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2130187Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2130628Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2131081Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2131575Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2132049Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2132499Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2132945Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2133402Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2133838Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2134244Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2134674Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2135115Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2135564Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2136028Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2136543Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2137005Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2137385Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2137878Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2138300Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2138674Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2139058Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2139445Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2139832Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2140185Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2140548Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2140924Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2141286Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2141657Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2142028Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2142399Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2142784Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2143150Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2143597Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2143957Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2144331Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2144716Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2145092Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2145474Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2145857Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2146211Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2146586Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2147011Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2147459Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2147875Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2148276Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2148678Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2149090Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2149508Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2149901Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2150298Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2150737Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2151154Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2151559Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2151952Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2152354Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2152738Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2153139Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2153592Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:53.2154035Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T05:18:54.3366593Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:18:54.3381608Z 
2022-05-18T05:18:54.3381875Z Running tests...
2022-05-18T05:18:54.3382339Z ----------------------------------------------------------------------
2022-05-18T05:18:55.9670132Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:18:56.0047579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69798
2022-05-18T05:18:56.0158036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69799
2022-05-18T05:18:57.2223472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:57.2224063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:57.2224849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:57.2225561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:57.2232192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:57.2233175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:58.5487742Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T05:18:58.5489094Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T05:18:58.5608465Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T05:18:58.5609377Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T05:18:59.5629690Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T05:18:59.5630788Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T05:18:59.5631481Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T05:18:59.5632315Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T05:18:59.5785606Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T05:18:59.5787034Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T05:18:59.5787727Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T05:18:59.5788579Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T05:18:59.5942208Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T05:18:59.5943085Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T05:18:59.5943764Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T05:18:59.5944591Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T05:18:59.9253376Z ok (5.587s)
2022-05-18T05:18:59.9253570Z 
2022-05-18T05:18:59.9254222Z ----------------------------------------------------------------------
2022-05-18T05:18:59.9254593Z Ran 1 test in 5.587s
2022-05-18T05:18:59.9254767Z 
2022-05-18T05:18:59.9254864Z OK
2022-05-18T05:18:59.9255000Z 
2022-05-18T05:18:59.9255115Z Generating XML reports...
2022-05-18T05:18:59.9311664Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051854.xml
2022-05-18T05:19:01.3545885Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:19:01.3560426Z 
2022-05-18T05:19:01.3560959Z Running tests...
2022-05-18T05:19:01.3561587Z ----------------------------------------------------------------------
2022-05-18T05:19:01.3607273Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.005s)
2022-05-18T05:19:01.3607712Z 
2022-05-18T05:19:01.3608157Z ----------------------------------------------------------------------
2022-05-18T05:19:01.3608498Z Ran 1 test in 0.005s
2022-05-18T05:19:01.3608662Z 
2022-05-18T05:19:01.3608769Z OK (skipped=1)
2022-05-18T05:19:01.3608968Z 
2022-05-18T05:19:01.3609202Z Generating XML reports...
2022-05-18T05:19:01.3650856Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051901.xml
2022-05-18T05:19:02.6429584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:19:02.6445332Z 
2022-05-18T05:19:02.6445723Z Running tests...
2022-05-18T05:19:02.6446253Z ----------------------------------------------------------------------
2022-05-18T05:19:04.2935768Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:19:04.3304673Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69951
2022-05-18T05:19:04.3413557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69952
2022-05-18T05:19:05.5424798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:05.5425663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:05.5426536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:05.5427449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:05.5534510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:05.6436129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:05.8462997Z ok (3.201s)
2022-05-18T05:19:05.8463251Z 
2022-05-18T05:19:05.8463632Z ----------------------------------------------------------------------
2022-05-18T05:19:05.8464059Z Ran 1 test in 3.202s
2022-05-18T05:19:05.8464370Z 
2022-05-18T05:19:05.8464490Z OK
2022-05-18T05:19:05.8464629Z 
2022-05-18T05:19:05.8464787Z Generating XML reports...
2022-05-18T05:19:05.8521063Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051902.xml
2022-05-18T05:19:07.2500797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:19:07.2515582Z 
2022-05-18T05:19:07.2515869Z Running tests...
2022-05-18T05:19:07.2516314Z ----------------------------------------------------------------------
2022-05-18T05:19:08.8684265Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:19:08.8801577Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.628s)
2022-05-18T05:19:08.8802253Z 
2022-05-18T05:19:08.8802806Z ----------------------------------------------------------------------
2022-05-18T05:19:08.8803138Z Ran 1 test in 1.629s
2022-05-18T05:19:08.8803304Z 
2022-05-18T05:19:08.8803412Z OK (skipped=1)
2022-05-18T05:19:08.8803579Z 
2022-05-18T05:19:08.8803705Z Generating XML reports...
2022-05-18T05:19:08.8840654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051907.xml
2022-05-18T05:19:10.2545550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:19:10.2559960Z 
2022-05-18T05:19:10.2560451Z Running tests...
2022-05-18T05:19:10.2560968Z ----------------------------------------------------------------------
2022-05-18T05:19:11.8744463Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:19:11.9113240Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70102
2022-05-18T05:19:11.9225346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70103
2022-05-18T05:19:13.1268006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:13.1268851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:13.1269700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:13.1270407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:13.1379128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:13.1472262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp688d7al6
2022-05-18T05:19:13.1474458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp688d7al6/_remote_module_non_scriptable.py
2022-05-18T05:19:13.2280481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:13.2377928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfs0xkpq
2022-05-18T05:19:13.2380534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfs0xkpq/_remote_module_non_scriptable.py
2022-05-18T05:19:13.2584001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:13.2584523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:13.4273772Z ok (3.171s)
2022-05-18T05:19:13.4273969Z 
2022-05-18T05:19:13.4274642Z ----------------------------------------------------------------------
2022-05-18T05:19:13.4275130Z Ran 1 test in 3.171s
2022-05-18T05:19:13.4275299Z 
2022-05-18T05:19:13.4275393Z OK
2022-05-18T05:19:13.4275528Z 
2022-05-18T05:19:13.4275644Z Generating XML reports...
2022-05-18T05:19:13.4331794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051910.xml
2022-05-18T05:19:14.8307948Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:19:14.8322096Z 
2022-05-18T05:19:14.8322454Z Running tests...
2022-05-18T05:19:14.8322900Z ----------------------------------------------------------------------
2022-05-18T05:19:16.5011353Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:19:16.5382937Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70221
2022-05-18T05:19:16.5494245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70222
2022-05-18T05:19:17.7347440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:17.7348000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:17.7349025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:17.7349742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:17.7458361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:17.7551355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsq2ruoil
2022-05-18T05:19:17.7553983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsq2ruoil/_remote_module_non_scriptable.py
2022-05-18T05:19:17.8358906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:17.8456572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzqwakmth
2022-05-18T05:19:17.8459295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzqwakmth/_remote_module_non_scriptable.py
2022-05-18T05:19:17.8661970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:17.8662481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:18.0544704Z ok (3.222s)
2022-05-18T05:19:18.0544896Z 
2022-05-18T05:19:18.0545259Z ----------------------------------------------------------------------
2022-05-18T05:19:18.0545596Z Ran 1 test in 3.222s
2022-05-18T05:19:18.0545770Z 
2022-05-18T05:19:18.0545864Z OK
2022-05-18T05:19:18.0546001Z 
2022-05-18T05:19:18.0546130Z Generating XML reports...
2022-05-18T05:19:18.0603225Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051914.xml
2022-05-18T05:19:19.4803125Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:19:19.4818593Z 
2022-05-18T05:19:19.4818787Z Running tests...
2022-05-18T05:19:19.4819235Z ----------------------------------------------------------------------
2022-05-18T05:19:21.1404723Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:19:21.1781072Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70340
2022-05-18T05:19:21.1892529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70341
2022-05-18T05:19:22.3871664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:22.3872239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:22.3873043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:22.3873734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:22.3980127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:22.4887380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:23.6952853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6pvybfkn
2022-05-18T05:19:23.6953908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6pvybfkn/_remote_module_non_scriptable.py
2022-05-18T05:19:23.7837782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfs691svx
2022-05-18T05:19:23.7839004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfs691svx/_remote_module_non_scriptable.py
2022-05-18T05:19:24.3465153Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:24.3465694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:24.3722456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:24.3722991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:24.4052965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:24.4053478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:24.4307379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:24.4307882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:24.5622244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:24.5622837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:24.5874182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:24.5874826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:25.0985704Z ok (5.616s)
2022-05-18T05:19:25.0985892Z 
2022-05-18T05:19:25.0986491Z ----------------------------------------------------------------------
2022-05-18T05:19:25.0987086Z Ran 1 test in 5.617s
2022-05-18T05:19:25.0987266Z 
2022-05-18T05:19:25.0987358Z OK
2022-05-18T05:19:25.0987476Z 
2022-05-18T05:19:25.0987608Z Generating XML reports...
2022-05-18T05:19:25.1044811Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051919.xml
2022-05-18T05:19:26.5576970Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:19:26.5592111Z 
2022-05-18T05:19:26.5592329Z Running tests...
2022-05-18T05:19:28.2209228Z ----------------------------------------------------------------------
2022-05-18T05:19:28.2210173Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:19:28.2589438Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70461
2022-05-18T05:19:28.2701900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70462
2022-05-18T05:19:29.4541448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:29.4541991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:29.4542795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:29.4543500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:29.4550522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:29.4551546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:30.7834611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhf3d5h5
2022-05-18T05:19:30.7835206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhf3d5h5/_remote_module_non_scriptable.py
2022-05-18T05:19:30.8193419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7gtuq4yi
2022-05-18T05:19:30.8195662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7gtuq4yi/_remote_module_non_scriptable.py
2022-05-18T05:19:30.8445411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:30.8445934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:30.8617189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:30.8617731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:31.1777589Z ok (4.618s)
2022-05-18T05:19:31.1777835Z 
2022-05-18T05:19:31.1778242Z ----------------------------------------------------------------------
2022-05-18T05:19:31.1778577Z Ran 1 test in 4.619s
2022-05-18T05:19:31.1778753Z 
2022-05-18T05:19:31.1778845Z OK
2022-05-18T05:19:31.1778979Z 
2022-05-18T05:19:31.1779093Z Generating XML reports...
2022-05-18T05:19:31.1835946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051926.xml
2022-05-18T05:19:32.6072202Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:19:32.6086372Z 
2022-05-18T05:19:32.6086610Z Running tests...
2022-05-18T05:19:32.6087032Z ----------------------------------------------------------------------
2022-05-18T05:19:34.2209092Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:19:34.2578780Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70582
2022-05-18T05:19:34.2689138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70583
2022-05-18T05:19:35.4296116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:35.4296939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:35.4297746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:35.4298451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:35.4304586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:35.4305245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:36.7424989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwx6jb02
2022-05-18T05:19:36.7425668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwx6jb02/_remote_module_non_scriptable.py
2022-05-18T05:19:36.7688143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2f090r3
2022-05-18T05:19:36.7691052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2f090r3/_remote_module_non_scriptable.py
2022-05-18T05:19:36.7961115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:36.7961621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:36.8159408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:36.8159918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:37.0765011Z ok (4.468s)
2022-05-18T05:19:37.0765352Z 
2022-05-18T05:19:37.0765771Z ----------------------------------------------------------------------
2022-05-18T05:19:37.0766113Z Ran 1 test in 4.468s
2022-05-18T05:19:37.0766281Z 
2022-05-18T05:19:37.0766382Z OK
2022-05-18T05:19:37.0766521Z 
2022-05-18T05:19:37.0766662Z Generating XML reports...
2022-05-18T05:19:37.0824484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051932.xml
2022-05-18T05:19:38.5269041Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:19:38.5284834Z 
2022-05-18T05:19:38.5285338Z Running tests...
2022-05-18T05:19:38.5285846Z ----------------------------------------------------------------------
2022-05-18T05:19:40.2028024Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:19:40.2409157Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70703
2022-05-18T05:19:40.2521191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70704
2022-05-18T05:19:41.4709690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:41.4710294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:41.4711079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:41.4711783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:41.4718453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:41.4719425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:42.7738268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3bhmit5
2022-05-18T05:19:42.7738908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3bhmit5/_remote_module_non_scriptable.py
2022-05-18T05:19:42.7934450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8uwfst7b
2022-05-18T05:19:42.7937073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8uwfst7b/_remote_module_non_scriptable.py
2022-05-18T05:19:42.8158698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:42.8159239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:43.3600358Z ok (4.831s)
2022-05-18T05:19:43.3600739Z 
2022-05-18T05:19:43.3601529Z ----------------------------------------------------------------------
2022-05-18T05:19:43.3602002Z Ran 1 test in 4.832s
2022-05-18T05:19:43.3602152Z 
2022-05-18T05:19:43.3602246Z OK
2022-05-18T05:19:43.3602380Z 
2022-05-18T05:19:43.3602516Z Generating XML reports...
2022-05-18T05:19:43.3659431Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051938.xml
2022-05-18T05:19:44.7912673Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:19:44.7927717Z 
2022-05-18T05:19:44.7928245Z Running tests...
2022-05-18T05:19:44.7928749Z ----------------------------------------------------------------------
2022-05-18T05:19:46.4468037Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:19:46.4845746Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70824
2022-05-18T05:19:46.4957885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70825
2022-05-18T05:19:47.7048295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:47.7049072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:47.7050121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:47.7050830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:47.7157767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:47.8063627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:49.0076211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpniysx0s7
2022-05-18T05:19:49.0077075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpniysx0s7/_remote_module_non_scriptable.py
2022-05-18T05:19:49.1120472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw32rhx56
2022-05-18T05:19:49.1121746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw32rhx56/_remote_module_non_scriptable.py
2022-05-18T05:19:49.6817777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:49.6818338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:49.7072185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:49.7072688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:50.1045844Z ok (5.311s)
2022-05-18T05:19:50.1046031Z 
2022-05-18T05:19:50.1046675Z ----------------------------------------------------------------------
2022-05-18T05:19:50.1047386Z Ran 1 test in 5.312s
2022-05-18T05:19:50.1047692Z 
2022-05-18T05:19:50.1047805Z OK
2022-05-18T05:19:50.1047944Z 
2022-05-18T05:19:50.1048074Z Generating XML reports...
2022-05-18T05:19:50.1106449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051944.xml
2022-05-18T05:19:51.5588787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:19:51.5603381Z 
2022-05-18T05:19:51.5603852Z Running tests...
2022-05-18T05:19:51.5604981Z ----------------------------------------------------------------------
2022-05-18T05:19:53.1929319Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:19:53.2309612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70945
2022-05-18T05:19:53.2421642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70946
2022-05-18T05:19:54.4555000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:54.4555561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:54.4556373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:54.4557066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:54.4564206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:54.4564702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:55.7916642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphffe32x6
2022-05-18T05:19:55.7917257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphffe32x6/_remote_module_non_scriptable.py
2022-05-18T05:19:55.8320266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpemxnsvq3
2022-05-18T05:19:55.8322623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpemxnsvq3/_remote_module_non_scriptable.py
2022-05-18T05:19:56.1459719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:56.1460286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:56.1672086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:56.1672602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:56.4503936Z ok (4.890s)
2022-05-18T05:19:56.4504166Z 
2022-05-18T05:19:56.4504560Z ----------------------------------------------------------------------
2022-05-18T05:19:56.4504903Z Ran 1 test in 4.890s
2022-05-18T05:19:56.4505078Z 
2022-05-18T05:19:56.4505172Z OK
2022-05-18T05:19:56.4505289Z 
2022-05-18T05:19:56.4505436Z Generating XML reports...
2022-05-18T05:19:56.4563822Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051951.xml
2022-05-18T05:19:57.8989568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:19:57.9003991Z 
2022-05-18T05:19:57.9004118Z Running tests...
2022-05-18T05:19:57.9004827Z ----------------------------------------------------------------------
2022-05-18T05:19:59.5566208Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:19:59.5938132Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71066
2022-05-18T05:19:59.6050315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71067
2022-05-18T05:20:00.7515501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:00.7516062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:00.7516868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:00.7517586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:00.7624058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:00.8529231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:02.0877787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznvaof3l
2022-05-18T05:20:02.0878411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznvaof3l/_remote_module_non_scriptable.py
2022-05-18T05:20:02.1500324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqn5n8nfp
2022-05-18T05:20:02.1501745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqn5n8nfp/_remote_module_non_scriptable.py
2022-05-18T05:20:02.1746525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:20:02.1747065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:20:02.1913652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:20:02.1914172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:20:02.5128545Z ok (4.612s)
2022-05-18T05:20:02.5128773Z 
2022-05-18T05:20:02.5129176Z ----------------------------------------------------------------------
2022-05-18T05:20:02.5129711Z Ran 1 test in 4.612s
2022-05-18T05:20:02.5130049Z 
2022-05-18T05:20:02.5130214Z OK
2022-05-18T05:20:02.5130455Z 
2022-05-18T05:20:02.5130685Z Generating XML reports...
2022-05-18T05:20:02.5188461Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051957.xml
2022-05-18T05:20:03.9408195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:03.9423264Z 
2022-05-18T05:20:03.9423748Z Running tests...
2022-05-18T05:20:03.9424271Z ----------------------------------------------------------------------
2022-05-18T05:20:05.5838606Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:20:05.5954950Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.653s)
2022-05-18T05:20:05.5955611Z 
2022-05-18T05:20:05.5955889Z ----------------------------------------------------------------------
2022-05-18T05:20:05.5956221Z Ran 1 test in 1.653s
2022-05-18T05:20:05.5956385Z 
2022-05-18T05:20:05.5956495Z OK (skipped=1)
2022-05-18T05:20:05.5956650Z 
2022-05-18T05:20:05.5956756Z Generating XML reports...
2022-05-18T05:20:05.5994251Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052003.xml
2022-05-18T05:20:06.9794407Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:06.9810340Z 
2022-05-18T05:20:06.9810811Z Running tests...
2022-05-18T05:20:06.9811420Z ----------------------------------------------------------------------
2022-05-18T05:20:08.6359147Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:20:08.6745517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71223
2022-05-18T05:20:08.6857771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71224
2022-05-18T05:20:09.8622569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:09.8623143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:09.8623966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:09.8624657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:09.8733065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:09.9633583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:10.2909355Z ok (3.310s)
2022-05-18T05:20:10.2909598Z 
2022-05-18T05:20:10.2910015Z ----------------------------------------------------------------------
2022-05-18T05:20:10.2910362Z Ran 1 test in 3.310s
2022-05-18T05:20:10.2910530Z 
2022-05-18T05:20:10.2910625Z OK
2022-05-18T05:20:10.2910742Z 
2022-05-18T05:20:10.2910878Z Generating XML reports...
2022-05-18T05:20:10.2967558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052006.xml
2022-05-18T05:20:11.7093301Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:11.7107520Z 
2022-05-18T05:20:11.7107832Z Running tests...
2022-05-18T05:20:11.7108272Z ----------------------------------------------------------------------
2022-05-18T05:20:13.3377234Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:20:13.3492783Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.638s)
2022-05-18T05:20:13.3493448Z 
2022-05-18T05:20:13.3493725Z ----------------------------------------------------------------------
2022-05-18T05:20:13.3494043Z Ran 1 test in 1.639s
2022-05-18T05:20:13.3494222Z 
2022-05-18T05:20:13.3494331Z OK (skipped=1)
2022-05-18T05:20:13.3494485Z 
2022-05-18T05:20:13.3494611Z Generating XML reports...
2022-05-18T05:20:13.3531794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052011.xml
2022-05-18T05:20:14.7392381Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:14.7409117Z 
2022-05-18T05:20:14.7410118Z Running tests...
2022-05-18T05:20:14.7410654Z ----------------------------------------------------------------------
2022-05-18T05:20:16.3800710Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:20:16.4179663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71374
2022-05-18T05:20:16.4290878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71375
2022-05-18T05:20:17.6053570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:17.6054146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:17.6054952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:17.6055655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:17.6162625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:17.7070126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:19.3365625Z ok (4.596s)
2022-05-18T05:20:19.3365853Z 
2022-05-18T05:20:19.3366222Z ----------------------------------------------------------------------
2022-05-18T05:20:19.3366571Z Ran 1 test in 4.596s
2022-05-18T05:20:19.3366741Z 
2022-05-18T05:20:19.3366836Z OK
2022-05-18T05:20:19.3366994Z 
2022-05-18T05:20:19.3367170Z Generating XML reports...
2022-05-18T05:20:19.3425405Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052014.xml
2022-05-18T05:20:20.7836001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:20.7850284Z 
2022-05-18T05:20:20.7850578Z Running tests...
2022-05-18T05:20:20.7850998Z ----------------------------------------------------------------------
2022-05-18T05:20:20.7873814Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-05-18T05:20:20.7874122Z 
2022-05-18T05:20:20.7874495Z ----------------------------------------------------------------------
2022-05-18T05:20:20.7874997Z Ran 1 test in 0.002s
2022-05-18T05:20:20.7875167Z 
2022-05-18T05:20:20.7875260Z OK (skipped=1)
2022-05-18T05:20:20.7875417Z 
2022-05-18T05:20:20.7875542Z Generating XML reports...
2022-05-18T05:20:20.7917559Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052020.xml
2022-05-18T05:20:22.0221165Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:22.0237062Z 
2022-05-18T05:20:22.0237349Z Running tests...
2022-05-18T05:20:22.0237785Z ----------------------------------------------------------------------
2022-05-18T05:20:22.0258118Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T05:20:23.6895062Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:20:23.7266766Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71526
2022-05-18T05:20:23.7378907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71527
2022-05-18T05:20:24.9122014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:24.9122995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:24.9124466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:24.9125874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:24.9131887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:24.9132592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:24.9231838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0omc6kh7
2022-05-18T05:20:24.9234212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0omc6kh7/_remote_module_non_scriptable.py
2022-05-18T05:20:24.9235554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4si1vp_p
2022-05-18T05:20:24.9237136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4si1vp_p/_remote_module_non_scriptable.py
2022-05-18T05:20:25.1427591Z ok (3.119s)
2022-05-18T05:20:25.1429545Z 
2022-05-18T05:20:25.1430021Z ----------------------------------------------------------------------
2022-05-18T05:20:25.1430351Z Ran 1 test in 3.119s
2022-05-18T05:20:25.1430526Z 
2022-05-18T05:20:25.1430620Z OK
2022-05-18T05:20:25.1430762Z 
2022-05-18T05:20:25.1430892Z Generating XML reports...
2022-05-18T05:20:25.1486269Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052022.xml
2022-05-18T05:20:26.5684704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:26.5700979Z 
2022-05-18T05:20:26.5701279Z Running tests...
2022-05-18T05:20:26.5701714Z ----------------------------------------------------------------------
2022-05-18T05:20:26.5726177Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T05:20:28.2102638Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:20:28.2477708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71645
2022-05-18T05:20:28.2586976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71646
2022-05-18T05:20:29.4235005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:29.4235572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:29.4236600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:29.4237313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:29.4244172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:29.4245265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:29.4344474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavz5acc8
2022-05-18T05:20:29.4346632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavz5acc8/_remote_module_non_scriptable.py
2022-05-18T05:20:29.4348621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1dq2rx1
2022-05-18T05:20:29.4352538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1dq2rx1/_remote_module_non_scriptable.py
2022-05-18T05:20:29.4607052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:20:29.4608077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:20:29.6634483Z ok (3.093s)
2022-05-18T05:20:29.6634708Z 
2022-05-18T05:20:29.6635111Z ----------------------------------------------------------------------
2022-05-18T05:20:29.6635458Z Ran 1 test in 3.093s
2022-05-18T05:20:29.6635605Z 
2022-05-18T05:20:29.6635710Z OK
2022-05-18T05:20:29.6635858Z 
2022-05-18T05:20:29.6635994Z Generating XML reports...
2022-05-18T05:20:29.6694124Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052026.xml
2022-05-18T05:20:31.0930733Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:31.0946609Z 
2022-05-18T05:20:31.0947128Z Running tests...
2022-05-18T05:20:31.0947633Z ----------------------------------------------------------------------
2022-05-18T05:20:31.0974448Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T05:20:32.7545780Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:20:32.7919531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71764
2022-05-18T05:20:32.8030049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71765
2022-05-18T05:20:33.9704403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:33.9704967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:33.9705742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:33.9706450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:33.9813614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:33.9913193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu6qv8bb9
2022-05-18T05:20:33.9915323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu6qv8bb9/_remote_module_non_scriptable.py
2022-05-18T05:20:34.0716411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:34.0820842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpee47os85
2022-05-18T05:20:34.0823630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpee47os85/_remote_module_non_scriptable.py
2022-05-18T05:20:34.1078212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:20:34.1079250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:20:34.3078880Z ok (3.213s)
2022-05-18T05:20:34.3079107Z 
2022-05-18T05:20:34.3079494Z ----------------------------------------------------------------------
2022-05-18T05:20:34.3079817Z Ran 1 test in 3.213s
2022-05-18T05:20:34.3079985Z 
2022-05-18T05:20:34.3080087Z OK
2022-05-18T05:20:34.3080230Z 
2022-05-18T05:20:34.3080385Z Generating XML reports...
2022-05-18T05:20:34.3137148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052031.xml
2022-05-18T05:20:35.7298660Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:35.7313365Z 
2022-05-18T05:20:35.7313821Z Running tests...
2022-05-18T05:20:35.7314305Z ----------------------------------------------------------------------
2022-05-18T05:20:35.7332783Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T05:20:37.3382246Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:20:37.3753772Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71883
2022-05-18T05:20:37.3864552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71884
2022-05-18T05:20:38.5511055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:38.5511600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:38.5512404Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:38.5513100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:38.5620568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:38.5720706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnu0opg3a
2022-05-18T05:20:38.5723321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnu0opg3a/_remote_module_non_scriptable.py
2022-05-18T05:20:38.6525358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:38.6629497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8zu5kj3
2022-05-18T05:20:38.6632241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8zu5kj3/_remote_module_non_scriptable.py
2022-05-18T05:20:38.8916806Z ok (3.160s)
2022-05-18T05:20:38.8917027Z 
2022-05-18T05:20:38.8917424Z ----------------------------------------------------------------------
2022-05-18T05:20:38.8917765Z Ran 1 test in 3.160s
2022-05-18T05:20:38.8917934Z 
2022-05-18T05:20:38.8918028Z OK
2022-05-18T05:20:38.8918169Z 
2022-05-18T05:20:38.8918307Z Generating XML reports...
2022-05-18T05:20:38.8976987Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052035.xml
2022-05-18T05:20:40.3444469Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:40.3459709Z 
2022-05-18T05:20:40.3459990Z Running tests...
2022-05-18T05:20:40.3460450Z ----------------------------------------------------------------------
2022-05-18T05:20:41.9877023Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:20:42.0257358Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72002
2022-05-18T05:20:42.0369148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72003
2022-05-18T05:20:43.2239009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:43.2239992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:43.2241455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:43.2242870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:43.2347487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:43.3252647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:43.5420236Z ok (3.196s)
2022-05-18T05:20:43.5420467Z 
2022-05-18T05:20:43.5420857Z ----------------------------------------------------------------------
2022-05-18T05:20:43.5421216Z Ran 1 test in 3.196s
2022-05-18T05:20:43.5421363Z 
2022-05-18T05:20:43.5421456Z OK
2022-05-18T05:20:43.5421597Z 
2022-05-18T05:20:43.5421737Z Generating XML reports...
2022-05-18T05:20:43.5479545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052040.xml
2022-05-18T05:20:44.9830811Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:44.9847206Z 
2022-05-18T05:20:44.9847632Z Running tests...
2022-05-18T05:20:44.9848119Z ----------------------------------------------------------------------
2022-05-18T05:20:46.6395402Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:20:46.6780519Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72121
2022-05-18T05:20:46.6893467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72122
2022-05-18T05:20:47.8595571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:47.8596130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:47.8596909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:47.8597629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:47.8605355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:47.8606078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:48.0941910Z ok (3.109s)
2022-05-18T05:20:48.0942099Z 
2022-05-18T05:20:48.0942470Z ----------------------------------------------------------------------
2022-05-18T05:20:48.0942817Z Ran 1 test in 3.109s
2022-05-18T05:20:48.0942988Z 
2022-05-18T05:20:48.0943062Z OK
2022-05-18T05:20:48.0943201Z 
2022-05-18T05:20:48.0943329Z Generating XML reports...
2022-05-18T05:20:48.1016459Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052044.xml
2022-05-18T05:20:49.5197195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:49.5213236Z 
2022-05-18T05:20:49.5213865Z Running tests...
2022-05-18T05:20:49.5214394Z ----------------------------------------------------------------------
2022-05-18T05:20:51.1557986Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:20:51.1932844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72240
2022-05-18T05:20:51.2042677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72241
2022-05-18T05:20:52.3954990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:52.3956054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:52.3957465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:52.3958860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:52.3965287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:52.3966295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:52.4071896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:20:52.4072927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:20:52.4074314Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:20:52.4075657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:20:52.6088984Z ok (3.087s)
2022-05-18T05:20:52.6089178Z 
2022-05-18T05:20:52.6089934Z ----------------------------------------------------------------------
2022-05-18T05:20:52.6090272Z Ran 1 test in 3.088s
2022-05-18T05:20:52.6090450Z 
2022-05-18T05:20:52.6090545Z OK
2022-05-18T05:20:52.6090683Z 
2022-05-18T05:20:52.6090818Z Generating XML reports...
2022-05-18T05:20:52.6147558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052049.xml
2022-05-18T05:20:54.0124782Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:54.0139850Z 
2022-05-18T05:20:54.0140172Z Running tests...
2022-05-18T05:20:54.0140612Z ----------------------------------------------------------------------
2022-05-18T05:20:55.6472091Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:20:55.6849963Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72365
2022-05-18T05:20:55.6961055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72366
2022-05-18T05:20:56.9087058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:56.9087618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:56.9088657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:56.9089390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:56.9197239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:57.0102264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:57.2010273Z skip: Skipped due to small world size. (3.187s)
2022-05-18T05:20:57.2010825Z 
2022-05-18T05:20:57.2011215Z ----------------------------------------------------------------------
2022-05-18T05:20:57.2011540Z Ran 1 test in 3.187s
2022-05-18T05:20:57.2011706Z 
2022-05-18T05:20:57.2011816Z OK (skipped=1)
2022-05-18T05:20:57.2011974Z 
2022-05-18T05:20:57.2013257Z Generating XML reports...
2022-05-18T05:20:57.2070383Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052054.xml
2022-05-18T05:20:58.6287948Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:20:58.6303349Z 
2022-05-18T05:20:58.6303796Z Running tests...
2022-05-18T05:20:58.6304236Z ----------------------------------------------------------------------
2022-05-18T05:21:00.3019775Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:00.3400009Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72480
2022-05-18T05:21:00.3511948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72481
2022-05-18T05:21:01.5433678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:21:01.5434251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:21:01.5435077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:01.5435766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:01.5542324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:01.6445352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:01.8562012Z ok (3.226s)
2022-05-18T05:21:01.8562225Z 
2022-05-18T05:21:01.8562619Z ----------------------------------------------------------------------
2022-05-18T05:21:01.8562960Z Ran 1 test in 3.226s
2022-05-18T05:21:01.8563127Z 
2022-05-18T05:21:01.8563223Z OK
2022-05-18T05:21:01.8563369Z 
2022-05-18T05:21:01.8563489Z Generating XML reports...
2022-05-18T05:21:01.8620417Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052058.xml
2022-05-18T05:21:03.2964974Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:03.2980850Z 
2022-05-18T05:21:03.2981346Z Running tests...
2022-05-18T05:21:03.2981849Z ----------------------------------------------------------------------
2022-05-18T05:21:04.9465605Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:04.9847401Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72599
2022-05-18T05:21:04.9960700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72600
2022-05-18T05:21:06.1964880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:21:06.1965461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:21:06.1966532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:06.1967272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:06.2075580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:06.2977035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:06.5009167Z ok (3.203s)
2022-05-18T05:21:06.5009385Z 
2022-05-18T05:21:06.5010108Z ----------------------------------------------------------------------
2022-05-18T05:21:06.5010607Z Ran 1 test in 3.203s
2022-05-18T05:21:06.5010779Z 
2022-05-18T05:21:06.5010877Z OK
2022-05-18T05:21:06.5010995Z 
2022-05-18T05:21:06.5011131Z Generating XML reports...
2022-05-18T05:21:06.5068794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052103.xml
2022-05-18T05:21:07.9057730Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:07.9072651Z 
2022-05-18T05:21:07.9072893Z Running tests...
2022-05-18T05:21:07.9073341Z ----------------------------------------------------------------------
2022-05-18T05:21:09.5121013Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:09.5493631Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72718
2022-05-18T05:21:09.5605940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72719
2022-05-18T05:21:10.7300292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:21:10.7300871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:21:10.7301678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:10.7302378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:10.7410766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:10.8311658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:11.0656917Z ok (3.158s)
2022-05-18T05:21:11.0657118Z 
2022-05-18T05:21:11.0657490Z ----------------------------------------------------------------------
2022-05-18T05:21:11.0657829Z Ran 1 test in 3.158s
2022-05-18T05:21:11.0658001Z 
2022-05-18T05:21:11.0658097Z OK
2022-05-18T05:21:11.0658233Z 
2022-05-18T05:21:11.0658352Z Generating XML reports...
2022-05-18T05:21:11.0714611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052107.xml
2022-05-18T05:21:12.4815129Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:12.4829845Z 
2022-05-18T05:21:12.4830069Z Running tests...
2022-05-18T05:21:12.4830507Z ----------------------------------------------------------------------
2022-05-18T05:21:12.4850959Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-05-18T05:21:12.4851284Z 
2022-05-18T05:21:12.4851543Z ----------------------------------------------------------------------
2022-05-18T05:21:12.4851876Z Ran 1 test in 0.002s
2022-05-18T05:21:12.4852045Z 
2022-05-18T05:21:12.4852157Z OK (skipped=1)
2022-05-18T05:21:12.4852318Z 
2022-05-18T05:21:12.4852444Z Generating XML reports...
2022-05-18T05:21:12.4894895Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052112.xml
2022-05-18T05:21:13.7421417Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:13.7435565Z 
2022-05-18T05:21:13.7436020Z Running tests...
2022-05-18T05:21:13.7436752Z ----------------------------------------------------------------------
2022-05-18T05:21:13.7457052Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-05-18T05:21:13.7457401Z 
2022-05-18T05:21:13.7457689Z ----------------------------------------------------------------------
2022-05-18T05:21:13.7458020Z Ran 1 test in 0.002s
2022-05-18T05:21:13.7458185Z 
2022-05-18T05:21:13.7458275Z OK (skipped=1)
2022-05-18T05:21:13.7458432Z 
2022-05-18T05:21:13.7458561Z Generating XML reports...
2022-05-18T05:21:13.7498639Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052113.xml
2022-05-18T05:21:15.0291798Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:15.0306928Z 
2022-05-18T05:21:15.0307311Z Running tests...
2022-05-18T05:21:15.0307820Z ----------------------------------------------------------------------
2022-05-18T05:21:16.6894784Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:16.7272765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72907
2022-05-18T05:21:16.7385105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72908
2022-05-18T05:21:17.9078779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:21:17.9079317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:21:17.9080106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:17.9080809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:17.9187863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:18.0091855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:18.0203566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:21:18.0204100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:21:18.0204798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:21:18.0205479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:21:18.2435257Z ok (3.212s)
2022-05-18T05:21:18.2435507Z 
2022-05-18T05:21:18.2435885Z ----------------------------------------------------------------------
2022-05-18T05:21:18.2436220Z Ran 1 test in 3.213s
2022-05-18T05:21:18.2436384Z 
2022-05-18T05:21:18.2436484Z OK
2022-05-18T05:21:18.2436601Z 
2022-05-18T05:21:18.2436751Z Generating XML reports...
2022-05-18T05:21:18.2493655Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052115.xml
2022-05-18T05:21:19.6744146Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:19.6759305Z 
2022-05-18T05:21:19.6759553Z Running tests...
2022-05-18T05:21:19.6759987Z ----------------------------------------------------------------------
2022-05-18T05:21:21.3440107Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:21.3823638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73032
2022-05-18T05:21:21.3936320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73033
2022-05-18T05:21:22.5564058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:21:22.5564827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:21:22.5565680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:22.5566400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:22.5675069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:22.6579555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:22.7985064Z skip: Skipped due to small world size. (3.122s)
2022-05-18T05:21:22.7985304Z 
2022-05-18T05:21:22.7985697Z ----------------------------------------------------------------------
2022-05-18T05:21:22.7986035Z Ran 1 test in 3.123s
2022-05-18T05:21:22.7986179Z 
2022-05-18T05:21:22.7986291Z OK (skipped=1)
2022-05-18T05:21:22.7986445Z 
2022-05-18T05:21:22.7986574Z Generating XML reports...
2022-05-18T05:21:22.8044073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052119.xml
2022-05-18T05:21:24.2041384Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:24.2058011Z 
2022-05-18T05:21:24.2058324Z Running tests...
2022-05-18T05:21:24.2058773Z ----------------------------------------------------------------------
2022-05-18T05:21:24.2081999Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-05-18T05:21:24.2082337Z 
2022-05-18T05:21:24.2082628Z ----------------------------------------------------------------------
2022-05-18T05:21:24.2082953Z Ran 1 test in 0.002s
2022-05-18T05:21:24.2083119Z 
2022-05-18T05:21:24.2083227Z OK (skipped=1)
2022-05-18T05:21:24.2083424Z 
2022-05-18T05:21:24.2083594Z Generating XML reports...
2022-05-18T05:21:24.2128788Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052124.xml
2022-05-18T05:21:25.4888233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:25.4904362Z 
2022-05-18T05:21:25.4904807Z Running tests...
2022-05-18T05:21:25.4905308Z ----------------------------------------------------------------------
2022-05-18T05:21:25.4928120Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-05-18T05:21:25.4928893Z 
2022-05-18T05:21:25.4929195Z ----------------------------------------------------------------------
2022-05-18T05:21:25.4929832Z Ran 1 test in 0.002s
2022-05-18T05:21:25.4930017Z 
2022-05-18T05:21:25.4930128Z OK (skipped=1)
2022-05-18T05:21:25.4930292Z 
2022-05-18T05:21:25.4930421Z Generating XML reports...
2022-05-18T05:21:25.4973698Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052125.xml
2022-05-18T05:21:26.7693326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:26.7709246Z 
2022-05-18T05:21:26.7709580Z Running tests...
2022-05-18T05:21:26.7710035Z ----------------------------------------------------------------------
2022-05-18T05:21:28.4322039Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:28.4694948Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73217
2022-05-18T05:21:28.4805694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73218
2022-05-18T05:21:29.6651719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:21:29.6652278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:21:29.6653286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:29.6654001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:29.6763202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:29.7664706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:29.9855772Z ok (3.214s)
2022-05-18T05:21:29.9856003Z 
2022-05-18T05:21:29.9856352Z ----------------------------------------------------------------------
2022-05-18T05:21:29.9856695Z Ran 1 test in 3.215s
2022-05-18T05:21:29.9856861Z 
2022-05-18T05:21:29.9856954Z OK
2022-05-18T05:21:29.9858196Z 
2022-05-18T05:21:29.9858788Z Generating XML reports...
2022-05-18T05:21:29.9914347Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052126.xml
2022-05-18T05:21:31.3801542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:31.3817998Z 
2022-05-18T05:21:31.3818479Z Running tests...
2022-05-18T05:21:31.3819005Z ----------------------------------------------------------------------
2022-05-18T05:21:33.0297400Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:33.0682710Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73332
2022-05-18T05:21:33.0793498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73333
2022-05-18T05:21:34.2777084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:21:34.2777637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:21:34.2778439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:34.2779160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:34.2885975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:34.3787735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:34.4100496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:21:34.4101021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:21:34.4101728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:21:34.4102408Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:21:34.4243523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:21:34.4244656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:21:34.4245367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:21:34.4345737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:21:34.4466452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T05:21:34.4466963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T05:21:34.4467647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T05:21:34.4468517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T05:21:34.6844072Z ok (3.302s)
2022-05-18T05:21:34.6844308Z 
2022-05-18T05:21:34.6844722Z ----------------------------------------------------------------------
2022-05-18T05:21:34.6845089Z Ran 1 test in 3.303s
2022-05-18T05:21:34.6845236Z 
2022-05-18T05:21:34.6845331Z OK
2022-05-18T05:21:34.6845467Z 
2022-05-18T05:21:34.6845608Z Generating XML reports...
2022-05-18T05:21:34.6903387Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052131.xml
2022-05-18T05:21:36.1516355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:36.1532838Z 
2022-05-18T05:21:36.1533094Z Running tests...
2022-05-18T05:21:36.1533540Z ----------------------------------------------------------------------
2022-05-18T05:21:37.8213407Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:37.8584056Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73471
2022-05-18T05:21:37.8696917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73472
2022-05-18T05:21:39.0935634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:21:39.0936369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:21:39.0937164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:39.0937864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:39.1046856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:39.1948122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:39.2157819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:21:39.2158570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:21:39.2159299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:21:39.2159976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:21:39.4749035Z ok (3.321s)
2022-05-18T05:21:39.4749352Z 
2022-05-18T05:21:39.4749870Z ----------------------------------------------------------------------
2022-05-18T05:21:39.4750218Z Ran 1 test in 3.322s
2022-05-18T05:21:39.4750385Z 
2022-05-18T05:21:39.4750466Z OK
2022-05-18T05:21:39.4750602Z 
2022-05-18T05:21:39.4750739Z Generating XML reports...
2022-05-18T05:21:39.4807588Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052136.xml
2022-05-18T05:21:40.9011790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:40.9028426Z 
2022-05-18T05:21:40.9028744Z Running tests...
2022-05-18T05:21:40.9029207Z ----------------------------------------------------------------------
2022-05-18T05:21:42.5590032Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:42.5970857Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73596
2022-05-18T05:21:42.6083300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73597
2022-05-18T05:21:43.7697131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:21:43.7697691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:21:43.7698727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:43.7699453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:43.7706489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:43.7707356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:43.7914626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:21:43.7915159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:21:43.7915899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:21:43.7916605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:21:44.0131639Z ok (3.110s)
2022-05-18T05:21:44.0131866Z 
2022-05-18T05:21:44.0132468Z ----------------------------------------------------------------------
2022-05-18T05:21:44.0132816Z Ran 1 test in 3.110s
2022-05-18T05:21:44.0133277Z 
2022-05-18T05:21:44.0133352Z OK
2022-05-18T05:21:44.0133489Z 
2022-05-18T05:21:44.0133621Z Generating XML reports...
2022-05-18T05:21:44.0191189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052140.xml
2022-05-18T05:21:45.4458149Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:45.4474458Z 
2022-05-18T05:21:45.4474981Z Running tests...
2022-05-18T05:21:45.4475502Z ----------------------------------------------------------------------
2022-05-18T05:21:47.1098968Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:47.1481810Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73721
2022-05-18T05:21:47.1594476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73722
2022-05-18T05:21:48.3262912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:21:48.3263499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:21:48.3264291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:48.3265063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:48.3373282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:48.4277367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:48.4487542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:21:48.4488053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:21:48.4488770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:21:48.4489474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:21:48.6646836Z ok (3.217s)
2022-05-18T05:21:48.6647037Z 
2022-05-18T05:21:48.6647661Z ----------------------------------------------------------------------
2022-05-18T05:21:48.6648012Z Ran 1 test in 3.217s
2022-05-18T05:21:48.6648180Z 
2022-05-18T05:21:48.6648277Z OK
2022-05-18T05:21:48.6648421Z 
2022-05-18T05:21:48.6648537Z Generating XML reports...
2022-05-18T05:21:48.6706528Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052145.xml
2022-05-18T05:21:50.0913645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:50.0929638Z 
2022-05-18T05:21:50.0929866Z Running tests...
2022-05-18T05:21:50.0930643Z ----------------------------------------------------------------------
2022-05-18T05:21:51.7634114Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:51.8005358Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73846
2022-05-18T05:21:51.8116256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73847
2022-05-18T05:21:52.9984599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:21:52.9985162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:21:52.9985964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:52.9986659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:53.0095644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:53.0996560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:53.1112589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:21:53.1113114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:21:53.1113822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:21:53.1114505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:21:53.3166105Z ok (3.223s)
2022-05-18T05:21:53.3166551Z 
2022-05-18T05:21:53.3166925Z ----------------------------------------------------------------------
2022-05-18T05:21:53.3167281Z Ran 1 test in 3.224s
2022-05-18T05:21:53.3167455Z 
2022-05-18T05:21:53.3167532Z OK
2022-05-18T05:21:53.3167669Z 
2022-05-18T05:21:53.3167808Z Generating XML reports...
2022-05-18T05:21:53.3225516Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052150.xml
2022-05-18T05:21:54.7451247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:54.7467159Z 
2022-05-18T05:21:54.7467665Z Running tests...
2022-05-18T05:21:54.7468341Z ----------------------------------------------------------------------
2022-05-18T05:21:56.4052890Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:56.4435399Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73971
2022-05-18T05:21:56.4549363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73972
2022-05-18T05:21:57.6684208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:21:57.6684771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:21:57.6685569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:57.6686286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:57.6793027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:57.7698921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:57.9599844Z skip: Skipped due to small world size. (3.213s)
2022-05-18T05:21:57.9600384Z 
2022-05-18T05:21:57.9600792Z ----------------------------------------------------------------------
2022-05-18T05:21:57.9601153Z Ran 1 test in 3.213s
2022-05-18T05:21:57.9601322Z 
2022-05-18T05:21:57.9601433Z OK (skipped=1)
2022-05-18T05:21:57.9601602Z 
2022-05-18T05:21:57.9601733Z Generating XML reports...
2022-05-18T05:21:57.9658218Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052154.xml
2022-05-18T05:21:59.3680405Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:21:59.3697383Z 
2022-05-18T05:21:59.3697722Z Running tests...
2022-05-18T05:21:59.3698421Z ----------------------------------------------------------------------
2022-05-18T05:22:01.0244513Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:01.0622717Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74086
2022-05-18T05:22:01.0738261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74087
2022-05-18T05:22:02.2850710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:02.2851515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:02.2852311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:02.2852995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:02.2860063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:02.2860784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:02.4787887Z skip: Skipped due to small world size. (3.109s)
2022-05-18T05:22:02.4788164Z 
2022-05-18T05:22:02.4788549Z ----------------------------------------------------------------------
2022-05-18T05:22:02.4788889Z Ran 1 test in 3.109s
2022-05-18T05:22:02.4789037Z 
2022-05-18T05:22:02.4789157Z OK (skipped=1)
2022-05-18T05:22:02.4789310Z 
2022-05-18T05:22:02.4789443Z Generating XML reports...
2022-05-18T05:22:02.4845591Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052159.xml
2022-05-18T05:22:03.9293060Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:22:03.9309317Z 
2022-05-18T05:22:03.9309801Z Running tests...
2022-05-18T05:22:03.9310307Z ----------------------------------------------------------------------
2022-05-18T05:22:05.5974575Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:05.6360031Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74201
2022-05-18T05:22:05.6474192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74202
2022-05-18T05:22:06.8681212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:06.8681810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:06.8682613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:06.8683324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:06.8691275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:06.8692160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:07.0522858Z skip: Skipped due to small world size. (3.121s)
2022-05-18T05:22:07.0523568Z 
2022-05-18T05:22:07.0523975Z ----------------------------------------------------------------------
2022-05-18T05:22:07.0524301Z Ran 1 test in 3.121s
2022-05-18T05:22:07.0524467Z 
2022-05-18T05:22:07.0524585Z OK (skipped=1)
2022-05-18T05:22:07.0524744Z 
2022-05-18T05:22:07.0524872Z Generating XML reports...
2022-05-18T05:22:07.0581008Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052203.xml
2022-05-18T05:22:08.4895855Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:22:08.4911684Z 
2022-05-18T05:22:08.4911959Z Running tests...
2022-05-18T05:22:08.4912384Z ----------------------------------------------------------------------
2022-05-18T05:22:10.1642587Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:10.2019071Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74316
2022-05-18T05:22:10.2131787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74317
2022-05-18T05:22:11.3719942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:11.3720795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:11.3721594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:11.3722308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:11.3828103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:11.4736529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:11.6180556Z skip: Skipped due to small world size. (3.127s)
2022-05-18T05:22:11.6180801Z 
2022-05-18T05:22:11.6181377Z ----------------------------------------------------------------------
2022-05-18T05:22:11.6181790Z Ran 1 test in 3.127s
2022-05-18T05:22:11.6181958Z 
2022-05-18T05:22:11.6182074Z OK (skipped=1)
2022-05-18T05:22:11.6182246Z 
2022-05-18T05:22:11.6182371Z Generating XML reports...
2022-05-18T05:22:11.6239532Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052208.xml
2022-05-18T05:22:13.0598432Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:22:13.0614312Z 
2022-05-18T05:22:13.0614509Z Running tests...
2022-05-18T05:22:13.0614948Z ----------------------------------------------------------------------
2022-05-18T05:22:14.7237487Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:14.7610230Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74431
2022-05-18T05:22:14.7730132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74432
2022-05-18T05:22:15.9402880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:15.9403486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:15.9404267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:15.9404974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:15.9411819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:15.9413018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:16.1778809Z ok (3.116s)
2022-05-18T05:22:16.1779017Z 
2022-05-18T05:22:16.1779646Z ----------------------------------------------------------------------
2022-05-18T05:22:16.1780008Z Ran 1 test in 3.116s
2022-05-18T05:22:16.1780175Z 
2022-05-18T05:22:16.1780612Z OK
2022-05-18T05:22:16.1780796Z 
2022-05-18T05:22:16.1780954Z Generating XML reports...
2022-05-18T05:22:16.1838231Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052213.xml
2022-05-18T05:22:17.5826301Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:22:17.5840838Z 
2022-05-18T05:22:17.5841269Z Running tests...
2022-05-18T05:22:17.5841761Z ----------------------------------------------------------------------
2022-05-18T05:22:19.1894454Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:19.2267043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74550
2022-05-18T05:22:19.2378159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74551
2022-05-18T05:22:20.3970055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:20.3970851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:20.3971893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:20.3972602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:20.4079742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:20.4982719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:20.7429092Z ok (3.158s)
2022-05-18T05:22:20.7429313Z 
2022-05-18T05:22:20.7429693Z ----------------------------------------------------------------------
2022-05-18T05:22:20.7430060Z Ran 1 test in 3.159s
2022-05-18T05:22:20.7430231Z 
2022-05-18T05:22:20.7430306Z OK
2022-05-18T05:22:20.7430439Z 
2022-05-18T05:22:20.7430572Z Generating XML reports...
2022-05-18T05:22:20.7486689Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052217.xml
2022-05-18T05:22:22.1848852Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:22:22.1865105Z 
2022-05-18T05:22:22.1865394Z Running tests...
2022-05-18T05:22:22.1866100Z ----------------------------------------------------------------------
2022-05-18T05:22:23.8562447Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:23.8935087Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74665
2022-05-18T05:22:23.9046157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74666
2022-05-18T05:22:25.0746771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:25.0747354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:25.0748160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:25.0748870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:25.0855481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:25.1760834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:25.4097988Z ok (3.223s)
2022-05-18T05:22:25.4098343Z 
2022-05-18T05:22:25.4099120Z ----------------------------------------------------------------------
2022-05-18T05:22:25.4099797Z Ran 1 test in 3.223s
2022-05-18T05:22:25.4099971Z 
2022-05-18T05:22:25.4100295Z OK
2022-05-18T05:22:25.4100458Z 
2022-05-18T05:22:25.4100596Z Generating XML reports...
2022-05-18T05:22:25.4157598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052222.xml
2022-05-18T05:22:26.8397321Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:22:26.8413225Z 
2022-05-18T05:22:26.8413658Z Running tests...
2022-05-18T05:22:26.8414181Z ----------------------------------------------------------------------
2022-05-18T05:22:28.4790179Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:28.5165398Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74784
2022-05-18T05:22:28.5276141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74785
2022-05-18T05:22:29.6888066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:29.6888646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:29.6889464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:29.6890747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:29.6897782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:29.6898293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:29.9327626Z ok (3.091s)
2022-05-18T05:22:29.9327846Z 
2022-05-18T05:22:29.9328209Z ----------------------------------------------------------------------
2022-05-18T05:22:29.9328558Z Ran 1 test in 3.091s
2022-05-18T05:22:29.9328721Z 
2022-05-18T05:22:29.9331343Z OK
2022-05-18T05:22:29.9331549Z 
2022-05-18T05:22:29.9332069Z Generating XML reports...
2022-05-18T05:22:29.9386454Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052226.xml
2022-05-18T05:22:31.3640355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:22:31.3656487Z 
2022-05-18T05:22:31.3656647Z Running tests...
2022-05-18T05:22:31.3657092Z ----------------------------------------------------------------------
2022-05-18T05:22:33.0277980Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:33.0659928Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74903
2022-05-18T05:22:33.0772556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74904
2022-05-18T05:22:34.2725159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:34.2725754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:34.2726539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:34.2727255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:34.2833856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:34.3737533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:34.5827142Z ok (3.217s)
2022-05-18T05:22:34.5827540Z 
2022-05-18T05:22:34.5828038Z ----------------------------------------------------------------------
2022-05-18T05:22:34.5828380Z Ran 1 test in 3.217s
2022-05-18T05:22:34.5828549Z 
2022-05-18T05:22:34.5828645Z OK
2022-05-18T05:22:34.5828788Z 
2022-05-18T05:22:34.5828906Z Generating XML reports...
2022-05-18T05:22:34.5887653Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052231.xml
2022-05-18T05:22:36.0149490Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:22:36.0164201Z 
2022-05-18T05:22:36.0164611Z Running tests...
2022-05-18T05:22:36.0165094Z ----------------------------------------------------------------------
2022-05-18T05:22:37.7009837Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:37.7392172Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75022
2022-05-18T05:22:37.7505126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75023
2022-05-18T05:22:38.9529703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:38.9530487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:38.9531275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:38.9532200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:38.9539265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:38.9539741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:39.1553693Z ok (3.139s)
2022-05-18T05:22:39.1553884Z 
2022-05-18T05:22:39.1554252Z ----------------------------------------------------------------------
2022-05-18T05:22:39.1554577Z Ran 1 test in 3.139s
2022-05-18T05:22:39.1554743Z 
2022-05-18T05:22:39.1554839Z OK
2022-05-18T05:22:39.1554974Z 
2022-05-18T05:22:39.1555110Z Generating XML reports...
2022-05-18T05:22:39.1612295Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052236.xml
2022-05-18T05:22:40.5766918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:22:40.5782435Z 
2022-05-18T05:22:40.5782741Z Running tests...
2022-05-18T05:22:40.5783192Z ----------------------------------------------------------------------
2022-05-18T05:22:42.2358102Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:42.2738543Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75137
2022-05-18T05:22:42.2850811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75138
2022-05-18T05:22:43.4506313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:43.4506860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:43.4507670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:43.4508382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:43.4515911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:43.4516393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:43.4726077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:22:43.4726601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:22:43.4727297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:22:43.4728315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:22:43.7900821Z ok (3.212s)
2022-05-18T05:22:43.7901041Z 
2022-05-18T05:22:43.7901429Z ----------------------------------------------------------------------
2022-05-18T05:22:43.7901765Z Ran 1 test in 3.212s
2022-05-18T05:22:43.7901933Z 
2022-05-18T05:22:43.7902033Z OK
2022-05-18T05:22:43.7902167Z 
2022-05-18T05:22:43.7902300Z Generating XML reports...
2022-05-18T05:22:43.7958486Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052240.xml
2022-05-18T05:22:45.2170876Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:22:45.2186301Z 
2022-05-18T05:22:45.2186557Z Running tests...
2022-05-18T05:22:45.2187005Z ----------------------------------------------------------------------
2022-05-18T05:22:46.8769242Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:46.9152738Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75262
2022-05-18T05:22:46.9266709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75263
2022-05-18T05:22:48.1038613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:48.1039718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:48.1040516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:48.1041221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:48.1047334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:48.1047825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:48.1156111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:22:48.1156629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:22:48.1157576Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:22:48.1158414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:22:48.3314265Z ok (3.112s)
2022-05-18T05:22:48.3314638Z 
2022-05-18T05:22:48.3315391Z ----------------------------------------------------------------------
2022-05-18T05:22:48.3316068Z Ran 1 test in 3.113s
2022-05-18T05:22:48.3316297Z 
2022-05-18T05:22:48.3316396Z OK
2022-05-18T05:22:48.3316513Z 
2022-05-18T05:22:48.3316646Z Generating XML reports...
2022-05-18T05:22:48.3372182Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052245.xml
2022-05-18T05:22:49.7535917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:22:49.7551369Z 
2022-05-18T05:22:49.7551822Z Running tests...
2022-05-18T05:22:49.7552319Z ----------------------------------------------------------------------
2022-05-18T05:22:51.4288916Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:51.4661679Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75387
2022-05-18T05:22:51.4771145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75388
2022-05-18T05:22:52.6933930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:52.6934489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:52.6935488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:52.6936225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:52.6942999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:52.6944975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:52.7150822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:22:52.7151340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:22:52.7152025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:22:52.7152960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:22:52.9820877Z ok (3.227s)
2022-05-18T05:22:52.9821095Z 
2022-05-18T05:22:52.9821454Z ----------------------------------------------------------------------
2022-05-18T05:22:52.9822063Z Ran 1 test in 3.227s
2022-05-18T05:22:52.9822209Z 
2022-05-18T05:22:52.9822315Z OK
2022-05-18T05:22:52.9822452Z 
2022-05-18T05:22:52.9822587Z Generating XML reports...
2022-05-18T05:22:52.9888279Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052249.xml
2022-05-18T05:22:54.4219948Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:22:54.4234865Z 
2022-05-18T05:22:54.4235200Z Running tests...
2022-05-18T05:22:54.4235646Z ----------------------------------------------------------------------
2022-05-18T05:22:56.0697366Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:56.1080341Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75512
2022-05-18T05:22:56.1192985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75513
2022-05-18T05:22:57.3464433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:57.3465024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:57.3465828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:57.3466537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:57.3576237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:57.4477818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:57.4592616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:22:57.4593165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:22:57.4593898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:22:57.4594583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:22:57.7244880Z ok (3.301s)
2022-05-18T05:22:57.7245125Z 
2022-05-18T05:22:57.7245530Z ----------------------------------------------------------------------
2022-05-18T05:22:57.7245872Z Ran 1 test in 3.301s
2022-05-18T05:22:57.7246040Z 
2022-05-18T05:22:57.7246142Z OK
2022-05-18T05:22:57.7246290Z 
2022-05-18T05:22:57.7246411Z Generating XML reports...
2022-05-18T05:22:57.7313544Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052254.xml
2022-05-18T05:22:59.1516873Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:22:59.1531628Z 
2022-05-18T05:22:59.1531912Z Running tests...
2022-05-18T05:22:59.1532398Z ----------------------------------------------------------------------
2022-05-18T05:23:00.8200598Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:00.8588438Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75637
2022-05-18T05:23:00.8702748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75638
2022-05-18T05:23:02.0741388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:02.0741971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:02.0742754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:02.0743464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:02.0750831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:02.0751734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:02.2751367Z skip: Skipped due to small world size. (3.122s)
2022-05-18T05:23:02.2751619Z 
2022-05-18T05:23:02.2752002Z ----------------------------------------------------------------------
2022-05-18T05:23:02.2752346Z Ran 1 test in 3.122s
2022-05-18T05:23:02.2752491Z 
2022-05-18T05:23:02.2752801Z OK (skipped=1)
2022-05-18T05:23:02.2752968Z 
2022-05-18T05:23:02.2753105Z Generating XML reports...
2022-05-18T05:23:02.2810445Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052259.xml
2022-05-18T05:23:03.7188166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:23:03.7202974Z 
2022-05-18T05:23:03.7203284Z Running tests...
2022-05-18T05:23:03.7203742Z ----------------------------------------------------------------------
2022-05-18T05:23:05.3853474Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:05.4235435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75752
2022-05-18T05:23:05.4347630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75753
2022-05-18T05:23:06.6228209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:06.6228773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:06.6229594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:06.6230283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:06.6337670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:06.7243286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:06.9396759Z skip: Skipped due to small world size. (3.219s)
2022-05-18T05:23:06.9397024Z 
2022-05-18T05:23:06.9397400Z ----------------------------------------------------------------------
2022-05-18T05:23:06.9397743Z Ran 1 test in 3.219s
2022-05-18T05:23:06.9397912Z 
2022-05-18T05:23:06.9398021Z OK (skipped=1)
2022-05-18T05:23:06.9398178Z 
2022-05-18T05:23:06.9400596Z Generating XML reports...
2022-05-18T05:23:06.9455014Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052303.xml
2022-05-18T05:23:08.3799063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:23:08.3814043Z 
2022-05-18T05:23:08.3814172Z Running tests...
2022-05-18T05:23:08.3815104Z ----------------------------------------------------------------------
2022-05-18T05:23:10.0367002Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:10.0748917Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75867
2022-05-18T05:23:10.0861373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75868
2022-05-18T05:23:11.3020439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:11.3021282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:11.3022109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:11.3022822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:11.3028990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:11.3030379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:11.4910332Z skip: Skipped due to small world size. (3.109s)
2022-05-18T05:23:11.4910816Z 
2022-05-18T05:23:11.4911449Z ----------------------------------------------------------------------
2022-05-18T05:23:11.4912075Z Ran 1 test in 3.110s
2022-05-18T05:23:11.4912366Z 
2022-05-18T05:23:11.4912562Z OK (skipped=1)
2022-05-18T05:23:11.4912868Z 
2022-05-18T05:23:11.4913100Z Generating XML reports...
2022-05-18T05:23:11.4970961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052308.xml
2022-05-18T05:23:12.9152911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:23:12.9168144Z 
2022-05-18T05:23:12.9168366Z Running tests...
2022-05-18T05:23:12.9168816Z ----------------------------------------------------------------------
2022-05-18T05:23:14.5624468Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:14.5999489Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75982
2022-05-18T05:23:14.6112056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75983
2022-05-18T05:23:15.8239931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:15.8240730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:15.8241808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:15.8343320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:15.8350869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:15.9257246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:16.1160520Z skip: Skipped due to small world size. (3.199s)
2022-05-18T05:23:16.1160738Z 
2022-05-18T05:23:16.1161398Z ----------------------------------------------------------------------
2022-05-18T05:23:16.1161754Z Ran 1 test in 3.199s
2022-05-18T05:23:16.1161923Z 
2022-05-18T05:23:16.1162042Z OK (skipped=1)
2022-05-18T05:23:16.1162250Z 
2022-05-18T05:23:16.1162382Z Generating XML reports...
2022-05-18T05:23:16.1220416Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052312.xml
2022-05-18T05:23:17.5507558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:23:17.5522092Z 
2022-05-18T05:23:17.5522351Z Running tests...
2022-05-18T05:23:17.5522780Z ----------------------------------------------------------------------
2022-05-18T05:23:19.2121988Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:19.2504197Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76097
2022-05-18T05:23:19.2616532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76098
2022-05-18T05:23:20.4767681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:20.4768242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:20.4769055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:20.4770053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:20.4876563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:20.5780134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:20.7666421Z ok (3.214s)
2022-05-18T05:23:20.7666651Z 
2022-05-18T05:23:20.7667040Z ----------------------------------------------------------------------
2022-05-18T05:23:20.7667355Z Ran 1 test in 3.214s
2022-05-18T05:23:20.7667524Z 
2022-05-18T05:23:20.7667628Z OK
2022-05-18T05:23:20.7667762Z 
2022-05-18T05:23:20.7667904Z Generating XML reports...
2022-05-18T05:23:20.7723617Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052317.xml
2022-05-18T05:23:22.1940452Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:23:22.1955970Z 
2022-05-18T05:23:22.1956398Z Running tests...
2022-05-18T05:23:22.1956876Z ----------------------------------------------------------------------
2022-05-18T05:23:23.8530272Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:23.8910717Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76216
2022-05-18T05:23:23.9022949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76217
2022-05-18T05:23:25.1082646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:25.1083724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:25.1085201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:25.1086171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:25.1191091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:25.2095450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:25.4072975Z ok (3.211s)
2022-05-18T05:23:25.4073198Z 
2022-05-18T05:23:25.4073587Z ----------------------------------------------------------------------
2022-05-18T05:23:25.4073912Z Ran 1 test in 3.212s
2022-05-18T05:23:25.4074087Z 
2022-05-18T05:23:25.4074181Z OK
2022-05-18T05:23:25.4074320Z 
2022-05-18T05:23:25.4074456Z Generating XML reports...
2022-05-18T05:23:25.4130860Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052322.xml
2022-05-18T05:23:26.8138491Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:23:26.8153583Z 
2022-05-18T05:23:26.8153971Z Running tests...
2022-05-18T05:23:26.8154782Z ----------------------------------------------------------------------
2022-05-18T05:23:28.4666198Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:28.5040469Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76335
2022-05-18T05:23:28.5152812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76336
2022-05-18T05:23:29.6826660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:29.6827240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:29.6828058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:29.6828766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:29.6835406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:29.6835910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:31.7233450Z ok (4.908s)
2022-05-18T05:23:31.7233678Z 
2022-05-18T05:23:31.7234079Z ----------------------------------------------------------------------
2022-05-18T05:23:31.7234431Z Ran 1 test in 4.908s
2022-05-18T05:23:31.7234580Z 
2022-05-18T05:23:31.7234678Z OK
2022-05-18T05:23:31.7234818Z 
2022-05-18T05:23:31.7234959Z Generating XML reports...
2022-05-18T05:23:31.7292818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052326.xml
2022-05-18T05:23:33.1666248Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:23:33.1681213Z 
2022-05-18T05:23:33.1681403Z Running tests...
2022-05-18T05:23:33.1681857Z ----------------------------------------------------------------------
2022-05-18T05:23:34.8251351Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:34.8624404Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76456
2022-05-18T05:23:34.8735765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76457
2022-05-18T05:23:36.0900231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:36.0900803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:36.0901590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:36.0902299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:36.0909974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:36.0911200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:38.0816026Z ok (4.913s)
2022-05-18T05:23:38.0816259Z 
2022-05-18T05:23:38.0816675Z ----------------------------------------------------------------------
2022-05-18T05:23:38.0817017Z Ran 1 test in 4.913s
2022-05-18T05:23:38.0817182Z 
2022-05-18T05:23:38.0817256Z OK
2022-05-18T05:23:38.0818301Z 
2022-05-18T05:23:38.0818462Z Generating XML reports...
2022-05-18T05:23:38.0878002Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052333.xml
2022-05-18T05:23:39.5367435Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:23:39.5383017Z 
2022-05-18T05:23:39.5383321Z Running tests...
2022-05-18T05:23:39.5383769Z ----------------------------------------------------------------------
2022-05-18T05:23:41.1993970Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:41.2375469Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76577
2022-05-18T05:23:41.2486895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76578
2022-05-18T05:23:42.4800366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:42.4801441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:42.4802891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:42.4804302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:42.4810929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:42.4811935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:42.6533864Z ok (3.115s)
2022-05-18T05:23:42.6534142Z 
2022-05-18T05:23:42.6534690Z ----------------------------------------------------------------------
2022-05-18T05:23:42.6535314Z Ran 1 test in 3.115s
2022-05-18T05:23:42.6535482Z 
2022-05-18T05:23:42.6535555Z OK
2022-05-18T05:23:42.6535701Z 
2022-05-18T05:23:42.6535833Z Generating XML reports...
2022-05-18T05:23:42.6591709Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052339.xml
2022-05-18T05:23:44.0861519Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:23:44.0876647Z 
2022-05-18T05:23:44.0877056Z Running tests...
2022-05-18T05:23:44.0877537Z ----------------------------------------------------------------------
2022-05-18T05:23:45.7382322Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:45.7755787Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76696
2022-05-18T05:23:45.7867851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76697
2022-05-18T05:23:47.0339493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:47.0340050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:47.0340853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:47.0341568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:47.0348664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:47.0349473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:48.7945749Z ok (4.707s)
2022-05-18T05:23:48.7945988Z 
2022-05-18T05:23:48.7946933Z ----------------------------------------------------------------------
2022-05-18T05:23:48.7947573Z Ran 1 test in 4.707s
2022-05-18T05:23:48.7947906Z 
2022-05-18T05:23:48.7948073Z OK
2022-05-18T05:23:48.7948344Z 
2022-05-18T05:23:48.7948592Z Generating XML reports...
2022-05-18T05:23:48.8006920Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052344.xml
2022-05-18T05:23:50.2463274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:23:50.2478582Z 
2022-05-18T05:23:50.2478798Z Running tests...
2022-05-18T05:23:50.2479242Z ----------------------------------------------------------------------
2022-05-18T05:23:51.8941684Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:51.9315595Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76813
2022-05-18T05:23:51.9427185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76814
2022-05-18T05:23:53.1366651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:53.1367248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:53.1368045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:53.1368727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:53.1475333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:53.2379422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:53.4477504Z ok (3.200s)
2022-05-18T05:23:53.4477711Z 
2022-05-18T05:23:53.4478104Z ----------------------------------------------------------------------
2022-05-18T05:23:53.4478443Z Ran 1 test in 3.200s
2022-05-18T05:23:53.4478610Z 
2022-05-18T05:23:53.4478712Z OK
2022-05-18T05:23:53.4479096Z 
2022-05-18T05:23:53.4479211Z Generating XML reports...
2022-05-18T05:23:53.4536232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052350.xml
2022-05-18T05:23:54.8712140Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:23:54.8727185Z 
2022-05-18T05:23:54.8727606Z Running tests...
2022-05-18T05:23:54.8728376Z ----------------------------------------------------------------------
2022-05-18T05:23:56.5182122Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:56.5564205Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76932
2022-05-18T05:23:56.5675384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76933
2022-05-18T05:23:57.7396833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:57.7397417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:57.7398225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:57.7398934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:57.7506113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:57.8410149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:58.0727205Z ok (3.200s)
2022-05-18T05:23:58.0727429Z 
2022-05-18T05:23:58.0727814Z ----------------------------------------------------------------------
2022-05-18T05:23:58.0728157Z Ran 1 test in 3.200s
2022-05-18T05:23:58.0728327Z 
2022-05-18T05:23:58.0728422Z OK
2022-05-18T05:23:58.0728559Z 
2022-05-18T05:23:58.0730644Z Generating XML reports...
2022-05-18T05:23:58.0785762Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052354.xml
2022-05-18T05:23:59.5048562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:23:59.5064175Z 
2022-05-18T05:23:59.5064573Z Running tests...
2022-05-18T05:23:59.5065100Z ----------------------------------------------------------------------
2022-05-18T05:24:01.1568267Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:01.1942438Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77051
2022-05-18T05:24:01.2053621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77052
2022-05-18T05:24:02.4246483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:02.4247076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:02.4247878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:02.4248584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:02.4356639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:02.5258092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:02.7104607Z ok (3.204s)
2022-05-18T05:24:02.7104844Z 
2022-05-18T05:24:02.7105241Z ----------------------------------------------------------------------
2022-05-18T05:24:02.7105590Z Ran 1 test in 3.204s
2022-05-18T05:24:02.7105739Z 
2022-05-18T05:24:02.7105865Z OK
2022-05-18T05:24:02.7106009Z 
2022-05-18T05:24:02.7106148Z Generating XML reports...
2022-05-18T05:24:02.7162208Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052359.xml
2022-05-18T05:24:04.1315278Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:04.1329691Z 
2022-05-18T05:24:04.1330112Z Running tests...
2022-05-18T05:24:04.1330622Z ----------------------------------------------------------------------
2022-05-18T05:24:05.7880711Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:05.8261658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77170
2022-05-18T05:24:05.8374357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77171
2022-05-18T05:24:07.0310895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:07.0311477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:07.0312260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:07.0312956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:07.0421207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:07.1326093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:11.1495391Z ok (7.016s)
2022-05-18T05:24:11.1495634Z 
2022-05-18T05:24:11.1496013Z ----------------------------------------------------------------------
2022-05-18T05:24:11.1496362Z Ran 1 test in 7.017s
2022-05-18T05:24:11.1496534Z 
2022-05-18T05:24:11.1496631Z OK
2022-05-18T05:24:11.1496768Z 
2022-05-18T05:24:11.1496918Z Generating XML reports...
2022-05-18T05:24:11.1552877Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052404.xml
2022-05-18T05:24:12.5797288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:12.5812880Z 
2022-05-18T05:24:12.5813134Z Running tests...
2022-05-18T05:24:12.5813588Z ----------------------------------------------------------------------
2022-05-18T05:24:14.2313796Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:14.2694677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77293
2022-05-18T05:24:14.2806233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77294
2022-05-18T05:24:15.4803842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:15.4804640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:15.4805474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:15.4806456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:15.4913139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:15.5820882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:19.5927508Z ok (7.011s)
2022-05-18T05:24:19.5927737Z 
2022-05-18T05:24:19.5928120Z ----------------------------------------------------------------------
2022-05-18T05:24:19.5928444Z Ran 1 test in 7.011s
2022-05-18T05:24:19.5928614Z 
2022-05-18T05:24:19.5928710Z OK
2022-05-18T05:24:19.5928847Z 
2022-05-18T05:24:19.5928989Z Generating XML reports...
2022-05-18T05:24:19.5986501Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052412.xml
2022-05-18T05:24:21.0227117Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:21.0241985Z 
2022-05-18T05:24:21.0242183Z Running tests...
2022-05-18T05:24:21.0242617Z ----------------------------------------------------------------------
2022-05-18T05:24:22.6437361Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:22.6812778Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77416
2022-05-18T05:24:22.6926453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77417
2022-05-18T05:24:23.8783544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:23.8784143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:23.8784949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:23.8785670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:23.8894019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:23.9799297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:28.0043808Z ok (6.980s)
2022-05-18T05:24:28.0044042Z 
2022-05-18T05:24:28.0044415Z ----------------------------------------------------------------------
2022-05-18T05:24:28.0044759Z Ran 1 test in 6.980s
2022-05-18T05:24:28.0044926Z 
2022-05-18T05:24:28.0045018Z OK
2022-05-18T05:24:28.0045151Z 
2022-05-18T05:24:28.0045283Z Generating XML reports...
2022-05-18T05:24:28.0102198Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052421.xml
2022-05-18T05:24:29.4552943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:29.4568124Z 
2022-05-18T05:24:29.4568556Z Running tests...
2022-05-18T05:24:29.4568996Z ----------------------------------------------------------------------
2022-05-18T05:24:29.4588471Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T05:24:29.4588774Z 
2022-05-18T05:24:29.4589034Z ----------------------------------------------------------------------
2022-05-18T05:24:29.4589366Z Ran 1 test in 0.002s
2022-05-18T05:24:29.4589532Z 
2022-05-18T05:24:29.4589643Z OK (skipped=1)
2022-05-18T05:24:29.4589799Z 
2022-05-18T05:24:29.4589927Z Generating XML reports...
2022-05-18T05:24:29.4632310Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052429.xml
2022-05-18T05:24:30.7132525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:30.7147673Z 
2022-05-18T05:24:30.7147829Z Running tests...
2022-05-18T05:24:30.7148483Z ----------------------------------------------------------------------
2022-05-18T05:24:30.7167963Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T05:24:30.7168285Z 
2022-05-18T05:24:30.7168566Z ----------------------------------------------------------------------
2022-05-18T05:24:30.7168896Z Ran 1 test in 0.002s
2022-05-18T05:24:30.7169059Z 
2022-05-18T05:24:30.7169345Z OK (skipped=1)
2022-05-18T05:24:30.7169784Z 
2022-05-18T05:24:30.7169913Z Generating XML reports...
2022-05-18T05:24:30.7212046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052430.xml
2022-05-18T05:24:31.9495815Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:31.9511947Z 
2022-05-18T05:24:31.9512300Z Running tests...
2022-05-18T05:24:31.9512737Z ----------------------------------------------------------------------
2022-05-18T05:24:31.9534181Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T05:24:31.9534808Z 
2022-05-18T05:24:31.9535096Z ----------------------------------------------------------------------
2022-05-18T05:24:31.9535426Z Ran 1 test in 0.002s
2022-05-18T05:24:31.9535594Z 
2022-05-18T05:24:31.9535702Z OK (skipped=1)
2022-05-18T05:24:31.9535858Z 
2022-05-18T05:24:31.9535964Z Generating XML reports...
2022-05-18T05:24:31.9578387Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052431.xml
2022-05-18T05:24:33.2208455Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:33.2223791Z 
2022-05-18T05:24:33.2224202Z Running tests...
2022-05-18T05:24:33.2224725Z ----------------------------------------------------------------------
2022-05-18T05:24:33.2245707Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T05:24:33.2246205Z 
2022-05-18T05:24:33.2246787Z ----------------------------------------------------------------------
2022-05-18T05:24:33.2247432Z Ran 1 test in 0.002s
2022-05-18T05:24:33.2247612Z 
2022-05-18T05:24:33.2247721Z OK (skipped=1)
2022-05-18T05:24:33.2247877Z 
2022-05-18T05:24:33.2247984Z Generating XML reports...
2022-05-18T05:24:33.2289204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052433.xml
2022-05-18T05:24:34.5046628Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:34.5062244Z 
2022-05-18T05:24:34.5062662Z Running tests...
2022-05-18T05:24:34.5063157Z ----------------------------------------------------------------------
2022-05-18T05:24:34.5083056Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T05:24:34.5083377Z 
2022-05-18T05:24:34.5083665Z ----------------------------------------------------------------------
2022-05-18T05:24:34.5084019Z Ran 1 test in 0.002s
2022-05-18T05:24:34.5084166Z 
2022-05-18T05:24:34.5084274Z OK (skipped=1)
2022-05-18T05:24:34.5084431Z 
2022-05-18T05:24:34.5084557Z Generating XML reports...
2022-05-18T05:24:34.5126911Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052434.xml
2022-05-18T05:24:35.7544328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:35.7560967Z 
2022-05-18T05:24:35.7561561Z Running tests...
2022-05-18T05:24:35.7562410Z ----------------------------------------------------------------------
2022-05-18T05:24:35.7585332Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T05:24:35.7586012Z 
2022-05-18T05:24:35.7586582Z ----------------------------------------------------------------------
2022-05-18T05:24:35.7587219Z Ran 1 test in 0.002s
2022-05-18T05:24:35.7587547Z 
2022-05-18T05:24:35.7587752Z OK (skipped=1)
2022-05-18T05:24:35.7588042Z 
2022-05-18T05:24:35.7588279Z Generating XML reports...
2022-05-18T05:24:35.7631327Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052435.xml
2022-05-18T05:24:37.0378580Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:37.0393575Z 
2022-05-18T05:24:37.0393721Z Running tests...
2022-05-18T05:24:37.0394448Z ----------------------------------------------------------------------
2022-05-18T05:24:37.0414488Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T05:24:37.0414814Z 
2022-05-18T05:24:37.0415139Z ----------------------------------------------------------------------
2022-05-18T05:24:37.0415477Z Ran 1 test in 0.002s
2022-05-18T05:24:37.0415651Z 
2022-05-18T05:24:37.0415744Z OK (skipped=1)
2022-05-18T05:24:37.0416189Z 
2022-05-18T05:24:37.0416317Z Generating XML reports...
2022-05-18T05:24:37.0458243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052437.xml
2022-05-18T05:24:38.3301269Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:38.3316434Z 
2022-05-18T05:24:38.3317027Z Running tests...
2022-05-18T05:24:38.3317645Z ----------------------------------------------------------------------
2022-05-18T05:24:38.3338962Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:24:38.3339425Z 
2022-05-18T05:24:38.3339869Z ----------------------------------------------------------------------
2022-05-18T05:24:38.3340207Z Ran 1 test in 0.002s
2022-05-18T05:24:38.3340355Z 
2022-05-18T05:24:38.3340464Z OK (skipped=1)
2022-05-18T05:24:38.3341651Z 
2022-05-18T05:24:38.3342288Z Generating XML reports...
2022-05-18T05:24:38.3383019Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052438.xml
2022-05-18T05:24:39.5791510Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:39.5807439Z 
2022-05-18T05:24:39.5807687Z Running tests...
2022-05-18T05:24:39.5808268Z ----------------------------------------------------------------------
2022-05-18T05:24:39.5828893Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:24:39.5829476Z 
2022-05-18T05:24:39.5829766Z ----------------------------------------------------------------------
2022-05-18T05:24:39.5830099Z Ran 1 test in 0.002s
2022-05-18T05:24:39.5830265Z 
2022-05-18T05:24:39.5830375Z OK (skipped=1)
2022-05-18T05:24:39.5830531Z 
2022-05-18T05:24:39.5830655Z Generating XML reports...
2022-05-18T05:24:39.5873847Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052439.xml
2022-05-18T05:24:40.8295343Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:40.8310454Z 
2022-05-18T05:24:40.8310684Z Running tests...
2022-05-18T05:24:40.8311257Z ----------------------------------------------------------------------
2022-05-18T05:24:40.8331841Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:24:40.8332467Z 
2022-05-18T05:24:40.8332950Z ----------------------------------------------------------------------
2022-05-18T05:24:40.8333313Z Ran 1 test in 0.002s
2022-05-18T05:24:40.8333485Z 
2022-05-18T05:24:40.8333597Z OK (skipped=1)
2022-05-18T05:24:40.8334026Z 
2022-05-18T05:24:40.8334168Z Generating XML reports...
2022-05-18T05:24:40.8376057Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052440.xml
2022-05-18T05:24:42.1115046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:42.1129696Z 
2022-05-18T05:24:42.1130059Z Running tests...
2022-05-18T05:24:42.1130842Z ----------------------------------------------------------------------
2022-05-18T05:24:42.1151736Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:24:42.1152077Z 
2022-05-18T05:24:42.1152341Z ----------------------------------------------------------------------
2022-05-18T05:24:42.1152670Z Ran 1 test in 0.002s
2022-05-18T05:24:42.1152839Z 
2022-05-18T05:24:42.1152948Z OK (skipped=1)
2022-05-18T05:24:42.1153103Z 
2022-05-18T05:24:42.1153248Z Generating XML reports...
2022-05-18T05:24:42.1195444Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052442.xml
2022-05-18T05:24:43.3902337Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:43.3916761Z 
2022-05-18T05:24:43.3917172Z Running tests...
2022-05-18T05:24:43.3917755Z ----------------------------------------------------------------------
2022-05-18T05:24:43.3939456Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:24:43.3939810Z 
2022-05-18T05:24:43.3940173Z ----------------------------------------------------------------------
2022-05-18T05:24:43.3940723Z Ran 1 test in 0.002s
2022-05-18T05:24:43.3940898Z 
2022-05-18T05:24:43.3941007Z OK (skipped=1)
2022-05-18T05:24:43.3941162Z 
2022-05-18T05:24:43.3941287Z Generating XML reports...
2022-05-18T05:24:43.3982683Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052443.xml
2022-05-18T05:24:44.6813893Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:44.6829149Z 
2022-05-18T05:24:44.6829442Z Running tests...
2022-05-18T05:24:44.6829878Z ----------------------------------------------------------------------
2022-05-18T05:24:44.6850169Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:24:44.6851189Z 
2022-05-18T05:24:44.6851764Z ----------------------------------------------------------------------
2022-05-18T05:24:44.6852160Z Ran 1 test in 0.002s
2022-05-18T05:24:44.6852325Z 
2022-05-18T05:24:44.6852433Z OK (skipped=1)
2022-05-18T05:24:44.6852591Z 
2022-05-18T05:24:44.6852697Z Generating XML reports...
2022-05-18T05:24:44.6894176Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052444.xml
2022-05-18T05:24:45.9426998Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:45.9442016Z 
2022-05-18T05:24:45.9442255Z Running tests...
2022-05-18T05:24:45.9442674Z ----------------------------------------------------------------------
2022-05-18T05:24:45.9464389Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:24:45.9464748Z 
2022-05-18T05:24:45.9465274Z ----------------------------------------------------------------------
2022-05-18T05:24:45.9466000Z Ran 1 test in 0.002s
2022-05-18T05:24:45.9466301Z 
2022-05-18T05:24:45.9466436Z OK (skipped=1)
2022-05-18T05:24:45.9466595Z 
2022-05-18T05:24:45.9466720Z Generating XML reports...
2022-05-18T05:24:45.9507414Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052445.xml
2022-05-18T05:24:47.2200277Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:47.2215354Z 
2022-05-18T05:24:47.2215560Z Running tests...
2022-05-18T05:24:47.2216204Z ----------------------------------------------------------------------
2022-05-18T05:24:47.2236287Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:24:47.2236854Z 
2022-05-18T05:24:47.2237157Z ----------------------------------------------------------------------
2022-05-18T05:24:47.2237466Z Ran 1 test in 0.002s
2022-05-18T05:24:47.2237630Z 
2022-05-18T05:24:47.2237744Z OK (skipped=1)
2022-05-18T05:24:47.2237900Z 
2022-05-18T05:24:47.2238027Z Generating XML reports...
2022-05-18T05:24:47.2279659Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052447.xml
2022-05-18T05:24:48.5145331Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:48.5160287Z 
2022-05-18T05:24:48.5160707Z Running tests...
2022-05-18T05:24:48.5161272Z ----------------------------------------------------------------------
2022-05-18T05:24:48.5182451Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:24:48.5183132Z 
2022-05-18T05:24:48.5183745Z ----------------------------------------------------------------------
2022-05-18T05:24:48.5184296Z Ran 1 test in 0.002s
2022-05-18T05:24:48.5184460Z 
2022-05-18T05:24:48.5184569Z OK (skipped=1)
2022-05-18T05:24:48.5184708Z 
2022-05-18T05:24:48.5184831Z Generating XML reports...
2022-05-18T05:24:48.5227152Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052448.xml
2022-05-18T05:24:49.7544188Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:49.7560061Z 
2022-05-18T05:24:49.7560683Z Running tests...
2022-05-18T05:24:49.7562101Z ----------------------------------------------------------------------
2022-05-18T05:24:49.7583018Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:24:49.7583669Z 
2022-05-18T05:24:49.7584253Z ----------------------------------------------------------------------
2022-05-18T05:24:49.7584903Z Ran 1 test in 0.002s
2022-05-18T05:24:49.7585222Z 
2022-05-18T05:24:49.7585428Z OK (skipped=1)
2022-05-18T05:24:49.7585719Z 
2022-05-18T05:24:49.7585926Z Generating XML reports...
2022-05-18T05:24:49.7630134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052449.xml
2022-05-18T05:24:51.0017708Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:51.0032663Z 
2022-05-18T05:24:51.0032831Z Running tests...
2022-05-18T05:24:51.0033510Z ----------------------------------------------------------------------
2022-05-18T05:24:51.0054014Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:24:51.0054542Z 
2022-05-18T05:24:51.0054821Z ----------------------------------------------------------------------
2022-05-18T05:24:51.0055153Z Ran 1 test in 0.002s
2022-05-18T05:24:51.0055301Z 
2022-05-18T05:24:51.0055412Z OK (skipped=1)
2022-05-18T05:24:51.0055567Z 
2022-05-18T05:24:51.0055692Z Generating XML reports...
2022-05-18T05:24:51.0097977Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052451.xml
2022-05-18T05:24:52.2443311Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:52.2460695Z 
2022-05-18T05:24:52.2461173Z Running tests...
2022-05-18T05:24:52.2461963Z ----------------------------------------------------------------------
2022-05-18T05:24:52.2483444Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:24:52.2484090Z 
2022-05-18T05:24:52.2484422Z ----------------------------------------------------------------------
2022-05-18T05:24:52.2484741Z Ran 1 test in 0.002s
2022-05-18T05:24:52.2484905Z 
2022-05-18T05:24:52.2485017Z OK (skipped=1)
2022-05-18T05:24:52.2485177Z 
2022-05-18T05:24:52.2485301Z Generating XML reports...
2022-05-18T05:24:52.2529865Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052452.xml
2022-05-18T05:24:53.5308302Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:53.5324366Z 
2022-05-18T05:24:53.5324871Z Running tests...
2022-05-18T05:24:53.5325360Z ----------------------------------------------------------------------
2022-05-18T05:24:53.5347677Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:24:53.5348332Z 
2022-05-18T05:24:53.5348932Z ----------------------------------------------------------------------
2022-05-18T05:24:53.5349275Z Ran 1 test in 0.002s
2022-05-18T05:24:53.5349436Z 
2022-05-18T05:24:53.5349545Z OK (skipped=1)
2022-05-18T05:24:53.5349698Z 
2022-05-18T05:24:53.5349821Z Generating XML reports...
2022-05-18T05:24:53.5392411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052453.xml
2022-05-18T05:24:54.8094106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:54.8109428Z 
2022-05-18T05:24:54.8109701Z Running tests...
2022-05-18T05:24:54.8110156Z ----------------------------------------------------------------------
2022-05-18T05:24:54.8130344Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:24:54.8131293Z 
2022-05-18T05:24:54.8131724Z ----------------------------------------------------------------------
2022-05-18T05:24:54.8132078Z Ran 1 test in 0.002s
2022-05-18T05:24:54.8132243Z 
2022-05-18T05:24:54.8132332Z OK (skipped=1)
2022-05-18T05:24:54.8132490Z 
2022-05-18T05:24:54.8132616Z Generating XML reports...
2022-05-18T05:24:54.8174145Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052454.xml
2022-05-18T05:24:56.0757228Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:56.0772249Z 
2022-05-18T05:24:56.0772682Z Running tests...
2022-05-18T05:24:56.0773163Z ----------------------------------------------------------------------
2022-05-18T05:24:56.0795067Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:24:56.0795450Z 
2022-05-18T05:24:56.0795968Z ----------------------------------------------------------------------
2022-05-18T05:24:56.0796329Z Ran 1 test in 0.002s
2022-05-18T05:24:56.0796496Z 
2022-05-18T05:24:56.0796586Z OK (skipped=1)
2022-05-18T05:24:56.0796744Z 
2022-05-18T05:24:56.0796870Z Generating XML reports...
2022-05-18T05:24:56.0839327Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052456.xml
2022-05-18T05:24:57.3645833Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:57.3661753Z 
2022-05-18T05:24:57.3662320Z Running tests...
2022-05-18T05:24:57.3662804Z ----------------------------------------------------------------------
2022-05-18T05:24:57.3682631Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:24:57.3683519Z 
2022-05-18T05:24:57.3683841Z ----------------------------------------------------------------------
2022-05-18T05:24:57.3684176Z Ran 1 test in 0.002s
2022-05-18T05:24:57.3684322Z 
2022-05-18T05:24:57.3684440Z OK (skipped=1)
2022-05-18T05:24:57.3684596Z 
2022-05-18T05:24:57.3684730Z Generating XML reports...
2022-05-18T05:24:57.3726606Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052457.xml
2022-05-18T05:24:58.6323951Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:58.6339644Z 
2022-05-18T05:24:58.6340140Z Running tests...
2022-05-18T05:24:58.6340649Z ----------------------------------------------------------------------
2022-05-18T05:24:58.6362026Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:24:58.6362654Z 
2022-05-18T05:24:58.6362973Z ----------------------------------------------------------------------
2022-05-18T05:24:58.6363289Z Ran 1 test in 0.002s
2022-05-18T05:24:58.6363457Z 
2022-05-18T05:24:58.6363568Z OK (skipped=1)
2022-05-18T05:24:58.6364006Z 
2022-05-18T05:24:58.6364133Z Generating XML reports...
2022-05-18T05:24:58.6405310Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052458.xml
2022-05-18T05:24:59.9084883Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:24:59.9100399Z 
2022-05-18T05:24:59.9100796Z Running tests...
2022-05-18T05:24:59.9101315Z ----------------------------------------------------------------------
2022-05-18T05:25:01.5742309Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:01.6115084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78379
2022-05-18T05:25:01.6225869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78380
2022-05-18T05:25:02.7931659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:02.7932262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:02.7933086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:02.7933792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:02.8040628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:02.8948340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:05.0691130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:25:05.0691720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:25:05.0692516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:25:05.0693232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:25:05.5321776Z ok (5.622s)
2022-05-18T05:25:05.5322046Z 
2022-05-18T05:25:05.5322423Z ----------------------------------------------------------------------
2022-05-18T05:25:05.5322765Z Ran 1 test in 5.622s
2022-05-18T05:25:05.5322936Z 
2022-05-18T05:25:05.5323029Z OK
2022-05-18T05:25:05.5323145Z 
2022-05-18T05:25:05.5323282Z Generating XML reports...
2022-05-18T05:25:05.5381475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052459.xml
2022-05-18T05:25:07.0038344Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:25:07.0053322Z 
2022-05-18T05:25:07.0053622Z Running tests...
2022-05-18T05:25:07.0054063Z ----------------------------------------------------------------------
2022-05-18T05:25:08.6766740Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:08.7147915Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78508
2022-05-18T05:25:08.7262346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78509
2022-05-18T05:25:09.8952477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:09.8953037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:09.8953855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:09.8954582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:09.9061234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:09.9967668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:10.1310825Z skip: Need at least 3 CUDA devices (3.125s)
2022-05-18T05:25:10.1311065Z 
2022-05-18T05:25:10.1311454Z ----------------------------------------------------------------------
2022-05-18T05:25:10.1311771Z Ran 1 test in 3.126s
2022-05-18T05:25:10.1311940Z 
2022-05-18T05:25:10.1312055Z OK (skipped=1)
2022-05-18T05:25:10.1312213Z 
2022-05-18T05:25:10.1312342Z Generating XML reports...
2022-05-18T05:25:10.1370585Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052507.xml
2022-05-18T05:25:11.5751655Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:25:11.5766997Z 
2022-05-18T05:25:11.5767227Z Running tests...
2022-05-18T05:25:11.5767667Z ----------------------------------------------------------------------
2022-05-18T05:25:11.5788694Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-05-18T05:25:11.5789007Z 
2022-05-18T05:25:11.5789288Z ----------------------------------------------------------------------
2022-05-18T05:25:11.5789601Z Ran 1 test in 0.002s
2022-05-18T05:25:11.5789771Z 
2022-05-18T05:25:11.5789885Z OK (skipped=1)
2022-05-18T05:25:11.5790044Z 
2022-05-18T05:25:11.5790170Z Generating XML reports...
2022-05-18T05:25:11.5833457Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052511.xml
2022-05-18T05:25:12.8639524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:25:12.8654812Z 
2022-05-18T05:25:12.8655182Z Running tests...
2022-05-18T05:25:12.8655723Z ----------------------------------------------------------------------
2022-05-18T05:25:14.5213447Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:14.5592733Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78658
2022-05-18T05:25:14.5704974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78659
2022-05-18T05:25:15.7322855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:15.7323446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:15.7324241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:15.7324951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:15.7332015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:15.7332873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:16.7767506Z ok (3.911s)
2022-05-18T05:25:16.7767751Z 
2022-05-18T05:25:16.7768129Z ----------------------------------------------------------------------
2022-05-18T05:25:16.7768452Z Ran 1 test in 3.911s
2022-05-18T05:25:16.7768623Z 
2022-05-18T05:25:16.7768717Z OK
2022-05-18T05:25:16.7768854Z 
2022-05-18T05:25:16.7768990Z Generating XML reports...
2022-05-18T05:25:16.7825931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052512.xml
2022-05-18T05:25:18.2044005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:25:18.2059141Z 
2022-05-18T05:25:18.2059631Z Running tests...
2022-05-18T05:25:18.2060112Z ----------------------------------------------------------------------
2022-05-18T05:25:19.8264564Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:19.8640079Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78773
2022-05-18T05:25:19.8754445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78774
2022-05-18T05:25:21.0890945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:21.0891528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:21.0892329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:21.0893038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:21.0899599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:21.0900109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:23.4839550Z ok (5.278s)
2022-05-18T05:25:23.4839780Z 
2022-05-18T05:25:23.4840171Z ----------------------------------------------------------------------
2022-05-18T05:25:23.4840516Z Ran 1 test in 5.278s
2022-05-18T05:25:23.4840683Z 
2022-05-18T05:25:23.4840778Z OK
2022-05-18T05:25:23.4840917Z 
2022-05-18T05:25:23.4841052Z Generating XML reports...
2022-05-18T05:25:23.4896920Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052518.xml
2022-05-18T05:25:24.9019595Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:25:24.9033673Z 
2022-05-18T05:25:24.9033920Z Running tests...
2022-05-18T05:25:24.9034353Z ----------------------------------------------------------------------
2022-05-18T05:25:26.5186615Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:26.5559727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78890
2022-05-18T05:25:26.5674223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78891
2022-05-18T05:25:27.7630149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:27.7630724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:27.7631506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:27.7632217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:27.7639117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:27.7639854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:27.7746523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:25:27.7747049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:25:27.7747759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:25:27.7748458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:25:28.8735281Z ok (3.970s)
2022-05-18T05:25:28.8735518Z 
2022-05-18T05:25:28.8735909Z ----------------------------------------------------------------------
2022-05-18T05:25:28.8736255Z Ran 1 test in 3.970s
2022-05-18T05:25:28.8736406Z 
2022-05-18T05:25:28.8736507Z OK
2022-05-18T05:25:28.8736644Z 
2022-05-18T05:25:28.8736779Z Generating XML reports...
2022-05-18T05:25:28.8793426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052524.xml
2022-05-18T05:25:30.3160636Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:25:30.3175744Z 
2022-05-18T05:25:30.3175984Z Running tests...
2022-05-18T05:25:30.3176430Z ----------------------------------------------------------------------
2022-05-18T05:25:31.9605079Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:31.9978958Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79011
2022-05-18T05:25:32.0089486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79012
2022-05-18T05:25:33.1733446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:33.1734026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:33.1734833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:33.1735535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:33.1845025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:33.2749353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:33.4137797Z skip: Skipped due to small world size. (3.096s)
2022-05-18T05:25:33.4138063Z 
2022-05-18T05:25:33.4138446Z ----------------------------------------------------------------------
2022-05-18T05:25:33.4138765Z Ran 1 test in 3.096s
2022-05-18T05:25:33.4138940Z 
2022-05-18T05:25:33.4139051Z OK (skipped=1)
2022-05-18T05:25:33.4139206Z 
2022-05-18T05:25:33.4139332Z Generating XML reports...
2022-05-18T05:25:33.4195688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052530.xml
2022-05-18T05:25:34.8368789Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:25:34.8384340Z 
2022-05-18T05:25:34.8384768Z Running tests...
2022-05-18T05:25:34.8385276Z ----------------------------------------------------------------------
2022-05-18T05:25:36.4851850Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:36.5236567Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79126
2022-05-18T05:25:36.5351534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79127
2022-05-18T05:25:37.7079882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:37.7080444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:37.7081450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:37.7082183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:37.7088744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:37.7089786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:37.9399894Z skip: Skipped due to small world size. (3.101s)
2022-05-18T05:25:37.9400174Z 
2022-05-18T05:25:37.9400539Z ----------------------------------------------------------------------
2022-05-18T05:25:37.9400875Z Ran 1 test in 3.101s
2022-05-18T05:25:37.9401039Z 
2022-05-18T05:25:37.9401157Z OK (skipped=1)
2022-05-18T05:25:37.9401293Z 
2022-05-18T05:25:37.9401418Z Generating XML reports...
2022-05-18T05:25:37.9458782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052534.xml
2022-05-18T05:25:39.3348099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:25:39.3363802Z 
2022-05-18T05:25:39.3364319Z Running tests...
2022-05-18T05:25:39.3364755Z ----------------------------------------------------------------------
2022-05-18T05:25:41.0021898Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:41.0394825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79241
2022-05-18T05:25:41.0508507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79242
2022-05-18T05:25:42.2248712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:42.2249270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:42.2250385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:42.2251098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:42.2257738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:42.2258250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:42.4555197Z skip: Skipped due to small world size. (3.119s)
2022-05-18T05:25:42.4555466Z 
2022-05-18T05:25:42.4555845Z ----------------------------------------------------------------------
2022-05-18T05:25:42.4556184Z Ran 1 test in 3.119s
2022-05-18T05:25:42.4556333Z 
2022-05-18T05:25:42.4556444Z OK (skipped=1)
2022-05-18T05:25:42.4557610Z 
2022-05-18T05:25:42.4558129Z Generating XML reports...
2022-05-18T05:25:42.4613767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052539.xml
2022-05-18T05:25:43.8868917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:25:43.8883865Z 
2022-05-18T05:25:43.8884372Z Running tests...
2022-05-18T05:25:43.8885257Z ----------------------------------------------------------------------
2022-05-18T05:25:45.5506385Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:45.5882893Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79356
2022-05-18T05:25:45.5995776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79357
2022-05-18T05:25:46.7581892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:46.7582487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:46.7583510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:46.7584259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:46.7591098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:46.7591573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:46.7698312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:25:46.7699041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:25:46.7699743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:25:46.7700420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:25:48.0060666Z ok (4.117s)
2022-05-18T05:25:48.0061077Z 
2022-05-18T05:25:48.0061834Z ----------------------------------------------------------------------
2022-05-18T05:25:48.0062481Z Ran 1 test in 4.118s
2022-05-18T05:25:48.0062948Z 
2022-05-18T05:25:48.0063049Z OK
2022-05-18T05:25:48.0063166Z 
2022-05-18T05:25:48.0063308Z Generating XML reports...
2022-05-18T05:25:48.0118895Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052543.xml
2022-05-18T05:25:49.4560494Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:25:49.4576211Z 
2022-05-18T05:25:49.4576497Z Running tests...
2022-05-18T05:25:49.4576917Z ----------------------------------------------------------------------
2022-05-18T05:25:51.1087193Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:51.1473015Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79477
2022-05-18T05:25:51.1587288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79478
2022-05-18T05:25:52.3768334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:52.3768895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:52.3769944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:52.3770672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:52.3877629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:52.4779747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:52.5190836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:52.5191377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:52.5192083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:52.5192774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:53.7657178Z ok (4.308s)
2022-05-18T05:25:53.7657414Z 
2022-05-18T05:25:53.7658007Z ----------------------------------------------------------------------
2022-05-18T05:25:53.7658366Z Ran 1 test in 4.308s
2022-05-18T05:25:53.7658534Z 
2022-05-18T05:25:53.7658629Z OK
2022-05-18T05:25:53.7658768Z 
2022-05-18T05:25:53.7658886Z Generating XML reports...
2022-05-18T05:25:53.7715924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052549.xml
2022-05-18T05:25:55.1862405Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:25:55.1876635Z 
2022-05-18T05:25:55.1877225Z Running tests...
2022-05-18T05:25:55.1877829Z ----------------------------------------------------------------------
2022-05-18T05:25:56.8165463Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:56.8542158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79598
2022-05-18T05:25:56.8652026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79599
2022-05-18T05:25:58.0335000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:58.0335562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:58.0336358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:58.0337084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:58.0443709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:58.1350633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:58.2700121Z skip: Skipped due to small world size. (3.082s)
2022-05-18T05:25:58.2700558Z 
2022-05-18T05:25:58.2700909Z ----------------------------------------------------------------------
2022-05-18T05:25:58.2701252Z Ran 1 test in 3.082s
2022-05-18T05:25:58.2701415Z 
2022-05-18T05:25:58.2701528Z OK (skipped=1)
2022-05-18T05:25:58.2701686Z 
2022-05-18T05:25:58.2701795Z Generating XML reports...
2022-05-18T05:25:58.2758670Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052555.xml
2022-05-18T05:25:59.6967829Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:25:59.6983784Z 
2022-05-18T05:25:59.6984180Z Running tests...
2022-05-18T05:25:59.6984602Z ----------------------------------------------------------------------
2022-05-18T05:26:01.3477129Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:26:01.3866696Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79713
2022-05-18T05:26:01.3984269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79714
2022-05-18T05:26:02.6242047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:02.6242653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:02.6243445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:02.6244314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:02.6251355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:02.6253165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:02.8033431Z ok (3.105s)
2022-05-18T05:26:02.8033644Z 
2022-05-18T05:26:02.8034032Z ----------------------------------------------------------------------
2022-05-18T05:26:02.8034357Z Ran 1 test in 3.105s
2022-05-18T05:26:02.8034528Z 
2022-05-18T05:26:02.8034632Z OK
2022-05-18T05:26:02.8034775Z 
2022-05-18T05:26:02.8034915Z Generating XML reports...
2022-05-18T05:26:02.8092216Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052559.xml
2022-05-18T05:26:04.2629375Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:04.2645192Z 
2022-05-18T05:26:04.2645576Z Running tests...
2022-05-18T05:26:04.2646085Z ----------------------------------------------------------------------
2022-05-18T05:26:05.9540942Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:26:05.9944430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79828
2022-05-18T05:26:06.0065880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79829
2022-05-18T05:26:07.2427655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:07.2428652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:07.2430097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:07.2431520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:07.2437327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:07.2438216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:07.4113977Z ok (3.147s)
2022-05-18T05:26:07.4114196Z 
2022-05-18T05:26:07.4114811Z ----------------------------------------------------------------------
2022-05-18T05:26:07.4115146Z Ran 1 test in 3.147s
2022-05-18T05:26:07.4115311Z 
2022-05-18T05:26:07.4115414Z OK
2022-05-18T05:26:07.4115550Z 
2022-05-18T05:26:07.4115685Z Generating XML reports...
2022-05-18T05:26:07.4172730Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052604.xml
2022-05-18T05:26:08.8616642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:08.8632416Z 
2022-05-18T05:26:08.8632571Z Running tests...
2022-05-18T05:26:08.8633029Z ----------------------------------------------------------------------
2022-05-18T05:26:08.8661997Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T05:26:08.8662501Z 
2022-05-18T05:26:08.8662802Z ----------------------------------------------------------------------
2022-05-18T05:26:08.8663147Z Ran 1 test in 0.003s
2022-05-18T05:26:08.8663311Z 
2022-05-18T05:26:08.8663420Z OK (skipped=1)
2022-05-18T05:26:08.8663581Z 
2022-05-18T05:26:08.8663691Z Generating XML reports...
2022-05-18T05:26:08.8707933Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052608.xml
2022-05-18T05:26:10.1554719Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:10.1571318Z 
2022-05-18T05:26:10.1571680Z Running tests...
2022-05-18T05:26:10.1572211Z ----------------------------------------------------------------------
2022-05-18T05:26:10.1602165Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T05:26:10.1602583Z 
2022-05-18T05:26:10.1603252Z ----------------------------------------------------------------------
2022-05-18T05:26:10.1603870Z Ran 1 test in 0.003s
2022-05-18T05:26:10.1604238Z 
2022-05-18T05:26:10.1604465Z OK (skipped=1)
2022-05-18T05:26:10.1604785Z 
2022-05-18T05:26:10.1604946Z Generating XML reports...
2022-05-18T05:26:10.1649493Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052610.xml
2022-05-18T05:26:11.4462656Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:11.4478310Z 
2022-05-18T05:26:11.4478772Z Running tests...
2022-05-18T05:26:11.4479729Z ----------------------------------------------------------------------
2022-05-18T05:26:11.4511569Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T05:26:11.4512309Z 
2022-05-18T05:26:11.4512923Z ----------------------------------------------------------------------
2022-05-18T05:26:11.4513554Z Ran 1 test in 0.003s
2022-05-18T05:26:11.4513719Z 
2022-05-18T05:26:11.4513809Z OK (skipped=1)
2022-05-18T05:26:11.4513967Z 
2022-05-18T05:26:11.4514091Z Generating XML reports...
2022-05-18T05:26:11.4557740Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052611.xml
2022-05-18T05:26:12.7461750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:12.7478407Z 
2022-05-18T05:26:12.7478867Z Running tests...
2022-05-18T05:26:12.7479348Z ----------------------------------------------------------------------
2022-05-18T05:26:12.7504601Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T05:26:12.7504939Z 
2022-05-18T05:26:12.7505236Z ----------------------------------------------------------------------
2022-05-18T05:26:12.7505565Z Ran 1 test in 0.003s
2022-05-18T05:26:12.7505729Z 
2022-05-18T05:26:12.7506054Z OK (skipped=1)
2022-05-18T05:26:12.7506237Z 
2022-05-18T05:26:12.7506367Z Generating XML reports...
2022-05-18T05:26:12.7550593Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052612.xml
2022-05-18T05:26:14.0241132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:14.0256793Z 
2022-05-18T05:26:14.0257107Z Running tests...
2022-05-18T05:26:14.0257562Z ----------------------------------------------------------------------
2022-05-18T05:26:14.0278720Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T05:26:14.0279247Z 
2022-05-18T05:26:14.0279717Z ----------------------------------------------------------------------
2022-05-18T05:26:14.0280078Z Ran 1 test in 0.002s
2022-05-18T05:26:14.0280257Z 
2022-05-18T05:26:14.0280376Z OK (skipped=1)
2022-05-18T05:26:14.0280535Z 
2022-05-18T05:26:14.0280673Z Generating XML reports...
2022-05-18T05:26:14.0322312Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052614.xml
2022-05-18T05:26:15.2943831Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:15.2960171Z 
2022-05-18T05:26:15.2960325Z Running tests...
2022-05-18T05:26:15.2961067Z ----------------------------------------------------------------------
2022-05-18T05:26:15.2990251Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T05:26:15.2990614Z 
2022-05-18T05:26:15.2991157Z ----------------------------------------------------------------------
2022-05-18T05:26:15.2991501Z Ran 1 test in 0.003s
2022-05-18T05:26:15.2991687Z 
2022-05-18T05:26:15.2991805Z OK (skipped=1)
2022-05-18T05:26:15.2991946Z 
2022-05-18T05:26:15.2992168Z Generating XML reports...
2022-05-18T05:26:15.3036053Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052615.xml
2022-05-18T05:26:16.5904056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:16.5926248Z 
2022-05-18T05:26:16.5926461Z Running tests...
2022-05-18T05:26:16.5926942Z ----------------------------------------------------------------------
2022-05-18T05:26:16.5956265Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T05:26:16.5956592Z 
2022-05-18T05:26:16.5956923Z ----------------------------------------------------------------------
2022-05-18T05:26:16.5957257Z Ran 1 test in 0.003s
2022-05-18T05:26:16.5957406Z 
2022-05-18T05:26:16.5957514Z OK (skipped=1)
2022-05-18T05:26:16.5957893Z 
2022-05-18T05:26:16.5958040Z Generating XML reports...
2022-05-18T05:26:16.6002110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052616.xml
2022-05-18T05:26:17.8962843Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:17.8979747Z 
2022-05-18T05:26:17.8979905Z Running tests...
2022-05-18T05:26:17.8980342Z ----------------------------------------------------------------------
2022-05-18T05:26:17.9003917Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T05:26:17.9004540Z 
2022-05-18T05:26:17.9004973Z ----------------------------------------------------------------------
2022-05-18T05:26:17.9005345Z Ran 1 test in 0.003s
2022-05-18T05:26:17.9005519Z 
2022-05-18T05:26:17.9005614Z OK (skipped=1)
2022-05-18T05:26:17.9005773Z 
2022-05-18T05:26:17.9005901Z Generating XML reports...
2022-05-18T05:26:17.9050896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052617.xml
2022-05-18T05:26:19.1916167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:19.1932942Z 
2022-05-18T05:26:19.1933358Z Running tests...
2022-05-18T05:26:19.1933855Z ----------------------------------------------------------------------
2022-05-18T05:26:20.8873042Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:26:20.9294971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80223
2022-05-18T05:26:20.9426551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80224
2022-05-18T05:26:22.1612907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:22.1613495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:22.1614302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:22.1614987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:22.1623201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:22.1623697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:22.4476220Z ok (3.254s)
2022-05-18T05:26:22.4476444Z 
2022-05-18T05:26:22.4476898Z ----------------------------------------------------------------------
2022-05-18T05:26:22.4477222Z Ran 1 test in 3.254s
2022-05-18T05:26:22.4477395Z 
2022-05-18T05:26:22.4477487Z OK
2022-05-18T05:26:22.4477623Z 
2022-05-18T05:26:22.4477757Z Generating XML reports...
2022-05-18T05:26:22.4535517Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052619.xml
2022-05-18T05:26:23.8625531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:23.8640390Z 
2022-05-18T05:26:23.8640866Z Running tests...
2022-05-18T05:26:23.8641361Z ----------------------------------------------------------------------
2022-05-18T05:26:25.4996415Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:26:25.5388066Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80342
2022-05-18T05:26:25.5506286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80343
2022-05-18T05:26:26.7158087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:26.7158642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:26.7159699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:26.7160413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:26.7269549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:26.8173275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:28.8637437Z ok (4.994s)
2022-05-18T05:26:28.8637740Z 
2022-05-18T05:26:28.8638134Z ----------------------------------------------------------------------
2022-05-18T05:26:28.8638469Z Ran 1 test in 4.995s
2022-05-18T05:26:28.8638615Z 
2022-05-18T05:26:28.8638707Z OK
2022-05-18T05:26:28.8638841Z 
2022-05-18T05:26:28.8638971Z Generating XML reports...
2022-05-18T05:26:28.8646135Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052623.xml
2022-05-18T05:26:30.2901531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:30.2916413Z 
2022-05-18T05:26:30.2916765Z Running tests...
2022-05-18T05:26:30.2917502Z ----------------------------------------------------------------------
2022-05-18T05:26:31.9105233Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:26:31.9498432Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80463
2022-05-18T05:26:31.9622490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80464
2022-05-18T05:26:33.1272130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:33.1272680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:33.1273517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:33.1274209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:33.1382650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:33.2285465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:33.2397766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:26:33.2398293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:26:33.2399026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:26:33.2399712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:26:33.4674631Z ok (3.175s)
2022-05-18T05:26:33.4674878Z 
2022-05-18T05:26:33.4675280Z ----------------------------------------------------------------------
2022-05-18T05:26:33.4675627Z Ran 1 test in 3.176s
2022-05-18T05:26:33.4675781Z 
2022-05-18T05:26:33.4675887Z OK
2022-05-18T05:26:33.4676025Z 
2022-05-18T05:26:33.4676165Z Generating XML reports...
2022-05-18T05:26:33.4732654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052630.xml
2022-05-18T05:26:34.9020925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:34.9036720Z 
2022-05-18T05:26:34.9036992Z Running tests...
2022-05-18T05:26:34.9037433Z ----------------------------------------------------------------------
2022-05-18T05:26:36.5621382Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:26:36.6023533Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80588
2022-05-18T05:26:36.6143536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80589
2022-05-18T05:26:37.8280705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:37.8281305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:37.8282084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:37.8282791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:37.8289611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:37.8291290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:38.0192165Z skip: Skipped due to small world size. (3.115s)
2022-05-18T05:26:38.0192666Z 
2022-05-18T05:26:38.0193331Z ----------------------------------------------------------------------
2022-05-18T05:26:38.0193998Z Ran 1 test in 3.115s
2022-05-18T05:26:38.0194303Z 
2022-05-18T05:26:38.0194492Z OK (skipped=1)
2022-05-18T05:26:38.0195182Z 
2022-05-18T05:26:38.0195416Z Generating XML reports...
2022-05-18T05:26:38.0253810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052634.xml
2022-05-18T05:26:39.4594276Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:39.4609185Z 
2022-05-18T05:26:39.4609506Z Running tests...
2022-05-18T05:26:39.4610230Z ----------------------------------------------------------------------
2022-05-18T05:26:41.1055435Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:26:41.1457257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80703
2022-05-18T05:26:41.1578786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80704
2022-05-18T05:26:42.3624542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:42.3625146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:42.3625945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:42.3626650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:42.3633328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:42.3634592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:43.9651424Z ok (4.504s)
2022-05-18T05:26:43.9651679Z 
2022-05-18T05:26:43.9652081Z ----------------------------------------------------------------------
2022-05-18T05:26:43.9652416Z Ran 1 test in 4.504s
2022-05-18T05:26:43.9652587Z 
2022-05-18T05:26:43.9652679Z OK
2022-05-18T05:26:43.9652815Z 
2022-05-18T05:26:43.9652935Z Generating XML reports...
2022-05-18T05:26:43.9709270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052639.xml
2022-05-18T05:26:45.4224313Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:45.4239365Z 
2022-05-18T05:26:45.4239615Z Running tests...
2022-05-18T05:26:45.4240045Z ----------------------------------------------------------------------
2022-05-18T05:26:47.0975377Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:26:47.1377902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80820
2022-05-18T05:26:47.1496681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80821
2022-05-18T05:26:48.3478177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:48.3478770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:48.3479555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:48.3480257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:48.3486610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:48.3487796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:48.5545816Z ok (3.130s)
2022-05-18T05:26:48.5546059Z 
2022-05-18T05:26:48.5546439Z ----------------------------------------------------------------------
2022-05-18T05:26:48.5546780Z Ran 1 test in 3.131s
2022-05-18T05:26:48.5546963Z 
2022-05-18T05:26:48.5547063Z OK
2022-05-18T05:26:48.5547181Z 
2022-05-18T05:26:48.5547311Z Generating XML reports...
2022-05-18T05:26:48.5604247Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052645.xml
2022-05-18T05:26:49.9921409Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:49.9936909Z 
2022-05-18T05:26:49.9937049Z Running tests...
2022-05-18T05:26:49.9937721Z ----------------------------------------------------------------------
2022-05-18T05:26:51.6631883Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:26:51.7037259Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80935
2022-05-18T05:26:51.7160406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80936
2022-05-18T05:26:52.9069697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:52.9070268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:52.9071091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:52.9071774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:52.9079844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:52.9080338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:52.9189151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:26:52.9189656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:26:52.9190372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:26:52.9191073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:26:52.9399546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:26:52.9400046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:26:52.9400717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:26:52.9401408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:26:54.2565830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8rukac1
2022-05-18T05:26:54.2566931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8rukac1/_remote_module_non_scriptable.py
2022-05-18T05:26:54.3060136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1j4shvhm
2022-05-18T05:26:54.3062370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1j4shvhm/_remote_module_non_scriptable.py
2022-05-18T05:26:54.6234173Z ok (4.629s)
2022-05-18T05:26:54.6234382Z 
2022-05-18T05:26:54.6235018Z ----------------------------------------------------------------------
2022-05-18T05:26:54.6235395Z Ran 1 test in 4.630s
2022-05-18T05:26:54.6235564Z 
2022-05-18T05:26:54.6235640Z OK
2022-05-18T05:26:54.6235775Z 
2022-05-18T05:26:54.6235905Z Generating XML reports...
2022-05-18T05:26:54.6293116Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052649.xml
2022-05-18T05:26:56.0817541Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:26:56.0832861Z 
2022-05-18T05:26:56.0833185Z Running tests...
2022-05-18T05:26:56.0833620Z ----------------------------------------------------------------------
2022-05-18T05:26:57.7386992Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:26:57.7789574Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81064
2022-05-18T05:26:57.7911166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81065
2022-05-18T05:26:58.9870594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:58.9871136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:58.9871926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:58.9872615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:58.9982577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:59.0883818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:59.1091858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:26:59.1092395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:26:59.1093127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:26:59.1093826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:26:59.1301200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:26:59.1301733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:26:59.1302422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:26:59.1303116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:27:00.4617763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4gd4xmys
2022-05-18T05:27:00.4618456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4gd4xmys/_remote_module_non_scriptable.py
2022-05-18T05:27:00.4647725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaiv3v9y6
2022-05-18T05:27:00.4650535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaiv3v9y6/_remote_module_non_scriptable.py
2022-05-18T05:27:00.7988696Z ok (4.715s)
2022-05-18T05:27:00.7988925Z 
2022-05-18T05:27:00.7989877Z ----------------------------------------------------------------------
2022-05-18T05:27:00.7990263Z Ran 1 test in 4.716s
2022-05-18T05:27:00.7990430Z 
2022-05-18T05:27:00.7990523Z OK
2022-05-18T05:27:00.7991878Z 
2022-05-18T05:27:00.7992257Z Generating XML reports...
2022-05-18T05:27:00.8048845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052656.xml
2022-05-18T05:27:02.2289897Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:27:02.2304984Z 
2022-05-18T05:27:02.2305225Z Running tests...
2022-05-18T05:27:02.2305650Z ----------------------------------------------------------------------
2022-05-18T05:27:03.8501273Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:03.8892711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81193
2022-05-18T05:27:03.9015361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81194
2022-05-18T05:27:05.0947660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:05.0948225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:05.0949300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:05.0950014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:05.0956640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:05.0957169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:06.4338145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpun139dml
2022-05-18T05:27:06.4338972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpun139dml/_remote_module_non_scriptable.py
2022-05-18T05:27:06.4374468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3co32ld7
2022-05-18T05:27:06.4377398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3co32ld7/_remote_module_non_scriptable.py
2022-05-18T05:27:06.7411289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:06.7411829Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:07.0093397Z ok (4.778s)
2022-05-18T05:27:07.0093616Z 
2022-05-18T05:27:07.0094239Z ----------------------------------------------------------------------
2022-05-18T05:27:07.0094585Z Ran 1 test in 4.779s
2022-05-18T05:27:07.0094733Z 
2022-05-18T05:27:07.0096993Z OK
2022-05-18T05:27:07.0097459Z 
2022-05-18T05:27:07.0097853Z Generating XML reports...
2022-05-18T05:27:07.0151634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052702.xml
2022-05-18T05:27:08.4400583Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:27:08.4416048Z 
2022-05-18T05:27:08.4416517Z Running tests...
2022-05-18T05:27:08.4417019Z ----------------------------------------------------------------------
2022-05-18T05:27:10.0695161Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:10.1089312Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81314
2022-05-18T05:27:10.1205907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81315
2022-05-18T05:27:11.3399388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:11.3399952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:11.3400986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:11.3401725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:11.3508519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:11.4415992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:12.6386175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ds0_exj
2022-05-18T05:27:12.6386786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ds0_exj/_remote_module_non_scriptable.py
2022-05-18T05:27:12.7367572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6p_k33fa
2022-05-18T05:27:12.7368720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6p_k33fa/_remote_module_non_scriptable.py
2022-05-18T05:27:13.0492827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:13.0493377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:13.0506203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:13.0506681Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:13.3286448Z ok (4.887s)
2022-05-18T05:27:13.3286685Z 
2022-05-18T05:27:13.3287088Z ----------------------------------------------------------------------
2022-05-18T05:27:13.3287429Z Ran 1 test in 4.887s
2022-05-18T05:27:13.3287576Z 
2022-05-18T05:27:13.3287673Z OK
2022-05-18T05:27:13.3287808Z 
2022-05-18T05:27:13.3287947Z Generating XML reports...
2022-05-18T05:27:13.3345523Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052708.xml
2022-05-18T05:27:14.7520404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:27:14.7535306Z 
2022-05-18T05:27:14.7535607Z Running tests...
2022-05-18T05:27:14.7536027Z ----------------------------------------------------------------------
2022-05-18T05:27:16.3863207Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:16.4260230Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81435
2022-05-18T05:27:16.4378128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81436
2022-05-18T05:27:17.6599051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:17.6599627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:17.6600419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:17.6601117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:17.6607808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:17.6608312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:18.9998710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi4u9k3nr
2022-05-18T05:27:18.9999300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi4u9k3nr/_remote_module_non_scriptable.py
2022-05-18T05:27:19.0301229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwrix90j
2022-05-18T05:27:19.0302516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwrix90j/_remote_module_non_scriptable.py
2022-05-18T05:27:19.3439602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:19.3440482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:19.3456108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:19.3457141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:19.3619656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:19.3620593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:19.3633503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:19.3634493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:19.6459312Z ok (4.892s)
2022-05-18T05:27:19.6459516Z 
2022-05-18T05:27:19.6459895Z ----------------------------------------------------------------------
2022-05-18T05:27:19.6460236Z Ran 1 test in 4.892s
2022-05-18T05:27:19.6460420Z 
2022-05-18T05:27:19.6460512Z OK
2022-05-18T05:27:19.6460629Z 
2022-05-18T05:27:19.6460762Z Generating XML reports...
2022-05-18T05:27:19.6519082Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052714.xml
2022-05-18T05:27:21.0918483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:27:21.0933393Z 
2022-05-18T05:27:21.0933620Z Running tests...
2022-05-18T05:27:21.0934122Z ----------------------------------------------------------------------
2022-05-18T05:27:22.7104949Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:22.7221916Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.628s)
2022-05-18T05:27:22.7222593Z 
2022-05-18T05:27:22.7222897Z ----------------------------------------------------------------------
2022-05-18T05:27:22.7223240Z Ran 1 test in 1.629s
2022-05-18T05:27:22.7223405Z 
2022-05-18T05:27:22.7223494Z OK (skipped=1)
2022-05-18T05:27:22.7223648Z 
2022-05-18T05:27:22.7223773Z Generating XML reports...
2022-05-18T05:27:22.7260204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052721.xml
2022-05-18T05:27:24.0948454Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:27:24.0962622Z 
2022-05-18T05:27:24.0962896Z Running tests...
2022-05-18T05:27:24.0963611Z ----------------------------------------------------------------------
2022-05-18T05:27:25.7368777Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:25.7765857Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81592
2022-05-18T05:27:25.7883437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81593
2022-05-18T05:27:26.9842820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:26.9843389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:26.9844177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:26.9844867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:26.9953316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:27.0857385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:28.3132489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2nzt9kp
2022-05-18T05:27:28.3133133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2nzt9kp/_remote_module_non_scriptable.py
2022-05-18T05:27:28.3629857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoi5d9ur_
2022-05-18T05:27:28.3630780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoi5d9ur_/_remote_module_non_scriptable.py
2022-05-18T05:27:28.3839628Z <class 'torch.nn.parameter.Parameter'>
2022-05-18T05:27:28.6958363Z ok (4.599s)
2022-05-18T05:27:28.6958850Z 
2022-05-18T05:27:28.6959634Z ----------------------------------------------------------------------
2022-05-18T05:27:28.6960012Z Ran 1 test in 4.600s
2022-05-18T05:27:28.6960197Z 
2022-05-18T05:27:28.6960294Z OK
2022-05-18T05:27:28.6960432Z 
2022-05-18T05:27:28.6960570Z Generating XML reports...
2022-05-18T05:27:28.7018311Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052724.xml
2022-05-18T05:27:30.1234584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:27:30.1249046Z 
2022-05-18T05:27:30.1249485Z Running tests...
2022-05-18T05:27:30.1250185Z ----------------------------------------------------------------------
2022-05-18T05:27:31.7536774Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:31.7933617Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81709
2022-05-18T05:27:31.8049272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81710
2022-05-18T05:27:32.9982127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:32.9982701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:32.9983488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:32.9984191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:33.0091453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:33.0996898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:34.2892361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdkfxhcn
2022-05-18T05:27:34.2892972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdkfxhcn/_remote_module_non_scriptable.py
2022-05-18T05:27:34.3904880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4vcfsn1
2022-05-18T05:27:34.3905788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4vcfsn1/_remote_module_non_scriptable.py
2022-05-18T05:27:34.7123825Z ok (4.587s)
2022-05-18T05:27:34.7124027Z 
2022-05-18T05:27:34.7124396Z ----------------------------------------------------------------------
2022-05-18T05:27:34.7124741Z Ran 1 test in 4.587s
2022-05-18T05:27:34.7124924Z 
2022-05-18T05:27:34.7125018Z OK
2022-05-18T05:27:34.7125157Z 
2022-05-18T05:27:34.7125289Z Generating XML reports...
2022-05-18T05:27:34.7182181Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052730.xml
2022-05-18T05:27:36.1400215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:27:36.1415210Z 
2022-05-18T05:27:36.1415467Z Running tests...
2022-05-18T05:27:36.1415886Z ----------------------------------------------------------------------
2022-05-18T05:27:37.7556761Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:37.7952404Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81826
2022-05-18T05:27:37.8072099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81827
2022-05-18T05:27:39.0216467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:39.0217010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:39.0217801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:39.0218501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:39.0225482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:39.0226367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:40.3628742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprg0t6c7x
2022-05-18T05:27:40.3629621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprg0t6c7x/_remote_module_non_scriptable.py
2022-05-18T05:27:40.3672072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyd_nvynw
2022-05-18T05:27:40.3674869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyd_nvynw/_remote_module_non_scriptable.py
2022-05-18T05:27:40.6863872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:40.6864417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:41.0152962Z ok (4.873s)
2022-05-18T05:27:41.0153174Z 
2022-05-18T05:27:41.0153575Z ----------------------------------------------------------------------
2022-05-18T05:27:41.0153933Z Ran 1 test in 4.874s
2022-05-18T05:27:41.0154080Z 
2022-05-18T05:27:41.0154175Z OK
2022-05-18T05:27:41.0154352Z 
2022-05-18T05:27:41.0154490Z Generating XML reports...
2022-05-18T05:27:41.0211329Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052736.xml
2022-05-18T05:27:42.5040668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:27:42.5057410Z 
2022-05-18T05:27:42.5057715Z Running tests...
2022-05-18T05:27:42.5058146Z ----------------------------------------------------------------------
2022-05-18T05:27:44.1919145Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:44.2330222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81947
2022-05-18T05:27:44.2450829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81948
2022-05-18T05:27:45.3881042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:45.3881904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:45.3882813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:45.3883713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:45.3992064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:45.4895860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:46.6947442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoirksp3m
2022-05-18T05:27:46.6948104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoirksp3m/_remote_module_non_scriptable.py
2022-05-18T05:27:46.7686433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprxqjhcmz
2022-05-18T05:27:46.7687705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprxqjhcmz/_remote_module_non_scriptable.py
2022-05-18T05:27:47.0697538Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:27:47.4533221Z ok (4.947s)
2022-05-18T05:27:47.4533578Z 
2022-05-18T05:27:47.4534242Z ----------------------------------------------------------------------
2022-05-18T05:27:47.4534879Z Ran 1 test in 4.947s
2022-05-18T05:27:47.4535189Z 
2022-05-18T05:27:47.4535353Z OK
2022-05-18T05:27:47.4535608Z 
2022-05-18T05:27:47.4535817Z Generating XML reports...
2022-05-18T05:27:47.4594683Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052742.xml
2022-05-18T05:27:48.9031297Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:27:48.9046245Z 
2022-05-18T05:27:48.9046785Z Running tests...
2022-05-18T05:27:48.9047554Z ----------------------------------------------------------------------
2022-05-18T05:27:50.5579258Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:50.5984431Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82068
2022-05-18T05:27:50.6106455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82069
2022-05-18T05:27:51.7794604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:51.7795430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:51.7796464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:51.7797219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:51.7803727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:51.7804442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:53.1201473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7cikby8
2022-05-18T05:27:53.1202107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7cikby8/_remote_module_non_scriptable.py
2022-05-18T05:27:53.1300702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjasxapo9
2022-05-18T05:27:53.1303273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjasxapo9/_remote_module_non_scriptable.py
2022-05-18T05:27:53.4383477Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:27:53.4385415Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:27:53.8186591Z ok (4.914s)
2022-05-18T05:27:53.8186824Z 
2022-05-18T05:27:53.8187198Z ----------------------------------------------------------------------
2022-05-18T05:27:53.8187541Z Ran 1 test in 4.914s
2022-05-18T05:27:53.8187711Z 
2022-05-18T05:27:53.8187811Z OK
2022-05-18T05:27:53.8187950Z 
2022-05-18T05:27:53.8188086Z Generating XML reports...
2022-05-18T05:27:53.8244411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052748.xml
2022-05-18T05:27:55.2745857Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:27:55.2760981Z 
2022-05-18T05:27:55.2761233Z Running tests...
2022-05-18T05:27:55.2761674Z ----------------------------------------------------------------------
2022-05-18T05:27:56.9446978Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:56.9842187Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82189
2022-05-18T05:27:56.9961245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82190
2022-05-18T05:27:58.2114764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:58.2115319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:58.2116112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:58.2116819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:58.2123592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:58.2124092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:58.2211139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn90xm98t
2022-05-18T05:27:58.2214530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn90xm98t/_remote_module_non_scriptable.py
2022-05-18T05:27:58.2215081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddck6zun
2022-05-18T05:27:58.2218193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddck6zun/_remote_module_non_scriptable.py
2022-05-18T05:27:58.2365348Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:27:58.2366932Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:27:58.2371372Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T05:27:58.2372628Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-05-18T05:27:58.2374100Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T05:27:58.2375101Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-05-18T05:27:58.2375816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:58.2376311Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:27:58.2379551Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:27:58.2381235Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:27:58.2386265Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:27:58.2387841Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:27:58.2391857Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:27:58.2393387Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:27:58.2397799Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:27:58.2399310Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:27:58.2403612Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:27:58.2405305Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:27:58.4009223Z ok (3.124s)
2022-05-18T05:27:58.4009451Z 
2022-05-18T05:27:58.4010051Z ----------------------------------------------------------------------
2022-05-18T05:27:58.4010369Z Ran 1 test in 3.125s
2022-05-18T05:27:58.4010555Z 
2022-05-18T05:27:58.4010651Z OK
2022-05-18T05:27:58.4010786Z 
2022-05-18T05:27:58.4010918Z Generating XML reports...
2022-05-18T05:27:58.4067094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052755.xml
2022-05-18T05:27:59.7978148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:27:59.7995148Z 
2022-05-18T05:27:59.7995624Z Running tests...
2022-05-18T05:27:59.7996107Z ----------------------------------------------------------------------
2022-05-18T05:28:01.4611359Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:01.5017732Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82308
2022-05-18T05:28:01.5138507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82309
2022-05-18T05:28:02.6752792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:28:02.6754384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:02.6755465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:28:02.6756791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:02.6860914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:02.7771203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:03.9913689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4scp4ysj
2022-05-18T05:28:03.9914887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4scp4ysj/_remote_module_non_scriptable.py
2022-05-18T05:28:04.0803061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6budaqnw
2022-05-18T05:28:04.0804197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6budaqnw/_remote_module_non_scriptable.py
2022-05-18T05:28:04.3831787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:04.3832837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:04.7220979Z ok (4.922s)
2022-05-18T05:28:04.7221199Z 
2022-05-18T05:28:04.7221598Z ----------------------------------------------------------------------
2022-05-18T05:28:04.7221924Z Ran 1 test in 4.923s
2022-05-18T05:28:04.7222094Z 
2022-05-18T05:28:04.7222199Z OK
2022-05-18T05:28:04.7222336Z 
2022-05-18T05:28:04.7222472Z Generating XML reports...
2022-05-18T05:28:04.7278579Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052759.xml
2022-05-18T05:28:06.1732351Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:28:06.1747686Z 
2022-05-18T05:28:06.1747943Z Running tests...
2022-05-18T05:28:06.1748388Z ----------------------------------------------------------------------
2022-05-18T05:28:07.8305095Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:07.8708539Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82429
2022-05-18T05:28:07.8829080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82430
2022-05-18T05:28:09.0477608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:28:09.0478166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:28:09.0478947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:09.0479662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:09.0589574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:09.1492472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:10.3881310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoucicfx8
2022-05-18T05:28:10.3882276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoucicfx8/_remote_module_non_scriptable.py
2022-05-18T05:28:10.4245311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpupexiy5u
2022-05-18T05:28:10.4247610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpupexiy5u/_remote_module_non_scriptable.py
2022-05-18T05:28:10.4459652Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T05:28:10.4461461Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T05:28:10.4463596Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T05:28:10.4465296Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T05:28:11.0909097Z ok (4.916s)
2022-05-18T05:28:11.0909291Z 
2022-05-18T05:28:11.0909692Z ----------------------------------------------------------------------
2022-05-18T05:28:11.0910046Z Ran 1 test in 4.916s
2022-05-18T05:28:11.0910215Z 
2022-05-18T05:28:11.0910310Z OK
2022-05-18T05:28:11.0910456Z 
2022-05-18T05:28:11.0910595Z Generating XML reports...
2022-05-18T05:28:11.0966387Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052806.xml
2022-05-18T05:28:12.5467971Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:28:12.5483094Z 
2022-05-18T05:28:12.5483418Z Running tests...
2022-05-18T05:28:12.5483835Z ----------------------------------------------------------------------
2022-05-18T05:28:14.2090728Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:14.2495167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82580
2022-05-18T05:28:14.2615085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82581
2022-05-18T05:28:15.5044451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:28:15.5045014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:28:15.5045839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:15.5046550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:15.5053832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:15.5054600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:16.8357980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhq2gcl2
2022-05-18T05:28:16.8362349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhq2gcl2/_remote_module_non_scriptable.py
2022-05-18T05:28:16.8658150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgfk35bef
2022-05-18T05:28:16.8660478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgfk35bef/_remote_module_non_scriptable.py
2022-05-18T05:28:17.1695413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:17.1695980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:17.1924666Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpxhq2gcl2'>
2022-05-18T05:28:17.1925156Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:28:17.1925738Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpgfk35bef'>
2022-05-18T05:28:17.1926205Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:28:17.4696020Z ok (4.921s)
2022-05-18T05:28:17.4696236Z 
2022-05-18T05:28:17.4696641Z ----------------------------------------------------------------------
2022-05-18T05:28:17.4696961Z Ran 1 test in 4.921s
2022-05-18T05:28:17.4697136Z 
2022-05-18T05:28:17.4697234Z OK
2022-05-18T05:28:17.4697381Z 
2022-05-18T05:28:17.4697517Z Generating XML reports...
2022-05-18T05:28:17.4754611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052812.xml
2022-05-18T05:28:18.9197774Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:28:18.9213084Z 
2022-05-18T05:28:18.9213222Z Running tests...
2022-05-18T05:28:18.9213990Z ----------------------------------------------------------------------
2022-05-18T05:28:20.5681812Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:20.5802702Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.659s)
2022-05-18T05:28:20.5803296Z 
2022-05-18T05:28:20.5803593Z ----------------------------------------------------------------------
2022-05-18T05:28:20.5803927Z Ran 1 test in 1.659s
2022-05-18T05:28:20.5804092Z 
2022-05-18T05:28:20.5804183Z OK (skipped=1)
2022-05-18T05:28:20.5804342Z 
2022-05-18T05:28:20.5804525Z Generating XML reports...
2022-05-18T05:28:20.5843410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052818.xml
2022-05-18T05:28:21.9811208Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:28:21.9826535Z 
2022-05-18T05:28:21.9826966Z Running tests...
2022-05-18T05:28:21.9827471Z ----------------------------------------------------------------------
2022-05-18T05:28:23.6453018Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:23.6857266Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82737
2022-05-18T05:28:23.6977637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82738
2022-05-18T05:28:24.9071224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:28:24.9071801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:28:24.9072605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:24.9073313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:24.9180251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:25.0084605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:25.0292928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:28:25.0293980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:28:25.0294689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:28:25.0295374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:28:26.3423954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1qbcvntf
2022-05-18T05:28:26.3424580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1qbcvntf/_remote_module_non_scriptable.py
2022-05-18T05:28:26.4010315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_5914gw
2022-05-18T05:28:26.4011479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_5914gw/_remote_module_non_scriptable.py
2022-05-18T05:28:26.7107798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:26.7108389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:26.7124090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:26.7124580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:27.3064066Z ok (5.323s)
2022-05-18T05:28:27.3064427Z 
2022-05-18T05:28:27.3064914Z ----------------------------------------------------------------------
2022-05-18T05:28:27.3065286Z Ran 1 test in 5.324s
2022-05-18T05:28:27.3065465Z 
2022-05-18T05:28:27.3065563Z OK
2022-05-18T05:28:27.3065702Z 
2022-05-18T05:28:27.3065840Z Generating XML reports...
2022-05-18T05:28:27.3122517Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052821.xml
2022-05-18T05:28:28.7530772Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:28:28.7546140Z 
2022-05-18T05:28:28.7546458Z Running tests...
2022-05-18T05:28:28.7546919Z ----------------------------------------------------------------------
2022-05-18T05:28:30.4170197Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:30.4562561Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82864
2022-05-18T05:28:30.4681478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82865
2022-05-18T05:28:31.6700828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:28:31.6701371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:28:31.6702461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:31.6703198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:31.6709994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:31.6710469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:31.6712497Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T05:28:31.6713089Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T05:28:33.0114467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0s37fld
2022-05-18T05:28:33.0115142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0s37fld/_remote_module_non_scriptable.py
2022-05-18T05:28:33.0127633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3acg1k3i
2022-05-18T05:28:33.0130730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3acg1k3i/_remote_module_non_scriptable.py
2022-05-18T05:28:33.3215924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:33.3216483Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:33.3233579Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:33.3234240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:33.3488876Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T05:28:33.3489485Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T05:28:33.6166559Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T05:28:33.6167411Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T05:28:33.6250384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:33.6251208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:33.6269036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:33.6269515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:33.6522641Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T05:28:33.6523234Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T05:28:33.8163737Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T05:28:33.8164538Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T05:28:33.8245536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:33.8246047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:33.8264377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:33.8264875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:34.4776368Z ok (5.723s)
2022-05-18T05:28:34.4776796Z 
2022-05-18T05:28:34.4777465Z ----------------------------------------------------------------------
2022-05-18T05:28:34.4778093Z Ran 1 test in 5.723s
2022-05-18T05:28:34.4778737Z 
2022-05-18T05:28:34.4778956Z OK
2022-05-18T05:28:34.4779192Z 
2022-05-18T05:28:34.4780902Z Generating XML reports...
2022-05-18T05:28:34.4835431Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052828.xml
2022-05-18T05:28:35.9210415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:28:35.9225618Z 
2022-05-18T05:28:35.9226143Z Running tests...
2022-05-18T05:28:35.9226634Z ----------------------------------------------------------------------
2022-05-18T05:28:37.5944853Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:37.6350703Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82985
2022-05-18T05:28:37.6473258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82986
2022-05-18T05:28:38.8338372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:28:38.8338943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:28:38.8339999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:38.8340716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:38.8347739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:38.8348235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:38.8349981Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T05:28:38.8351123Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T05:28:40.1740291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3x5u79cr
2022-05-18T05:28:40.1740913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3x5u79cr/_remote_module_non_scriptable.py
2022-05-18T05:28:40.1769563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhor5edx
2022-05-18T05:28:40.1773693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhor5edx/_remote_module_non_scriptable.py
2022-05-18T05:28:40.4829888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:40.4830449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:40.4847374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:40.4848077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:40.4853775Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T05:28:40.4854359Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T05:28:40.4886854Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T05:28:40.4887506Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T05:28:40.4888621Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T05:28:40.4889319Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T05:28:40.4890273Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T05:28:40.4890940Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T05:28:40.8462669Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T05:28:40.8464771Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T05:28:40.8546895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:40.8547682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:40.8564632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:40.8565454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:40.8569846Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T05:28:40.8570665Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T05:28:40.8602717Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T05:28:40.8603382Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T05:28:40.8604215Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T05:28:40.8604892Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T05:28:41.5565847Z ok (5.634s)
2022-05-18T05:28:41.5566072Z 
2022-05-18T05:28:41.5566466Z ----------------------------------------------------------------------
2022-05-18T05:28:41.5566791Z Ran 1 test in 5.634s
2022-05-18T05:28:41.5566993Z 
2022-05-18T05:28:41.5567095Z OK
2022-05-18T05:28:41.5567232Z 
2022-05-18T05:28:41.5567371Z Generating XML reports...
2022-05-18T05:28:41.5626719Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052835.xml
2022-05-18T05:28:43.0102431Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:28:43.0117935Z 
2022-05-18T05:28:43.0118361Z Running tests...
2022-05-18T05:28:43.0118877Z ----------------------------------------------------------------------
2022-05-18T05:28:44.6523415Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:44.6922522Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83106
2022-05-18T05:28:44.7041041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83107
2022-05-18T05:28:45.9213251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:28:45.9213842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:28:45.9214629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:45.9215334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:45.9221414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:45.9222461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:47.2752857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjz8uzas9
2022-05-18T05:28:47.2753513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjz8uzas9/_remote_module_non_scriptable.py
2022-05-18T05:28:47.2977396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxajupqe6
2022-05-18T05:28:47.2978701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxajupqe6/_remote_module_non_scriptable.py
2022-05-18T05:28:47.6556490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:47.6557044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:47.7262238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:47.7263217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:48.1127040Z ok (5.101s)
2022-05-18T05:28:48.1127233Z 
2022-05-18T05:28:48.1127593Z ----------------------------------------------------------------------
2022-05-18T05:28:48.1127936Z Ran 1 test in 5.101s
2022-05-18T05:28:48.1128104Z 
2022-05-18T05:28:48.1128201Z OK
2022-05-18T05:28:48.1128352Z 
2022-05-18T05:28:48.1128487Z Generating XML reports...
2022-05-18T05:28:48.1185461Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052843.xml
2022-05-18T05:28:49.5332582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:28:49.5347696Z 
2022-05-18T05:28:49.5348115Z Running tests...
2022-05-18T05:28:49.5348611Z ----------------------------------------------------------------------
2022-05-18T05:28:51.1945052Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:51.2348921Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83257
2022-05-18T05:28:51.2468473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83258
2022-05-18T05:28:52.4543872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:28:52.4544445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:28:52.4545229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:52.4545929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:52.4552909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:52.4553407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:53.7852666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9um24b4
2022-05-18T05:28:53.7853573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9um24b4/_remote_module_non_scriptable.py
2022-05-18T05:28:53.8035263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9dw4yiam
2022-05-18T05:28:53.8036759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9dw4yiam/_remote_module_non_scriptable.py
2022-05-18T05:28:54.1554456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:54.1555026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:54.2311689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:54.2312259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:54.6551197Z ok (5.120s)
2022-05-18T05:28:54.6551383Z 
2022-05-18T05:28:54.6551766Z ----------------------------------------------------------------------
2022-05-18T05:28:54.6552099Z Ran 1 test in 5.120s
2022-05-18T05:28:54.6552310Z 
2022-05-18T05:28:54.6552406Z OK
2022-05-18T05:28:54.6552545Z 
2022-05-18T05:28:54.6552674Z Generating XML reports...
2022-05-18T05:28:54.6610767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052849.xml
2022-05-18T05:28:56.0814770Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:28:56.0829170Z 
2022-05-18T05:28:56.0829685Z Running tests...
2022-05-18T05:28:56.0830320Z ----------------------------------------------------------------------
2022-05-18T05:28:57.7165063Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:57.7561265Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83408
2022-05-18T05:28:57.7682527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83409
2022-05-18T05:28:58.9899248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:28:58.9900303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:28:58.9901685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:58.9903057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:58.9909764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:58.9910576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:29:00.3360893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobs6fskx
2022-05-18T05:29:00.3362030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobs6fskx/_remote_module_non_scriptable.py
2022-05-18T05:29:00.3813126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2jicmgy
2022-05-18T05:29:00.3814507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2jicmgy/_remote_module_non_scriptable.py
2022-05-18T05:29:00.7342892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:00.7343458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:00.8020290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:00.8020844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:01.1766281Z ok (5.093s)
2022-05-18T05:29:01.1766498Z 
2022-05-18T05:29:01.1767139Z ----------------------------------------------------------------------
2022-05-18T05:29:01.1767498Z Ran 1 test in 5.094s
2022-05-18T05:29:01.1767671Z 
2022-05-18T05:29:01.1767746Z OK
2022-05-18T05:29:01.1767880Z 
2022-05-18T05:29:01.1768045Z Generating XML reports...
2022-05-18T05:29:01.1824713Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052856.xml
2022-05-18T05:29:02.5949785Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:29:02.5966997Z 
2022-05-18T05:29:02.5967477Z Running tests...
2022-05-18T05:29:02.5967977Z ----------------------------------------------------------------------
2022-05-18T05:29:04.2553442Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:29:04.2947281Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83559
2022-05-18T05:29:04.3065378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83560
2022-05-18T05:29:05.4778694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:29:05.4779478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:29:05.4780524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:05.4781694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:05.4887620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:29:05.5795513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:29:06.7996675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnm40l7t0
2022-05-18T05:29:06.7998034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnm40l7t0/_remote_module_non_scriptable.py
2022-05-18T05:29:06.9028385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4detftv
2022-05-18T05:29:06.9029579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4detftv/_remote_module_non_scriptable.py
2022-05-18T05:29:07.2564460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:07.2565104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:07.3226208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:07.3226768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:07.7151331Z ok (5.118s)
2022-05-18T05:29:07.7151526Z 
2022-05-18T05:29:07.7152444Z ----------------------------------------------------------------------
2022-05-18T05:29:07.7152821Z Ran 1 test in 5.119s
2022-05-18T05:29:07.7153000Z 
2022-05-18T05:29:07.7153093Z OK
2022-05-18T05:29:07.7153211Z 
2022-05-18T05:29:07.7153345Z Generating XML reports...
2022-05-18T05:29:07.7222590Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052902.xml
2022-05-18T05:29:09.1726397Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:29:09.1742168Z 
2022-05-18T05:29:09.1742319Z Running tests...
2022-05-18T05:29:09.1743262Z ----------------------------------------------------------------------
2022-05-18T05:29:10.8328411Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:29:10.8735293Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83710
2022-05-18T05:29:10.8856995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83711
2022-05-18T05:29:12.0594433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:29:12.0595242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:29:12.0596069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:12.0596777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:12.0603460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:29:12.0603947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:29:13.4166001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5k62s8ni
2022-05-18T05:29:13.4261757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5k62s8ni/_remote_module_non_scriptable.py
2022-05-18T05:29:13.4262314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52_h8md8
2022-05-18T05:29:13.4263678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52_h8md8/_remote_module_non_scriptable.py
2022-05-18T05:29:13.7862738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:13.7863275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:13.8541409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:13.8541985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:14.1940775Z ok (5.019s)
2022-05-18T05:29:14.1941102Z 
2022-05-18T05:29:14.1941827Z ----------------------------------------------------------------------
2022-05-18T05:29:14.1942515Z Ran 1 test in 5.020s
2022-05-18T05:29:14.1942731Z 
2022-05-18T05:29:14.1942826Z OK
2022-05-18T05:29:14.1942965Z 
2022-05-18T05:29:14.1943099Z Generating XML reports...
2022-05-18T05:29:14.1999487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052909.xml
2022-05-18T05:29:15.6408856Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:29:15.6424743Z 
2022-05-18T05:29:15.6425093Z Running tests...
2022-05-18T05:29:15.6425555Z ----------------------------------------------------------------------
2022-05-18T05:29:17.2964352Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:29:17.3369362Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83861
2022-05-18T05:29:17.3489588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83862
2022-05-18T05:29:18.5668719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:29:18.5669292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:29:18.5670108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:18.5670798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:18.5778162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:29:18.6685713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:29:19.9059254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsec7nmd_
2022-05-18T05:29:19.9060418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsec7nmd_/_remote_module_non_scriptable.py
2022-05-18T05:29:19.9985403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3xvaktwh
2022-05-18T05:29:19.9986558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3xvaktwh/_remote_module_non_scriptable.py
2022-05-18T05:29:20.3626357Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:20.3626993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:20.4298695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:20.4299278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:20.8575341Z ok (5.215s)
2022-05-18T05:29:20.8575555Z 
2022-05-18T05:29:20.8575955Z ----------------------------------------------------------------------
2022-05-18T05:29:20.8576293Z Ran 1 test in 5.215s
2022-05-18T05:29:20.8576462Z 
2022-05-18T05:29:20.8576557Z OK
2022-05-18T05:29:20.8576692Z 
2022-05-18T05:29:20.8576825Z Generating XML reports...
2022-05-18T05:29:20.8634144Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052915.xml
2022-05-18T05:29:22.3027946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:29:22.3043001Z 
2022-05-18T05:29:22.3043154Z Running tests...
2022-05-18T05:29:22.3043596Z ----------------------------------------------------------------------
2022-05-18T05:29:23.9711364Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:29:24.0113244Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84012
2022-05-18T05:29:24.0235048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84013
2022-05-18T05:29:25.2655255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:29:25.2655824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:29:25.2656641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:25.2657340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:25.2664312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:29:25.2665387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:29:26.6188998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpblxhpxzl
2022-05-18T05:29:26.6189863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpblxhpxzl/_remote_module_non_scriptable.py
2022-05-18T05:29:26.6410213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9cpqmrm2
2022-05-18T05:29:26.6411566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9cpqmrm2/_remote_module_non_scriptable.py
2022-05-18T05:29:26.9888663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:26.9889228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:27.0528191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:27.0528736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:27.4320001Z ok (5.127s)
2022-05-18T05:29:27.4320392Z 
2022-05-18T05:29:27.4321081Z ----------------------------------------------------------------------
2022-05-18T05:29:27.4321697Z Ran 1 test in 5.128s
2022-05-18T05:29:27.4322008Z 
2022-05-18T05:29:27.4322176Z OK
2022-05-18T05:29:27.4322442Z 
2022-05-18T05:29:27.4322682Z Generating XML reports...
2022-05-18T05:29:27.4380426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052922.xml
2022-05-18T05:29:28.8660483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:29:28.8675120Z 
2022-05-18T05:29:28.8675487Z Running tests...
2022-05-18T05:29:28.8675931Z ----------------------------------------------------------------------
2022-05-18T05:29:30.5185411Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:29:30.5581898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84163
2022-05-18T05:29:30.5700305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84164
2022-05-18T05:29:31.7724531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:29:31.7725086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:29:31.7725915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:31.7726620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:31.7833785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:29:31.8739344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:29:33.1022854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_m54qgx1
2022-05-18T05:29:33.1024001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_m54qgx1/_remote_module_non_scriptable.py
2022-05-18T05:29:33.1919483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwt0usmq
2022-05-18T05:29:33.1920327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwt0usmq/_remote_module_non_scriptable.py
2022-05-18T05:29:33.5441167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:33.5441725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:33.6077027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:33.6077605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:33.9784990Z ok (5.111s)
2022-05-18T05:29:33.9785214Z 
2022-05-18T05:29:33.9785622Z ----------------------------------------------------------------------
2022-05-18T05:29:33.9785958Z Ran 1 test in 5.111s
2022-05-18T05:29:33.9786106Z 
2022-05-18T05:29:33.9786201Z OK
2022-05-18T05:29:33.9786343Z 
2022-05-18T05:29:33.9786478Z Generating XML reports...
2022-05-18T05:29:33.9843208Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052928.xml
2022-05-18T05:29:35.3926480Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:29:35.3941665Z 
2022-05-18T05:29:35.3941993Z Running tests...
2022-05-18T05:29:35.3942417Z ----------------------------------------------------------------------
2022-05-18T05:29:37.0394291Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:29:37.0803743Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84314
2022-05-18T05:29:37.0924355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84315
2022-05-18T05:29:38.3156467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:29:38.3157050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:29:38.3157836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:38.3158760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:38.3267345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:29:38.4170538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:29:39.6538547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk51b2g4m
2022-05-18T05:29:39.6539865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk51b2g4m/_remote_module_non_scriptable.py
2022-05-18T05:29:39.7450631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrbymbg4
2022-05-18T05:29:39.7451866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrbymbg4/_remote_module_non_scriptable.py
2022-05-18T05:29:40.1146702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:40.1147689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:40.1823064Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:40.1824453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:40.6007886Z ok (5.206s)
2022-05-18T05:29:40.6008300Z 
2022-05-18T05:29:40.6008764Z ----------------------------------------------------------------------
2022-05-18T05:29:40.6009155Z Ran 1 test in 5.207s
2022-05-18T05:29:40.6009321Z 
2022-05-18T05:29:40.6009414Z OK
2022-05-18T05:29:40.6009787Z 
2022-05-18T05:29:40.6009933Z Generating XML reports...
2022-05-18T05:29:40.6066449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052935.xml
2022-05-18T05:29:42.0583861Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:29:42.0599333Z 
2022-05-18T05:29:42.0599730Z Running tests...
2022-05-18T05:29:42.0600228Z ----------------------------------------------------------------------
2022-05-18T05:29:43.7305780Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:29:43.7697595Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84465
2022-05-18T05:29:43.7816516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84466
2022-05-18T05:29:44.9578062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:29:44.9578626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:29:44.9579419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:44.9580148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:44.9686358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:29:45.0593240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:29:46.2959577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5_kevn_
2022-05-18T05:29:46.2960471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5_kevn_/_remote_module_non_scriptable.py
2022-05-18T05:29:46.3907163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3isl6bbk
2022-05-18T05:29:46.3908425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3isl6bbk/_remote_module_non_scriptable.py
2022-05-18T05:29:46.7557086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:46.7557934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:46.8225218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:46.8225751Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:47.1899641Z ok (5.130s)
2022-05-18T05:29:47.1899830Z 
2022-05-18T05:29:47.1900216Z ----------------------------------------------------------------------
2022-05-18T05:29:47.1900538Z Ran 1 test in 5.130s
2022-05-18T05:29:47.1900708Z 
2022-05-18T05:29:47.1900925Z OK
2022-05-18T05:29:47.1901064Z 
2022-05-18T05:29:47.1901202Z Generating XML reports...
2022-05-18T05:29:47.1958723Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052942.xml
2022-05-18T05:29:48.6533370Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:29:48.6549483Z 
2022-05-18T05:29:48.6549687Z Running tests...
2022-05-18T05:29:48.6550142Z ----------------------------------------------------------------------
2022-05-18T05:29:50.3052991Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:29:50.3461078Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84616
2022-05-18T05:29:50.3581460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84617
2022-05-18T05:29:51.5013418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:29:51.5013986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:29:51.5014769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:51.5015490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:51.5124272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:29:51.6028371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:29:52.8190426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvyc5e5v1
2022-05-18T05:29:52.8191051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvyc5e5v1/_remote_module_non_scriptable.py
2022-05-18T05:29:52.9142725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz1kfb7b9
2022-05-18T05:29:52.9143685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz1kfb7b9/_remote_module_non_scriptable.py
2022-05-18T05:29:53.2584621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:53.2585169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:53.3255848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:53.3256406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:53.7666290Z ok (5.111s)
2022-05-18T05:29:53.7666494Z 
2022-05-18T05:29:53.7666873Z ----------------------------------------------------------------------
2022-05-18T05:29:53.7667221Z Ran 1 test in 5.112s
2022-05-18T05:29:53.7667387Z 
2022-05-18T05:29:53.7667480Z OK
2022-05-18T05:29:53.7667620Z 
2022-05-18T05:29:53.7667731Z Generating XML reports...
2022-05-18T05:29:53.7726295Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052948.xml
2022-05-18T05:29:55.2179719Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:29:55.2195420Z 
2022-05-18T05:29:55.2195825Z Running tests...
2022-05-18T05:29:55.2196268Z ----------------------------------------------------------------------
2022-05-18T05:29:56.8660153Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:29:56.9067952Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84767
2022-05-18T05:29:56.9189673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84768
2022-05-18T05:29:58.0964884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:29:58.0965975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:29:58.0966778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:58.0967471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:29:58.0974297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:29:58.0974822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:29:59.4404634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7izsnqlh
2022-05-18T05:29:59.4405480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7izsnqlh/_remote_module_non_scriptable.py
2022-05-18T05:29:59.4789911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2t_zpm0f
2022-05-18T05:29:59.4791150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2t_zpm0f/_remote_module_non_scriptable.py
2022-05-18T05:29:59.8314248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:59.8314800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:59.8941073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:29:59.8941734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:30:00.2272418Z ok (5.007s)
2022-05-18T05:30:00.2272604Z 
2022-05-18T05:30:00.2272990Z ----------------------------------------------------------------------
2022-05-18T05:30:00.2273362Z Ran 1 test in 5.008s
2022-05-18T05:30:00.2273531Z 
2022-05-18T05:30:00.2273626Z OK
2022-05-18T05:30:00.2273764Z 
2022-05-18T05:30:00.2273877Z Generating XML reports...
2022-05-18T05:30:00.2331513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052955.xml
2022-05-18T05:30:01.6815486Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:30:01.6831573Z 
2022-05-18T05:30:01.6831972Z Running tests...
2022-05-18T05:30:01.6832421Z ----------------------------------------------------------------------
2022-05-18T05:30:03.3644517Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:30:03.3765815Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.693s)
2022-05-18T05:30:03.3766409Z 
2022-05-18T05:30:03.3766686Z ----------------------------------------------------------------------
2022-05-18T05:30:03.3767017Z Ran 1 test in 1.693s
2022-05-18T05:30:03.3767181Z 
2022-05-18T05:30:03.3767272Z OK (skipped=1)
2022-05-18T05:30:03.3767430Z 
2022-05-18T05:30:03.3767558Z Generating XML reports...
2022-05-18T05:30:03.3807565Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053001.xml
2022-05-18T05:30:04.7701373Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:30:04.7715941Z 
2022-05-18T05:30:04.7716713Z Running tests...
2022-05-18T05:30:04.7717236Z ----------------------------------------------------------------------
2022-05-18T05:30:06.3854253Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:30:06.4248621Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84954
2022-05-18T05:30:06.4369467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84955
2022-05-18T05:30:07.6433718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:30:07.6434277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:30:07.6435070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:07.6435775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:07.6442489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:30:07.6443195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:30:08.9623946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2hw5yc9
2022-05-18T05:30:08.9624908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2hw5yc9/_remote_module_non_scriptable.py
2022-05-18T05:30:08.9930437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mepkarf
2022-05-18T05:30:08.9933401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mepkarf/_remote_module_non_scriptable.py
2022-05-18T05:30:09.9457937Z ok (5.174s)
2022-05-18T05:30:09.9458146Z 
2022-05-18T05:30:09.9458557Z ----------------------------------------------------------------------
2022-05-18T05:30:09.9458883Z Ran 1 test in 5.174s
2022-05-18T05:30:09.9459049Z 
2022-05-18T05:30:09.9459160Z OK
2022-05-18T05:30:09.9459298Z 
2022-05-18T05:30:09.9459438Z Generating XML reports...
2022-05-18T05:30:09.9516044Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053004.xml
2022-05-18T05:30:11.3933509Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:30:11.3949512Z 
2022-05-18T05:30:11.3950176Z Running tests...
2022-05-18T05:30:11.3950655Z ----------------------------------------------------------------------
2022-05-18T05:30:13.0442720Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:30:13.0848740Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85071
2022-05-18T05:30:13.0971306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85072
2022-05-18T05:30:14.2962496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:30:14.2963044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:30:14.2963826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:14.2964545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:14.3072608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:30:14.3977990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:30:15.8474482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fb5xew6
2022-05-18T05:30:15.8475083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fb5xew6/_remote_module_non_scriptable.py
2022-05-18T05:30:15.9674972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp709af_lt
2022-05-18T05:30:15.9675953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp709af_lt/_remote_module_non_scriptable.py
2022-05-18T05:30:15.9866029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:30:15.9866562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:30:15.9976127Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp3fb5xew6'>
2022-05-18T05:30:15.9976629Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:30:15.9977230Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp709af_lt'>
2022-05-18T05:30:15.9977666Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:30:16.3050028Z ok (4.910s)
2022-05-18T05:30:16.3050264Z 
2022-05-18T05:30:16.3050941Z ----------------------------------------------------------------------
2022-05-18T05:30:16.3051285Z Ran 1 test in 4.910s
2022-05-18T05:30:16.3051430Z 
2022-05-18T05:30:16.3051527Z OK
2022-05-18T05:30:16.3054975Z 
2022-05-18T05:30:16.3055416Z Generating XML reports...
2022-05-18T05:30:16.3107811Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053011.xml
2022-05-18T05:30:17.7338866Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:30:17.7360958Z 
2022-05-18T05:30:17.7361191Z Running tests...
2022-05-18T05:30:17.7361912Z ----------------------------------------------------------------------
2022-05-18T05:30:19.3624753Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:30:19.4021020Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85192
2022-05-18T05:30:19.4138040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85193
2022-05-18T05:30:20.6367374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:30:20.6367949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:30:20.6368768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:20.6369458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:20.6377180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:30:20.6377693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:30:20.6469853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp576wmeo3
2022-05-18T05:30:20.6472346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp576wmeo3/_remote_module_non_scriptable.py
2022-05-18T05:30:20.6476821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprtq6jbj
2022-05-18T05:30:20.6479976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprtq6jbj/_remote_module_non_scriptable.py
2022-05-18T05:30:20.6657265Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:30:20.6657783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:30:20.9189216Z ok (3.183s)
2022-05-18T05:30:20.9189435Z 
2022-05-18T05:30:20.9189797Z ----------------------------------------------------------------------
2022-05-18T05:30:20.9190132Z Ran 1 test in 3.183s
2022-05-18T05:30:20.9190301Z 
2022-05-18T05:30:20.9190394Z OK
2022-05-18T05:30:20.9190528Z 
2022-05-18T05:30:20.9190659Z Generating XML reports...
2022-05-18T05:30:20.9247204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053017.xml
2022-05-18T05:30:22.3335001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:30:22.3349372Z 
2022-05-18T05:30:22.3349877Z Running tests...
2022-05-18T05:30:22.3350423Z ----------------------------------------------------------------------
2022-05-18T05:30:23.9546472Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:30:23.9942587Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85341
2022-05-18T05:30:24.0064037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85342
2022-05-18T05:30:25.1691467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:30:25.1692301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:30:25.1693142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:25.1693843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:25.1701343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:30:25.1701846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:30:26.4736686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgdtj8d06
2022-05-18T05:30:26.4737301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgdtj8d06/_remote_module_non_scriptable.py
2022-05-18T05:30:26.5098674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpws7hz8ze
2022-05-18T05:30:26.5101100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpws7hz8ze/_remote_module_non_scriptable.py
2022-05-18T05:30:26.8139536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:30:26.8140060Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:30:27.1150648Z ok (4.780s)
2022-05-18T05:30:27.1150957Z 
2022-05-18T05:30:27.1151519Z ----------------------------------------------------------------------
2022-05-18T05:30:27.1151846Z Ran 1 test in 4.780s
2022-05-18T05:30:27.1152014Z 
2022-05-18T05:30:27.1152112Z OK
2022-05-18T05:30:27.1152264Z 
2022-05-18T05:30:27.1152398Z Generating XML reports...
2022-05-18T05:30:27.1208006Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053022.xml
2022-05-18T05:30:28.5647687Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:30:28.5663421Z 
2022-05-18T05:30:28.5663875Z Running tests...
2022-05-18T05:30:28.5664792Z ----------------------------------------------------------------------
2022-05-18T05:30:30.2033990Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:30:30.2435899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85462
2022-05-18T05:30:30.2555669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85463
2022-05-18T05:30:31.4632413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:30:31.4632998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:30:31.4633803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:31.4634515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:31.4742068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:30:31.5644051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:30:31.5854053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:30:31.5854611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:30:31.5855315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:30:31.5856026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:30:31.6064166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:30:31.6064695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:30:31.6065368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:30:31.6066074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:30:32.9337225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpslomf318
2022-05-18T05:30:32.9338323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpslomf318/_remote_module_non_scriptable.py
2022-05-18T05:30:32.9668595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7lmw6mo
2022-05-18T05:30:32.9670547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7lmw6mo/_remote_module_non_scriptable.py
2022-05-18T05:30:33.2632512Z ok (4.697s)
2022-05-18T05:30:33.2632730Z 
2022-05-18T05:30:33.2633116Z ----------------------------------------------------------------------
2022-05-18T05:30:33.2633464Z Ran 1 test in 4.697s
2022-05-18T05:30:33.2633631Z 
2022-05-18T05:30:33.2633713Z OK
2022-05-18T05:30:33.2633855Z 
2022-05-18T05:30:33.2634006Z Generating XML reports...
2022-05-18T05:30:33.2690818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053028.xml
2022-05-18T05:30:34.6868222Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:30:34.6882153Z 
2022-05-18T05:30:34.6882463Z Running tests...
2022-05-18T05:30:34.6882911Z ----------------------------------------------------------------------
2022-05-18T05:30:36.2978532Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:30:36.3373620Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85591
2022-05-18T05:30:36.3493621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85592
2022-05-18T05:30:37.5495189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:30:37.5495762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:30:37.5496541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:37.5497253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:37.5603988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:30:37.6506824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:30:37.6617241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:30:37.6617766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:30:37.6618682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:30:37.6619388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:30:37.6825039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:30:37.6825551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:30:37.6826234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:30:37.6826914Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:30:38.9938693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7bbl2ui
2022-05-18T05:30:38.9939319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7bbl2ui/_remote_module_non_scriptable.py
2022-05-18T05:30:39.0132102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_u1b72kc
2022-05-18T05:30:39.0135443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_u1b72kc/_remote_module_non_scriptable.py
2022-05-18T05:30:49.3744238Z ok (14.686s)
2022-05-18T05:30:49.3744453Z 
2022-05-18T05:30:49.3744848Z ----------------------------------------------------------------------
2022-05-18T05:30:49.3745200Z Ran 1 test in 14.686s
2022-05-18T05:30:49.3745349Z 
2022-05-18T05:30:49.3745447Z OK
2022-05-18T05:30:49.3745584Z 
2022-05-18T05:30:49.3745723Z Generating XML reports...
2022-05-18T05:30:49.3802109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053034.xml
2022-05-18T05:30:50.8273835Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:30:50.8289914Z 
2022-05-18T05:30:50.8290383Z Running tests...
2022-05-18T05:30:50.8290914Z ----------------------------------------------------------------------
2022-05-18T05:30:52.4864365Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:30:52.5268782Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85720
2022-05-18T05:30:52.5390312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85721
2022-05-18T05:30:53.7655774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:30:53.7656336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:30:53.7657152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:53.7657859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:30:53.7665495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:30:53.7666001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:30:55.1001232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuoy6du3j
2022-05-18T05:30:55.1001852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuoy6du3j/_remote_module_non_scriptable.py
2022-05-18T05:30:55.1247184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5pkgrj4
2022-05-18T05:30:55.1249689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5pkgrj4/_remote_module_non_scriptable.py
2022-05-18T05:30:56.0473644Z ok (5.218s)
2022-05-18T05:30:56.0473867Z 
2022-05-18T05:30:56.0474254Z ----------------------------------------------------------------------
2022-05-18T05:30:56.0474595Z Ran 1 test in 5.218s
2022-05-18T05:30:56.0474897Z 
2022-05-18T05:30:56.0475025Z OK
2022-05-18T05:30:56.0475175Z 
2022-05-18T05:30:56.0475554Z Generating XML reports...
2022-05-18T05:30:56.0532947Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053050.xml
2022-05-18T05:30:57.4988670Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:30:57.5003890Z 
2022-05-18T05:30:57.5004202Z Running tests...
2022-05-18T05:30:57.5004663Z ----------------------------------------------------------------------
2022-05-18T05:30:59.1542341Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:30:59.1949233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85841
2022-05-18T05:30:59.2070666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85842
2022-05-18T05:31:00.4094512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:31:00.4095266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:31:00.4096678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:00.4098154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:00.4104357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:31:00.4106807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:31:01.7663767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5ke1m4o
2022-05-18T05:31:01.7664888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5ke1m4o/_remote_module_non_scriptable.py
2022-05-18T05:31:01.7761314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhotcp31
2022-05-18T05:31:01.7763636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhotcp31/_remote_module_non_scriptable.py
2022-05-18T05:31:02.7158488Z ok (5.215s)
2022-05-18T05:31:02.7158715Z 
2022-05-18T05:31:02.7159137Z ----------------------------------------------------------------------
2022-05-18T05:31:02.7159490Z Ran 1 test in 5.215s
2022-05-18T05:31:02.7212044Z 
2022-05-18T05:31:02.7212177Z OK
2022-05-18T05:31:02.7212354Z 
2022-05-18T05:31:02.7212479Z Generating XML reports...
2022-05-18T05:31:02.7217995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053057.xml
2022-05-18T05:31:04.1500657Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:31:04.1515346Z 
2022-05-18T05:31:04.1515847Z Running tests...
2022-05-18T05:31:04.1516356Z ----------------------------------------------------------------------
2022-05-18T05:31:05.7854668Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:31:05.8254110Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85962
2022-05-18T05:31:05.8375222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85963
2022-05-18T05:31:07.0080490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:31:07.0081036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:31:07.0081841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:07.0082548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:07.0189723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:31:07.1095908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:31:08.3303481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuz9bkzfb
2022-05-18T05:31:08.3304090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuz9bkzfb/_remote_module_non_scriptable.py
2022-05-18T05:31:08.4130401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy700uaak
2022-05-18T05:31:08.4131014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy700uaak/_remote_module_non_scriptable.py
2022-05-18T05:31:09.0457105Z ok (4.894s)
2022-05-18T05:31:09.0457345Z 
2022-05-18T05:31:09.0457727Z ----------------------------------------------------------------------
2022-05-18T05:31:09.0458067Z Ran 1 test in 4.894s
2022-05-18T05:31:09.0458235Z 
2022-05-18T05:31:09.0458328Z OK
2022-05-18T05:31:09.0458446Z 
2022-05-18T05:31:09.0458578Z Generating XML reports...
2022-05-18T05:31:09.0516486Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053104.xml
2022-05-18T05:31:10.4982528Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:31:10.4997611Z 
2022-05-18T05:31:10.4998069Z Running tests...
2022-05-18T05:31:10.4998534Z ----------------------------------------------------------------------
2022-05-18T05:31:12.1716912Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:31:12.2120888Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86079
2022-05-18T05:31:12.2245086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86080
2022-05-18T05:31:13.3788677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:31:13.3789226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:31:13.3790047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:13.3790748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:13.3797543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:31:13.3798317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:31:14.7122136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjdcxksug
2022-05-18T05:31:14.7122770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjdcxksug/_remote_module_non_scriptable.py
2022-05-18T05:31:14.7212128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp52e415k
2022-05-18T05:31:14.7215099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp52e415k/_remote_module_non_scriptable.py
2022-05-18T05:31:15.0189723Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:31:15.0217093Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:31:15.0312691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:31:15.0313233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:31:15.3324489Z ok (4.832s)
2022-05-18T05:31:15.3324679Z 
2022-05-18T05:31:15.3325075Z ----------------------------------------------------------------------
2022-05-18T05:31:15.3325416Z Ran 1 test in 4.833s
2022-05-18T05:31:15.3325581Z 
2022-05-18T05:31:15.3325677Z OK
2022-05-18T05:31:15.3325818Z 
2022-05-18T05:31:15.3325956Z Generating XML reports...
2022-05-18T05:31:15.3382411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053110.xml
2022-05-18T05:31:16.7866969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:31:16.7882050Z 
2022-05-18T05:31:16.7882348Z Running tests...
2022-05-18T05:31:16.7882803Z ----------------------------------------------------------------------
2022-05-18T05:31:18.4707023Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:31:18.5114382Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86200
2022-05-18T05:31:18.5238958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86201
2022-05-18T05:31:19.7388552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:31:19.7389111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:31:19.7389911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:19.7390598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:19.7498154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:31:19.8404898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:31:21.0438529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2n8ky3i
2022-05-18T05:31:21.0439179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2n8ky3i/_remote_module_non_scriptable.py
2022-05-18T05:31:21.1478316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphiveromp
2022-05-18T05:31:21.1479369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphiveromp/_remote_module_non_scriptable.py
2022-05-18T05:31:21.1650824Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:31:21.1651734Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T05:31:21.1652918Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:31:21.1653757Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T05:31:21.7320405Z ok (4.944s)
2022-05-18T05:31:21.7320627Z 
2022-05-18T05:31:21.7321006Z ----------------------------------------------------------------------
2022-05-18T05:31:21.7321345Z Ran 1 test in 4.944s
2022-05-18T05:31:21.7321518Z 
2022-05-18T05:31:21.7321613Z OK
2022-05-18T05:31:21.7321979Z 
2022-05-18T05:31:21.7322116Z Generating XML reports...
2022-05-18T05:31:21.7388572Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053116.xml
2022-05-18T05:31:23.1858517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:31:23.1873395Z 
2022-05-18T05:31:23.1873887Z Running tests...
2022-05-18T05:31:23.1874405Z ----------------------------------------------------------------------
2022-05-18T05:31:24.8421848Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:31:24.8543254Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.667s)
2022-05-18T05:31:24.8543866Z 
2022-05-18T05:31:24.8544150Z ----------------------------------------------------------------------
2022-05-18T05:31:24.8544483Z Ran 1 test in 1.667s
2022-05-18T05:31:24.8544628Z 
2022-05-18T05:31:24.8545026Z OK (skipped=1)
2022-05-18T05:31:24.8545186Z 
2022-05-18T05:31:24.8545312Z Generating XML reports...
2022-05-18T05:31:24.8583716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053123.xml
2022-05-18T05:31:26.2517619Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:31:26.2533752Z 
2022-05-18T05:31:26.2534214Z Running tests...
2022-05-18T05:31:26.2534697Z ----------------------------------------------------------------------
2022-05-18T05:31:27.8929242Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:31:27.9327190Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86357
2022-05-18T05:31:27.9448472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86358
2022-05-18T05:31:29.1593468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:31:29.1594030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:31:29.1594823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:29.1595513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:29.1602541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:31:29.1603275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:31:30.4953045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58ucfk2v
2022-05-18T05:31:30.4953949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58ucfk2v/_remote_module_non_scriptable.py
2022-05-18T05:31:30.5123917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkqcpocwe
2022-05-18T05:31:30.5127107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkqcpocwe/_remote_module_non_scriptable.py
2022-05-18T05:31:31.2365913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:31:31.2366473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:31:31.2729267Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:31:31.2731335Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:31:31.6542201Z ok (5.401s)
2022-05-18T05:31:31.6542573Z 
2022-05-18T05:31:31.6542974Z ----------------------------------------------------------------------
2022-05-18T05:31:31.6543319Z Ran 1 test in 5.401s
2022-05-18T05:31:31.6543465Z 
2022-05-18T05:31:31.6543567Z OK
2022-05-18T05:31:31.6543711Z 
2022-05-18T05:31:31.6543842Z Generating XML reports...
2022-05-18T05:31:31.6601602Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053126.xml
2022-05-18T05:31:33.1037604Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:31:33.1053968Z 
2022-05-18T05:31:33.1054313Z Running tests...
2022-05-18T05:31:33.1054778Z ----------------------------------------------------------------------
2022-05-18T05:31:34.7556776Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:31:34.7951686Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86482
2022-05-18T05:31:34.8069510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86483
2022-05-18T05:31:36.0106779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:31:36.0107370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:31:36.0108163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:36.0108884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:36.0216242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:31:36.1121764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:31:37.3264869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg15x7x9
2022-05-18T05:31:37.3265794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg15x7x9/_remote_module_non_scriptable.py
2022-05-18T05:31:37.4178724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08rs9dh0
2022-05-18T05:31:37.4179571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08rs9dh0/_remote_module_non_scriptable.py
2022-05-18T05:31:37.7145880Z ok (4.609s)
2022-05-18T05:31:37.7146098Z 
2022-05-18T05:31:37.7146501Z ----------------------------------------------------------------------
2022-05-18T05:31:37.7146854Z Ran 1 test in 4.609s
2022-05-18T05:31:37.7147023Z 
2022-05-18T05:31:37.7147099Z OK
2022-05-18T05:31:37.7147238Z 
2022-05-18T05:31:37.7147372Z Generating XML reports...
2022-05-18T05:31:37.7204936Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053133.xml
2022-05-18T05:31:39.1596566Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:31:39.1611224Z 
2022-05-18T05:31:39.1611630Z Running tests...
2022-05-18T05:31:39.1612124Z ----------------------------------------------------------------------
2022-05-18T05:31:40.7899903Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:31:40.8300160Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86599
2022-05-18T05:31:40.8416248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86600
2022-05-18T05:31:42.0458033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:31:42.0458605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:31:42.0459399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:42.0460089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:42.0466802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:31:42.0467316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:31:43.3782056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqkpe10ck
2022-05-18T05:31:43.3782943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqkpe10ck/_remote_module_non_scriptable.py
2022-05-18T05:31:43.3989511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqd2jj8so
2022-05-18T05:31:43.3992025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqd2jj8so/_remote_module_non_scriptable.py
2022-05-18T05:31:43.4162381Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:31:43.4163283Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T05:31:43.4164430Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:31:43.4165292Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T05:31:43.7064712Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:31:43.7065275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:31:43.7140514Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:31:43.7142159Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:31:43.7292482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:31:43.7292999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:31:43.7382905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:31:43.7383418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:31:44.0497281Z ok (4.888s)
2022-05-18T05:31:44.0497490Z 
2022-05-18T05:31:44.0498088Z ----------------------------------------------------------------------
2022-05-18T05:31:44.0498529Z Ran 1 test in 4.889s
2022-05-18T05:31:44.0498710Z 
2022-05-18T05:31:44.0498806Z OK
2022-05-18T05:31:44.0498945Z 
2022-05-18T05:31:44.0499063Z Generating XML reports...
2022-05-18T05:31:44.0555932Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053139.xml
2022-05-18T05:31:45.4742344Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:31:45.4756755Z 
2022-05-18T05:31:45.4757008Z Running tests...
2022-05-18T05:31:45.4757452Z ----------------------------------------------------------------------
2022-05-18T05:31:47.0857236Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:31:47.1254720Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86720
2022-05-18T05:31:47.1376523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86721
2022-05-18T05:31:48.3911403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:31:48.3911990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:31:48.3912805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:48.3913509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:48.3919988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:31:48.3920823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:31:49.7282763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwso5hztn
2022-05-18T05:31:49.7283385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwso5hztn/_remote_module_non_scriptable.py
2022-05-18T05:31:49.7339768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1s_plwu
2022-05-18T05:31:49.7342774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1s_plwu/_remote_module_non_scriptable.py
2022-05-18T05:31:49.7514894Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:31:49.7515800Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T05:31:49.7516979Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1737: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:31:49.7517822Z   "You passed find_unused_parameters=true to DistributedDataParallel, "
2022-05-18T05:31:50.0411779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:31:50.0413180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:31:50.3460466Z ok (4.870s)
2022-05-18T05:31:50.3460708Z 
2022-05-18T05:31:50.3461326Z ----------------------------------------------------------------------
2022-05-18T05:31:50.3461771Z Ran 1 test in 4.870s
2022-05-18T05:31:50.3461944Z 
2022-05-18T05:31:50.3462018Z OK
2022-05-18T05:31:50.3464170Z 
2022-05-18T05:31:50.3464735Z Generating XML reports...
2022-05-18T05:31:50.3519074Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053145.xml
2022-05-18T05:31:51.7707599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:31:51.7722328Z 
2022-05-18T05:31:51.7722749Z Running tests...
2022-05-18T05:31:51.7723255Z ----------------------------------------------------------------------
2022-05-18T05:31:53.3933960Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:31:53.4051135Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2022-05-18T05:31:53.4052001Z 
2022-05-18T05:31:53.4052292Z ----------------------------------------------------------------------
2022-05-18T05:31:53.4052626Z Ran 1 test in 1.633s
2022-05-18T05:31:53.4052792Z 
2022-05-18T05:31:53.4052885Z OK (skipped=1)
2022-05-18T05:31:53.4053044Z 
2022-05-18T05:31:53.4053167Z Generating XML reports...
2022-05-18T05:31:53.4091037Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053151.xml
2022-05-18T05:31:54.7680908Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:31:54.7697712Z 
2022-05-18T05:31:54.7697969Z Running tests...
2022-05-18T05:31:54.7698425Z ----------------------------------------------------------------------
2022-05-18T05:31:56.4312435Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:31:56.4717450Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86877
2022-05-18T05:31:56.4838220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86878
2022-05-18T05:31:57.6530027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:31:57.6531101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:31:57.6532593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:57.6533843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:31:57.6638404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:31:57.7546848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:31:58.9818910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa7yvyxmv
2022-05-18T05:31:58.9820109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa7yvyxmv/_remote_module_non_scriptable.py
2022-05-18T05:31:59.0595893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfst8lzzw
2022-05-18T05:31:59.0597099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfst8lzzw/_remote_module_non_scriptable.py
2022-05-18T05:31:59.4368566Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:31:59.4369913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:32:00.4934330Z ok (5.723s)
2022-05-18T05:32:00.4934558Z 
2022-05-18T05:32:00.4935198Z ----------------------------------------------------------------------
2022-05-18T05:32:00.4935568Z Ran 1 test in 5.724s
2022-05-18T05:32:00.4935738Z 
2022-05-18T05:32:00.4935851Z OK
2022-05-18T05:32:00.4935989Z 
2022-05-18T05:32:00.4936129Z Generating XML reports...
2022-05-18T05:32:00.4993501Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053154.xml
2022-05-18T05:32:01.9332532Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:32:01.9347257Z 
2022-05-18T05:32:01.9347411Z Running tests...
2022-05-18T05:32:01.9348253Z ----------------------------------------------------------------------
2022-05-18T05:32:03.5661893Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:32:03.6061930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87002
2022-05-18T05:32:03.6177348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87003
2022-05-18T05:32:04.8183405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:32:04.8184219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:32:04.8185020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:04.8185727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:04.8294073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:32:04.9198769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:32:06.1367196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5zze4x59
2022-05-18T05:32:06.1368368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5zze4x59/_remote_module_non_scriptable.py
2022-05-18T05:32:06.2179048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw97ffl95
2022-05-18T05:32:06.2180656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw97ffl95/_remote_module_non_scriptable.py
2022-05-18T05:32:06.5254338Z ok (4.590s)
2022-05-18T05:32:06.5254550Z 
2022-05-18T05:32:06.5254927Z ----------------------------------------------------------------------
2022-05-18T05:32:06.5255274Z Ran 1 test in 4.591s
2022-05-18T05:32:06.5255443Z 
2022-05-18T05:32:06.5255540Z OK
2022-05-18T05:32:06.5255677Z 
2022-05-18T05:32:06.5255814Z Generating XML reports...
2022-05-18T05:32:06.5314292Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053201.xml
2022-05-18T05:32:07.9592308Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:32:07.9606754Z 
2022-05-18T05:32:07.9607118Z Running tests...
2022-05-18T05:32:07.9607639Z ----------------------------------------------------------------------
2022-05-18T05:32:09.5917244Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:32:09.6315506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87119
2022-05-18T05:32:09.6432271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87120
2022-05-18T05:32:10.8161190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:32:10.8161750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:32:10.8162553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:10.8163474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:10.8270071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:32:10.9176485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:32:12.1275770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodyvewd2
2022-05-18T05:32:12.1276906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodyvewd2/_remote_module_non_scriptable.py
2022-05-18T05:32:12.2062994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vt375sw
2022-05-18T05:32:12.2064307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vt375sw/_remote_module_non_scriptable.py
2022-05-18T05:32:12.5517813Z ok (4.591s)
2022-05-18T05:32:12.5518047Z 
2022-05-18T05:32:12.5518426Z ----------------------------------------------------------------------
2022-05-18T05:32:12.5518772Z Ran 1 test in 4.591s
2022-05-18T05:32:12.5518961Z 
2022-05-18T05:32:12.5519056Z OK
2022-05-18T05:32:12.5520260Z 
2022-05-18T05:32:12.5520841Z Generating XML reports...
2022-05-18T05:32:12.5575988Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053207.xml
2022-05-18T05:32:13.9875083Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:32:13.9889286Z 
2022-05-18T05:32:13.9889570Z Running tests...
2022-05-18T05:32:13.9890287Z ----------------------------------------------------------------------
2022-05-18T05:32:15.6124626Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:32:15.6522677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87236
2022-05-18T05:32:15.6642983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87237
2022-05-18T05:32:16.8620160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:32:16.8620752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:32:16.8621535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:16.8622240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:16.8629018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:32:16.8630220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:32:18.1856155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhoie8eh
2022-05-18T05:32:18.1859581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhoie8eh/_remote_module_non_scriptable.py
2022-05-18T05:32:18.2262195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphskwvoi8
2022-05-18T05:32:18.2264578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphskwvoi8/_remote_module_non_scriptable.py
2022-05-18T05:32:18.5333983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:32:18.5334556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:32:18.8725497Z ok (4.883s)
2022-05-18T05:32:18.8725848Z 
2022-05-18T05:32:18.8726829Z ----------------------------------------------------------------------
2022-05-18T05:32:18.8727494Z Ran 1 test in 4.884s
2022-05-18T05:32:18.8727808Z 
2022-05-18T05:32:18.8727965Z OK
2022-05-18T05:32:18.8728220Z 
2022-05-18T05:32:18.8728457Z Generating XML reports...
2022-05-18T05:32:18.8786987Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053213.xml
2022-05-18T05:32:20.2997141Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:32:20.3012079Z 
2022-05-18T05:32:20.3012544Z Running tests...
2022-05-18T05:32:20.3013029Z ----------------------------------------------------------------------
2022-05-18T05:32:21.9375835Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:32:21.9491967Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.648s)
2022-05-18T05:32:21.9492560Z 
2022-05-18T05:32:21.9492843Z ----------------------------------------------------------------------
2022-05-18T05:32:21.9493181Z Ran 1 test in 1.648s
2022-05-18T05:32:21.9493328Z 
2022-05-18T05:32:21.9493441Z OK (skipped=1)
2022-05-18T05:32:21.9493596Z 
2022-05-18T05:32:21.9493740Z Generating XML reports...
2022-05-18T05:32:21.9537936Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053220.xml
2022-05-18T05:32:23.3474207Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:32:23.3489803Z 
2022-05-18T05:32:23.3489948Z Running tests...
2022-05-18T05:32:23.3490416Z ----------------------------------------------------------------------
2022-05-18T05:32:24.9896897Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:32:25.0297043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87393
2022-05-18T05:32:25.0416568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87394
2022-05-18T05:32:26.2738420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:32:26.2739023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:32:26.2739816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:26.2740519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:26.2748641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:32:26.2749139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:32:27.6304213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58tkeyk6
2022-05-18T05:32:27.6305932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58tkeyk6/_remote_module_non_scriptable.py
2022-05-18T05:32:27.6306489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvzrqal0
2022-05-18T05:32:27.6309686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvzrqal0/_remote_module_non_scriptable.py
2022-05-18T05:32:27.9461819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:32:27.9462385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:32:27.9801989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:32:27.9802492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:32:27.9967562Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpmvzrqal0'>
2022-05-18T05:32:27.9968036Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:32:28.3501073Z ok (5.001s)
2022-05-18T05:32:28.3501407Z 
2022-05-18T05:32:28.3501827Z ----------------------------------------------------------------------
2022-05-18T05:32:28.3502161Z Ran 1 test in 5.001s
2022-05-18T05:32:28.3502581Z 
2022-05-18T05:32:28.3502698Z OK
2022-05-18T05:32:28.3502849Z 
2022-05-18T05:32:28.3502988Z Generating XML reports...
2022-05-18T05:32:28.3561369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053223.xml
2022-05-18T05:32:29.7784166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:32:29.7799601Z 
2022-05-18T05:32:29.7799902Z Running tests...
2022-05-18T05:32:29.7800365Z ----------------------------------------------------------------------
2022-05-18T05:32:31.4089552Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:32:31.4489337Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87514
2022-05-18T05:32:31.4607331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87515
2022-05-18T05:32:32.6223778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:32:32.6224376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:32:32.6225421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:32.6226105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:32.6333687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:32:32.7238613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:32:33.9223527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwk0md9t
2022-05-18T05:32:33.9224149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwk0md9t/_remote_module_non_scriptable.py
2022-05-18T05:32:34.0193562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54wwikun
2022-05-18T05:32:34.0194378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54wwikun/_remote_module_non_scriptable.py
2022-05-18T05:32:34.6687133Z ok (4.888s)
2022-05-18T05:32:34.6687350Z 
2022-05-18T05:32:34.6687746Z ----------------------------------------------------------------------
2022-05-18T05:32:34.6688068Z Ran 1 test in 4.889s
2022-05-18T05:32:34.6688243Z 
2022-05-18T05:32:34.6688338Z OK
2022-05-18T05:32:34.6688476Z 
2022-05-18T05:32:34.6688613Z Generating XML reports...
2022-05-18T05:32:34.6746314Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053229.xml
2022-05-18T05:32:36.1163451Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:32:36.1178976Z 
2022-05-18T05:32:36.1179119Z Running tests...
2022-05-18T05:32:36.1179915Z ----------------------------------------------------------------------
2022-05-18T05:32:37.7660978Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:32:37.8085581Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87635
2022-05-18T05:32:37.8211865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87636
2022-05-18T05:32:39.0518044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:32:39.0518601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:32:39.0519393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:39.0520100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:39.0527937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:32:39.0528459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:32:39.0636556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:32:39.0637078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:32:39.0637753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:32:39.0638452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:32:39.3262977Z ok (3.208s)
2022-05-18T05:32:39.3263201Z 
2022-05-18T05:32:39.3263553Z ----------------------------------------------------------------------
2022-05-18T05:32:39.3263895Z Ran 1 test in 3.208s
2022-05-18T05:32:39.3264070Z 
2022-05-18T05:32:39.3264165Z OK
2022-05-18T05:32:39.3264320Z 
2022-05-18T05:32:39.3264453Z Generating XML reports...
2022-05-18T05:32:39.3320629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053236.xml
2022-05-18T05:32:40.7531489Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:32:40.7547000Z 
2022-05-18T05:32:40.7547261Z Running tests...
2022-05-18T05:32:40.7547698Z ----------------------------------------------------------------------
2022-05-18T05:32:42.3981823Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:32:42.4389311Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87756
2022-05-18T05:32:42.4510693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87757
2022-05-18T05:32:43.6108642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:32:43.6109228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:32:43.6110023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:43.6110745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:43.6117915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:32:43.6118417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:32:43.6225608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:32:43.6226137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:32:43.6226886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:32:43.6227576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:32:43.9559585Z ok (3.201s)
2022-05-18T05:32:43.9559800Z 
2022-05-18T05:32:43.9560173Z ----------------------------------------------------------------------
2022-05-18T05:32:43.9560522Z Ran 1 test in 3.201s
2022-05-18T05:32:43.9560696Z 
2022-05-18T05:32:43.9560793Z OK
2022-05-18T05:32:43.9560931Z 
2022-05-18T05:32:43.9561067Z Generating XML reports...
2022-05-18T05:32:43.9618441Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053240.xml
2022-05-18T05:32:45.3915569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:32:45.3930813Z 
2022-05-18T05:32:45.3931018Z Running tests...
2022-05-18T05:32:45.3931685Z ----------------------------------------------------------------------
2022-05-18T05:32:47.0419782Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:32:47.0825692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87877
2022-05-18T05:32:47.0948810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87878
2022-05-18T05:32:48.2970804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:32:48.2971380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:32:48.2972174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:48.2972878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:48.2980337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:32:48.2980850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:32:49.6172997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ovdfgcx
2022-05-18T05:32:49.6173906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ovdfgcx/_remote_module_non_scriptable.py
2022-05-18T05:32:49.6596364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvcrtzw9j
2022-05-18T05:32:49.6598707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvcrtzw9j/_remote_module_non_scriptable.py
2022-05-18T05:32:49.9755608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:32:49.9756180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:32:49.9842867Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:32:49.9844491Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:32:50.3029765Z ok (4.910s)
2022-05-18T05:32:50.3029988Z 
2022-05-18T05:32:50.3030399Z ----------------------------------------------------------------------
2022-05-18T05:32:50.3030716Z Ran 1 test in 4.910s
2022-05-18T05:32:50.3030883Z 
2022-05-18T05:32:50.3030993Z OK
2022-05-18T05:32:50.3031133Z 
2022-05-18T05:32:50.3031266Z Generating XML reports...
2022-05-18T05:32:50.3090172Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053245.xml
2022-05-18T05:32:51.7391920Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:32:51.7406009Z 
2022-05-18T05:32:51.7406416Z Running tests...
2022-05-18T05:32:51.7406909Z ----------------------------------------------------------------------
2022-05-18T05:32:53.3564741Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:32:53.3960875Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87998
2022-05-18T05:32:53.4082532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87999
2022-05-18T05:32:54.5816516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:32:54.5817100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:32:54.5817888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:54.5818589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:32:54.5928469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:32:54.6831525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:32:55.8919082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvctrm3b3
2022-05-18T05:32:55.8919690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvctrm3b3/_remote_module_non_scriptable.py
2022-05-18T05:32:55.9949236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_vb9zrr
2022-05-18T05:32:55.9950320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_vb9zrr/_remote_module_non_scriptable.py
2022-05-18T05:32:56.3008126Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:32:56.3212418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:32:56.3212911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:32:56.6162662Z ok (4.875s)
2022-05-18T05:32:56.6162898Z 
2022-05-18T05:32:56.6163283Z ----------------------------------------------------------------------
2022-05-18T05:32:56.6163600Z Ran 1 test in 4.876s
2022-05-18T05:32:56.6163770Z 
2022-05-18T05:32:56.6163863Z OK
2022-05-18T05:32:56.6163998Z 
2022-05-18T05:32:56.6164137Z Generating XML reports...
2022-05-18T05:32:56.6220997Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053251.xml
2022-05-18T05:32:58.0376354Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:32:58.0390622Z 
2022-05-18T05:32:58.0390915Z Running tests...
2022-05-18T05:32:58.0391363Z ----------------------------------------------------------------------
2022-05-18T05:32:59.6585092Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:32:59.6988607Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88119
2022-05-18T05:32:59.7112898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88120
2022-05-18T05:33:00.8633887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:33:00.8634911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:33:00.8636404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:00.8637797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:00.8644065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:33:00.8644809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:33:01.0161602Z ok (2.977s)
2022-05-18T05:33:01.0161828Z 
2022-05-18T05:33:01.0162226Z ----------------------------------------------------------------------
2022-05-18T05:33:01.0162589Z Ran 1 test in 2.977s
2022-05-18T05:33:01.0162736Z 
2022-05-18T05:33:01.0162829Z OK
2022-05-18T05:33:01.0162964Z 
2022-05-18T05:33:01.0163102Z Generating XML reports...
2022-05-18T05:33:01.0220074Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053258.xml
2022-05-18T05:33:02.4544676Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:33:02.4560362Z 
2022-05-18T05:33:02.4560790Z Running tests...
2022-05-18T05:33:02.4561293Z ----------------------------------------------------------------------
2022-05-18T05:33:04.1145172Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:33:04.1552316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88234
2022-05-18T05:33:04.1673298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88235
2022-05-18T05:33:05.3961977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:33:05.3962548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:33:05.3963326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:05.3964031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:05.4070551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:33:05.4973558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:33:05.6721546Z ok (3.216s)
2022-05-18T05:33:05.6721739Z 
2022-05-18T05:33:05.6722125Z ----------------------------------------------------------------------
2022-05-18T05:33:05.6722456Z Ran 1 test in 3.216s
2022-05-18T05:33:05.6722638Z 
2022-05-18T05:33:05.6722730Z OK
2022-05-18T05:33:05.6722851Z 
2022-05-18T05:33:05.6722979Z Generating XML reports...
2022-05-18T05:33:05.6780759Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053302.xml
2022-05-18T05:33:07.0883748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:33:07.0898530Z 
2022-05-18T05:33:07.0899024Z Running tests...
2022-05-18T05:33:07.0899515Z ----------------------------------------------------------------------
2022-05-18T05:33:08.7067764Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:33:08.7466830Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88353
2022-05-18T05:33:08.7589641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88354
2022-05-18T05:33:09.9792538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:33:09.9793120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:33:09.9793920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:09.9794628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:09.9903571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:33:10.0804194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:33:10.2638807Z ok (3.174s)
2022-05-18T05:33:10.2639027Z 
2022-05-18T05:33:10.2639949Z ----------------------------------------------------------------------
2022-05-18T05:33:10.2640334Z Ran 1 test in 3.174s
2022-05-18T05:33:10.2640505Z 
2022-05-18T05:33:10.2640596Z OK
2022-05-18T05:33:10.2640751Z 
2022-05-18T05:33:10.2640884Z Generating XML reports...
2022-05-18T05:33:10.2697282Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053307.xml
2022-05-18T05:33:11.6642479Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:33:11.6658008Z 
2022-05-18T05:33:11.6658577Z Running tests...
2022-05-18T05:33:11.6659151Z ----------------------------------------------------------------------
2022-05-18T05:33:11.6679701Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T05:33:11.6680006Z 
2022-05-18T05:33:11.6680296Z ----------------------------------------------------------------------
2022-05-18T05:33:11.6680621Z Ran 1 test in 0.002s
2022-05-18T05:33:11.6680790Z 
2022-05-18T05:33:11.6680898Z OK (skipped=1)
2022-05-18T05:33:11.6681052Z 
2022-05-18T05:33:11.6681175Z Generating XML reports...
2022-05-18T05:33:11.6724099Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053311.xml
2022-05-18T05:33:12.9080419Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:33:12.9095899Z 
2022-05-18T05:33:12.9096239Z Running tests...
2022-05-18T05:33:12.9096676Z ----------------------------------------------------------------------
2022-05-18T05:33:14.5740073Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:33:14.6147393Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88503
2022-05-18T05:33:14.6268514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88504
2022-05-18T05:33:15.8283087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:33:15.8283671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:33:15.8284465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:15.8285171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:15.8391794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:33:15.9295054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:33:15.9502578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:33:15.9503080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:33:15.9503821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:33:15.9504699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:33:16.1317854Z ok (3.222s)
2022-05-18T05:33:16.1318043Z 
2022-05-18T05:33:16.1318413Z ----------------------------------------------------------------------
2022-05-18T05:33:16.1318765Z Ran 1 test in 3.222s
2022-05-18T05:33:16.1318930Z 
2022-05-18T05:33:16.1319022Z OK
2022-05-18T05:33:16.1319156Z 
2022-05-18T05:33:16.1319277Z Generating XML reports...
2022-05-18T05:33:16.1379109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053312.xml
2022-05-18T05:33:17.5805906Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:33:17.5820890Z 
2022-05-18T05:33:17.5821197Z Running tests...
2022-05-18T05:33:17.5821929Z ----------------------------------------------------------------------
2022-05-18T05:33:19.2508189Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:33:19.2913279Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88628
2022-05-18T05:33:19.3034675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88629
2022-05-18T05:33:20.4670739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:33:20.4671293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:33:20.4672072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:20.4672772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:20.4679030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:33:20.4680324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:33:20.6080398Z skip: Skipped due to small world size. (3.026s)
2022-05-18T05:33:20.6080682Z 
2022-05-18T05:33:20.6081395Z ----------------------------------------------------------------------
2022-05-18T05:33:20.6081773Z Ran 1 test in 3.026s
2022-05-18T05:33:20.6081945Z 
2022-05-18T05:33:20.6082071Z OK (skipped=1)
2022-05-18T05:33:20.6082234Z 
2022-05-18T05:33:20.6082343Z Generating XML reports...
2022-05-18T05:33:20.6140387Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053317.xml
2022-05-18T05:33:22.0380806Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:33:22.0395698Z 
2022-05-18T05:33:22.0396028Z Running tests...
2022-05-18T05:33:22.0396486Z ----------------------------------------------------------------------
2022-05-18T05:33:23.7102489Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:33:23.7510520Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88743
2022-05-18T05:33:23.7629579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88744
2022-05-18T05:33:24.9391797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:33:24.9392363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:33:24.9393159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:24.9393862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:24.9500104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:33:25.0403431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:33:25.2680359Z ok (3.228s)
2022-05-18T05:33:25.2680843Z 
2022-05-18T05:33:25.2681555Z ----------------------------------------------------------------------
2022-05-18T05:33:25.2681934Z Ran 1 test in 3.228s
2022-05-18T05:33:25.2682108Z 
2022-05-18T05:33:25.2682184Z OK
2022-05-18T05:33:25.2682319Z 
2022-05-18T05:33:25.2682453Z Generating XML reports...
2022-05-18T05:33:25.2749021Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053322.xml
2022-05-18T05:33:26.7092150Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:33:26.7106862Z 
2022-05-18T05:33:26.7107283Z Running tests...
2022-05-18T05:33:26.7107718Z ----------------------------------------------------------------------
2022-05-18T05:33:28.3296939Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:33:28.3698231Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88858
2022-05-18T05:33:28.3822256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88859
2022-05-18T05:33:29.5508061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:33:29.5508666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:33:29.5509472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:29.5510176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:29.5617127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:33:29.6519916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:33:29.6836221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:33:29.6938049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:33:29.6938761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:33:29.6939460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:33:29.7185060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:33:29.7185581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:33:29.7186272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:33:29.7186969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:33:29.7409600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T05:33:29.7410396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T05:33:29.7411072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T05:33:29.7411760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T05:33:29.9872838Z ok (3.276s)
2022-05-18T05:33:29.9873053Z 
2022-05-18T05:33:29.9873432Z ----------------------------------------------------------------------
2022-05-18T05:33:29.9873751Z Ran 1 test in 3.277s
2022-05-18T05:33:29.9873914Z 
2022-05-18T05:33:29.9874031Z OK
2022-05-18T05:33:29.9874171Z 
2022-05-18T05:33:29.9874305Z Generating XML reports...
2022-05-18T05:33:29.9933023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053326.xml
2022-05-18T05:33:31.4440684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:33:31.4456402Z 
2022-05-18T05:33:31.4456816Z Running tests...
2022-05-18T05:33:31.4457350Z ----------------------------------------------------------------------
2022-05-18T05:33:33.0922931Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:33:33.1323558Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88997
2022-05-18T05:33:33.1442899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88998
2022-05-18T05:33:34.3663257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:33:34.3663847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:33:34.3664662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:34.3665357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:34.3672145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:33:34.3672642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:33:34.3779931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:33:34.3780434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:33:34.3781138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:33:34.3781836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:33:34.6491631Z ok (3.203s)
2022-05-18T05:33:34.6491857Z 
2022-05-18T05:33:34.6492219Z ----------------------------------------------------------------------
2022-05-18T05:33:34.6492575Z Ran 1 test in 3.203s
2022-05-18T05:33:34.6492745Z 
2022-05-18T05:33:34.6492845Z OK
2022-05-18T05:33:34.6492985Z 
2022-05-18T05:33:34.6493118Z Generating XML reports...
2022-05-18T05:33:34.6550014Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053331.xml
2022-05-18T05:33:36.0822920Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:33:36.0837775Z 
2022-05-18T05:33:36.0838263Z Running tests...
2022-05-18T05:33:36.0838822Z ----------------------------------------------------------------------
2022-05-18T05:33:37.7368307Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:33:37.7775557Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89118
2022-05-18T05:33:37.7896799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89119
2022-05-18T05:33:38.9761573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:33:38.9762384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:33:38.9763194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:38.9763901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:38.9771185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:33:38.9771684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:33:39.1945754Z ok (3.110s)
2022-05-18T05:33:39.1946078Z 
2022-05-18T05:33:39.1946839Z ----------------------------------------------------------------------
2022-05-18T05:33:39.1947509Z Ran 1 test in 3.111s
2022-05-18T05:33:39.1947659Z 
2022-05-18T05:33:39.1947757Z OK
2022-05-18T05:33:39.1947895Z 
2022-05-18T05:33:39.1948025Z Generating XML reports...
2022-05-18T05:33:39.2004270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053336.xml
2022-05-18T05:33:40.6269808Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:33:40.6285708Z 
2022-05-18T05:33:40.6286012Z Running tests...
2022-05-18T05:33:40.6286459Z ----------------------------------------------------------------------
2022-05-18T05:33:42.3041247Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:33:42.3450513Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89233
2022-05-18T05:33:42.3572023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89234
2022-05-18T05:33:43.5226286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:33:43.5226872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:33:43.5227814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:43.5228529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:43.5334857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:33:43.6239135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:33:43.9626407Z ok (3.334s)
2022-05-18T05:33:43.9626650Z 
2022-05-18T05:33:43.9627287Z ----------------------------------------------------------------------
2022-05-18T05:33:43.9627965Z Ran 1 test in 3.334s
2022-05-18T05:33:43.9628221Z 
2022-05-18T05:33:43.9628397Z OK
2022-05-18T05:33:43.9628617Z 
2022-05-18T05:33:43.9628761Z Generating XML reports...
2022-05-18T05:33:43.9685222Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053340.xml
2022-05-18T05:33:45.4147283Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:33:45.4161631Z 
2022-05-18T05:33:45.4161778Z Running tests...
2022-05-18T05:33:45.4162628Z ----------------------------------------------------------------------
2022-05-18T05:33:47.0634206Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:33:47.1042209Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89348
2022-05-18T05:33:47.1165139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89349
2022-05-18T05:33:48.3347668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:33:48.3348232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:33:48.3349025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:48.3349732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:48.3456822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:33:48.4362601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:33:48.4471008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:33:48.4471742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:33:48.4472455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:33:48.4473134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:33:48.7218786Z ok (3.305s)
2022-05-18T05:33:48.7219124Z 
2022-05-18T05:33:48.7219511Z ----------------------------------------------------------------------
2022-05-18T05:33:48.7219854Z Ran 1 test in 3.306s
2022-05-18T05:33:48.7220021Z 
2022-05-18T05:33:48.7220115Z OK
2022-05-18T05:33:48.7220233Z 
2022-05-18T05:33:48.7220365Z Generating XML reports...
2022-05-18T05:33:48.7278886Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053345.xml
2022-05-18T05:33:50.1766567Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:33:50.1782779Z 
2022-05-18T05:33:50.1783424Z Running tests...
2022-05-18T05:33:50.1783929Z ----------------------------------------------------------------------
2022-05-18T05:33:51.8408197Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:33:51.8801844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89469
2022-05-18T05:33:51.8920380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89470
2022-05-18T05:33:53.0806000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:33:53.0806564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:33:53.0807355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:53.0808055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:53.0914952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:33:53.1817708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:33:53.2025510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:33:53.2026315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:33:53.2027033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:33:53.2027747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:33:53.3972916Z ok (3.219s)
2022-05-18T05:33:53.3973250Z 
2022-05-18T05:33:53.3973886Z ----------------------------------------------------------------------
2022-05-18T05:33:53.3974262Z Ran 1 test in 3.219s
2022-05-18T05:33:53.3974431Z 
2022-05-18T05:33:53.3974525Z OK
2022-05-18T05:33:53.3975351Z 
2022-05-18T05:33:53.3975722Z Generating XML reports...
2022-05-18T05:33:53.4030883Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053350.xml
2022-05-18T05:33:54.8228797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:33:54.8243776Z 
2022-05-18T05:33:54.8244083Z Running tests...
2022-05-18T05:33:54.8244536Z ----------------------------------------------------------------------
2022-05-18T05:33:56.4768110Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:33:56.5174039Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89590
2022-05-18T05:33:56.5295553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89591
2022-05-18T05:33:57.7100053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:33:57.7100619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:33:57.7101427Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:57.7102131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:33:57.7211545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:33:57.8116233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:33:59.0277730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdzyem7q
2022-05-18T05:33:59.0278365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdzyem7q/_remote_module_non_scriptable.py
2022-05-18T05:33:59.1355066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0u8thrl2
2022-05-18T05:33:59.1356188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0u8thrl2/_remote_module_non_scriptable.py
2022-05-18T05:33:59.7375715Z ok (4.913s)
2022-05-18T05:33:59.7375974Z 
2022-05-18T05:33:59.7376391Z ----------------------------------------------------------------------
2022-05-18T05:33:59.7376733Z Ran 1 test in 4.913s
2022-05-18T05:33:59.7376897Z 
2022-05-18T05:33:59.7376999Z OK
2022-05-18T05:33:59.7377117Z 
2022-05-18T05:33:59.7377248Z Generating XML reports...
2022-05-18T05:33:59.7434999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053354.xml
2022-05-18T05:34:01.2064599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:01.2080629Z 
2022-05-18T05:34:01.2081007Z Running tests...
2022-05-18T05:34:01.2081527Z ----------------------------------------------------------------------
2022-05-18T05:34:02.8823091Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:34:02.9228651Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89711
2022-05-18T05:34:02.9352183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89712
2022-05-18T05:34:04.0911057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:34:04.0911614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:34:04.0912410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:04.0913140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:04.0920314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:34:04.0920798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:34:04.3399780Z ok (3.132s)
2022-05-18T05:34:04.3400007Z 
2022-05-18T05:34:04.3400388Z ----------------------------------------------------------------------
2022-05-18T05:34:04.3400719Z Ran 1 test in 3.132s
2022-05-18T05:34:04.3400886Z 
2022-05-18T05:34:04.3400985Z OK
2022-05-18T05:34:04.3401120Z 
2022-05-18T05:34:04.3401256Z Generating XML reports...
2022-05-18T05:34:04.3459021Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053401.xml
2022-05-18T05:34:05.7757642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:05.7773299Z 
2022-05-18T05:34:05.7773796Z Running tests...
2022-05-18T05:34:05.7774271Z ----------------------------------------------------------------------
2022-05-18T05:34:07.4201756Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:34:07.4612297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89826
2022-05-18T05:34:07.4733125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89827
2022-05-18T05:34:08.6520296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:34:08.6520855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:34:08.6521650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:08.6522587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:08.6629487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:34:08.7532353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:34:08.9783663Z ok (3.201s)
2022-05-18T05:34:08.9784027Z 
2022-05-18T05:34:08.9784678Z ----------------------------------------------------------------------
2022-05-18T05:34:08.9785298Z Ran 1 test in 3.201s
2022-05-18T05:34:08.9785604Z 
2022-05-18T05:34:08.9785771Z OK
2022-05-18T05:34:08.9786026Z 
2022-05-18T05:34:08.9786279Z Generating XML reports...
2022-05-18T05:34:08.9843957Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053405.xml
2022-05-18T05:34:10.4310320Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:10.4325226Z 
2022-05-18T05:34:10.4325484Z Running tests...
2022-05-18T05:34:10.4325926Z ----------------------------------------------------------------------
2022-05-18T05:34:12.0957943Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:34:12.1363536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89941
2022-05-18T05:34:12.1484522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89942
2022-05-18T05:34:13.3515099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:34:13.3515677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:34:13.3516468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:13.3517177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:13.3524451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:34:13.3524984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:34:13.6535143Z ok (3.221s)
2022-05-18T05:34:13.6535370Z 
2022-05-18T05:34:13.6535761Z ----------------------------------------------------------------------
2022-05-18T05:34:13.6536085Z Ran 1 test in 3.221s
2022-05-18T05:34:13.6536249Z 
2022-05-18T05:34:13.6536343Z OK
2022-05-18T05:34:13.6536483Z 
2022-05-18T05:34:13.6536621Z Generating XML reports...
2022-05-18T05:34:13.6592625Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053410.xml
2022-05-18T05:34:15.0875374Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:15.0890828Z 
2022-05-18T05:34:15.0891276Z Running tests...
2022-05-18T05:34:15.0891786Z ----------------------------------------------------------------------
2022-05-18T05:34:16.7430827Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:34:16.7834229Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90060
2022-05-18T05:34:16.7953998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90061
2022-05-18T05:34:17.9632670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:34:17.9633226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:34:17.9634026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:17.9634730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:17.9741784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:34:18.0644018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:34:18.4004564Z ok (3.311s)
2022-05-18T05:34:18.4008003Z 
2022-05-18T05:34:18.4008792Z ----------------------------------------------------------------------
2022-05-18T05:34:18.4009397Z Ran 1 test in 3.311s
2022-05-18T05:34:18.4010037Z 
2022-05-18T05:34:18.4010206Z OK
2022-05-18T05:34:18.4010437Z 
2022-05-18T05:34:18.4010652Z Generating XML reports...
2022-05-18T05:34:18.4069922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053415.xml
2022-05-18T05:34:19.8566936Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:19.8583227Z 
2022-05-18T05:34:19.8583499Z Running tests...
2022-05-18T05:34:19.8584152Z ----------------------------------------------------------------------
2022-05-18T05:34:19.8606076Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T05:34:19.8606813Z 
2022-05-18T05:34:19.8607412Z ----------------------------------------------------------------------
2022-05-18T05:34:19.8608096Z Ran 1 test in 0.002s
2022-05-18T05:34:19.8608242Z 
2022-05-18T05:34:19.8608350Z OK (skipped=1)
2022-05-18T05:34:19.8608505Z 
2022-05-18T05:34:19.8608629Z Generating XML reports...
2022-05-18T05:34:19.8650863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053419.xml
2022-05-18T05:34:21.1412306Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:21.1428406Z 
2022-05-18T05:34:21.1428654Z Running tests...
2022-05-18T05:34:21.1429086Z ----------------------------------------------------------------------
2022-05-18T05:34:21.1450763Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T05:34:21.1451144Z 
2022-05-18T05:34:21.1451428Z ----------------------------------------------------------------------
2022-05-18T05:34:21.1451767Z Ran 1 test in 0.002s
2022-05-18T05:34:21.1451929Z 
2022-05-18T05:34:21.1452044Z OK (skipped=1)
2022-05-18T05:34:21.1452206Z 
2022-05-18T05:34:21.1452314Z Generating XML reports...
2022-05-18T05:34:21.1495717Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053421.xml
2022-05-18T05:34:22.4287663Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:22.4304355Z 
2022-05-18T05:34:22.4304812Z Running tests...
2022-05-18T05:34:22.4305323Z ----------------------------------------------------------------------
2022-05-18T05:34:24.0888456Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:34:24.1294119Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90249
2022-05-18T05:34:24.1414556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90250
2022-05-18T05:34:25.3577507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:34:25.3578087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:34:25.3578892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:25.3579586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:25.3686389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:34:25.4592121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:34:25.6466953Z skip: Skipped due to small world size. (3.216s)
2022-05-18T05:34:25.6467280Z 
2022-05-18T05:34:25.6467786Z ----------------------------------------------------------------------
2022-05-18T05:34:25.6468280Z Ran 1 test in 3.216s
2022-05-18T05:34:25.6468452Z 
2022-05-18T05:34:25.6468563Z OK (skipped=1)
2022-05-18T05:34:25.6469030Z 
2022-05-18T05:34:25.6469399Z Generating XML reports...
2022-05-18T05:34:25.6524000Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053422.xml
2022-05-18T05:34:27.0747901Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:27.0763597Z 
2022-05-18T05:34:27.0764048Z Running tests...
2022-05-18T05:34:27.0764550Z ----------------------------------------------------------------------
2022-05-18T05:34:28.7288326Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:34:28.7684466Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90364
2022-05-18T05:34:28.7801385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90365
2022-05-18T05:34:29.9727831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:34:29.9728395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:34:29.9729194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:29.9730143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:29.9836424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:34:30.0739661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:34:32.0881560Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T05:34:32.3887776Z ok (5.312s)
2022-05-18T05:34:32.3887983Z 
2022-05-18T05:34:32.3888375Z ----------------------------------------------------------------------
2022-05-18T05:34:32.3888738Z Ran 1 test in 5.312s
2022-05-18T05:34:32.3888907Z 
2022-05-18T05:34:32.3889008Z OK
2022-05-18T05:34:32.3889147Z 
2022-05-18T05:34:32.3889263Z Generating XML reports...
2022-05-18T05:34:32.3945996Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053427.xml
2022-05-18T05:34:33.8351138Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:33.8366242Z 
2022-05-18T05:34:33.8366651Z Running tests...
2022-05-18T05:34:33.8367174Z ----------------------------------------------------------------------
2022-05-18T05:34:35.4996053Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:34:35.5402426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90479
2022-05-18T05:34:35.5523921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90480
2022-05-18T05:34:36.7951965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:34:36.7952555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:34:36.7953346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:36.7954047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:36.7961739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:34:36.7962475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:34:36.8068858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:34:36.8069380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:34:36.8070077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:34:36.8070782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:34:36.8073212Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-05-18T05:34:36.8073638Z No ranks successfully processed in monitoredBarrier.
2022-05-18T05:34:36.8103550Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-05-18T05:34:37.0573981Z ok (3.220s)
2022-05-18T05:34:37.0574206Z 
2022-05-18T05:34:37.0574626Z ----------------------------------------------------------------------
2022-05-18T05:34:37.0574947Z Ran 1 test in 3.221s
2022-05-18T05:34:37.0575113Z 
2022-05-18T05:34:37.0575214Z OK
2022-05-18T05:34:37.0575349Z 
2022-05-18T05:34:37.0575748Z Generating XML reports...
2022-05-18T05:34:37.0632208Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053433.xml
2022-05-18T05:34:38.4695998Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:38.4710539Z 
2022-05-18T05:34:38.4710797Z Running tests...
2022-05-18T05:34:38.4711233Z ----------------------------------------------------------------------
2022-05-18T05:34:40.0945834Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:34:40.1343546Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90600
2022-05-18T05:34:40.1462594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90601
2022-05-18T05:34:41.3189804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:34:41.3190367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:34:41.3191201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:41.3191909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:41.3298601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:34:41.4202641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:34:41.4313731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:34:41.4314255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:34:41.4314952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:34:41.4315642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:34:41.5323223Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-05-18T05:34:41.7513773Z ok (3.280s)
2022-05-18T05:34:41.7514109Z 
2022-05-18T05:34:41.7514664Z ----------------------------------------------------------------------
2022-05-18T05:34:41.7515139Z Ran 1 test in 3.280s
2022-05-18T05:34:41.7515347Z 
2022-05-18T05:34:41.7515441Z OK
2022-05-18T05:34:41.7515597Z 
2022-05-18T05:34:41.7515712Z Generating XML reports...
2022-05-18T05:34:41.7572421Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053438.xml
2022-05-18T05:34:43.2035781Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:43.2050112Z 
2022-05-18T05:34:43.2050707Z Running tests...
2022-05-18T05:34:43.2051394Z ----------------------------------------------------------------------
2022-05-18T05:34:44.8573961Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:34:44.8967804Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90721
2022-05-18T05:34:44.9087416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90722
2022-05-18T05:34:46.0690710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:34:46.0691332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:34:46.0692363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:46.0693259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:34:46.0699206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:34:46.0699926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:34:46.2135296Z skip: Skipped due to small world size. (3.008s)
2022-05-18T05:34:46.2135561Z 
2022-05-18T05:34:46.2135952Z ----------------------------------------------------------------------
2022-05-18T05:34:46.2136295Z Ran 1 test in 3.009s
2022-05-18T05:34:46.2136443Z 
2022-05-18T05:34:46.2136555Z OK (skipped=1)
2022-05-18T05:34:46.2136708Z 
2022-05-18T05:34:46.2136836Z Generating XML reports...
2022-05-18T05:34:46.2193782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053443.xml
2022-05-18T05:34:47.6491787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:47.6507989Z 
2022-05-18T05:34:47.6508447Z Running tests...
2022-05-18T05:34:47.6508951Z ----------------------------------------------------------------------
2022-05-18T05:34:47.6537636Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T05:34:47.6537979Z 
2022-05-18T05:34:47.6538247Z ----------------------------------------------------------------------
2022-05-18T05:34:47.6538580Z Ran 1 test in 0.003s
2022-05-18T05:34:47.6538743Z 
2022-05-18T05:34:47.6538853Z OK (skipped=1)
2022-05-18T05:34:47.6540867Z 
2022-05-18T05:34:47.6541339Z Generating XML reports...
2022-05-18T05:34:47.6585263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053447.xml
2022-05-18T05:34:48.8971262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:48.8987205Z 
2022-05-18T05:34:48.8987522Z Running tests...
2022-05-18T05:34:48.8987963Z ----------------------------------------------------------------------
2022-05-18T05:34:48.9019013Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T05:34:48.9019675Z 
2022-05-18T05:34:48.9020096Z ----------------------------------------------------------------------
2022-05-18T05:34:48.9020414Z Ran 1 test in 0.003s
2022-05-18T05:34:48.9020577Z 
2022-05-18T05:34:48.9020687Z OK (skipped=1)
2022-05-18T05:34:48.9020850Z 
2022-05-18T05:34:48.9021512Z Generating XML reports...
2022-05-18T05:34:48.9066246Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053448.xml
2022-05-18T05:34:50.1796893Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:50.1813219Z 
2022-05-18T05:34:50.1813757Z Running tests...
2022-05-18T05:34:50.1814220Z ----------------------------------------------------------------------
2022-05-18T05:34:50.1840929Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T05:34:50.1841288Z 
2022-05-18T05:34:50.1841538Z ----------------------------------------------------------------------
2022-05-18T05:34:50.1841864Z Ran 1 test in 0.003s
2022-05-18T05:34:50.1842065Z 
2022-05-18T05:34:50.1842176Z OK (skipped=1)
2022-05-18T05:34:50.1842335Z 
2022-05-18T05:34:50.1842460Z Generating XML reports...
2022-05-18T05:34:50.1885774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053450.xml
2022-05-18T05:34:51.4595425Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:51.4611984Z 
2022-05-18T05:34:51.4612407Z Running tests...
2022-05-18T05:34:51.4612906Z ----------------------------------------------------------------------
2022-05-18T05:34:51.4646949Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T05:34:51.4647582Z 
2022-05-18T05:34:51.4648071Z ----------------------------------------------------------------------
2022-05-18T05:34:51.4648423Z Ran 1 test in 0.004s
2022-05-18T05:34:51.4648591Z 
2022-05-18T05:34:51.4648699Z OK (skipped=1)
2022-05-18T05:34:51.4648853Z 
2022-05-18T05:34:51.4648959Z Generating XML reports...
2022-05-18T05:34:51.4691481Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053451.xml
2022-05-18T05:34:52.7211040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:52.7226222Z 
2022-05-18T05:34:52.7226707Z Running tests...
2022-05-18T05:34:52.7227197Z ----------------------------------------------------------------------
2022-05-18T05:34:52.7253250Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.003s)
2022-05-18T05:34:52.7253605Z 
2022-05-18T05:34:52.7253899Z ----------------------------------------------------------------------
2022-05-18T05:34:52.7254230Z Ran 1 test in 0.003s
2022-05-18T05:34:52.7254397Z 
2022-05-18T05:34:52.7254488Z OK (skipped=1)
2022-05-18T05:34:52.7254643Z 
2022-05-18T05:34:52.7254765Z Generating XML reports...
2022-05-18T05:34:52.7296134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053452.xml
2022-05-18T05:34:53.9977695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:53.9994847Z 
2022-05-18T05:34:53.9995242Z Running tests...
2022-05-18T05:34:53.9996106Z ----------------------------------------------------------------------
2022-05-18T05:34:54.0021799Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-05-18T05:34:54.0022410Z 
2022-05-18T05:34:54.0022988Z ----------------------------------------------------------------------
2022-05-18T05:34:54.0023628Z Ran 1 test in 0.003s
2022-05-18T05:34:54.0023943Z 
2022-05-18T05:34:54.0024137Z OK (skipped=1)
2022-05-18T05:34:54.0024424Z 
2022-05-18T05:34:54.0024662Z Generating XML reports...
2022-05-18T05:34:54.0068759Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053453.xml
2022-05-18T05:34:55.2787393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:55.2802288Z 
2022-05-18T05:34:55.2802781Z Running tests...
2022-05-18T05:34:55.2803290Z ----------------------------------------------------------------------
2022-05-18T05:34:55.2831068Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-05-18T05:34:55.2831671Z 
2022-05-18T05:34:55.2831994Z ----------------------------------------------------------------------
2022-05-18T05:34:55.2832333Z Ran 1 test in 0.003s
2022-05-18T05:34:55.2832497Z 
2022-05-18T05:34:55.2832602Z OK (skipped=1)
2022-05-18T05:34:55.2832760Z 
2022-05-18T05:34:55.2832883Z Generating XML reports...
2022-05-18T05:34:55.2874749Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053455.xml
2022-05-18T05:34:56.5455470Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:56.5470168Z 
2022-05-18T05:34:56.5470508Z Running tests...
2022-05-18T05:34:56.5470945Z ----------------------------------------------------------------------
2022-05-18T05:34:56.5494568Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T05:34:56.5494922Z 
2022-05-18T05:34:56.5495224Z ----------------------------------------------------------------------
2022-05-18T05:34:56.5495546Z Ran 1 test in 0.002s
2022-05-18T05:34:56.5495714Z 
2022-05-18T05:34:56.5495826Z OK (skipped=1)
2022-05-18T05:34:56.5496276Z 
2022-05-18T05:34:56.5496403Z Generating XML reports...
2022-05-18T05:34:56.5536819Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053456.xml
2022-05-18T05:34:57.7879494Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:34:57.7896231Z 
2022-05-18T05:34:57.7896580Z Running tests...
2022-05-18T05:34:57.7897024Z ----------------------------------------------------------------------
2022-05-18T05:34:59.4636802Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:34:59.5041974Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91116
2022-05-18T05:34:59.5163454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91117
2022-05-18T05:35:00.6780137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:35:00.6780711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:35:00.6781502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:00.6782208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:00.6888547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:35:00.7792206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:35:01.0214353Z ok (3.231s)
2022-05-18T05:35:01.0214578Z 
2022-05-18T05:35:01.0214995Z ----------------------------------------------------------------------
2022-05-18T05:35:01.0215320Z Ran 1 test in 3.232s
2022-05-18T05:35:01.0215486Z 
2022-05-18T05:35:01.0215579Z OK
2022-05-18T05:35:01.0215713Z 
2022-05-18T05:35:01.0215854Z Generating XML reports...
2022-05-18T05:35:01.0272343Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053457.xml
2022-05-18T05:35:02.4906245Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:35:02.4920865Z 
2022-05-18T05:35:02.4921015Z Running tests...
2022-05-18T05:35:02.4921744Z ----------------------------------------------------------------------
2022-05-18T05:35:04.1589086Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:35:04.1997206Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91231
2022-05-18T05:35:04.2120051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91232
2022-05-18T05:35:05.4618405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:35:05.4618935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:35:05.4619758Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:05.4620459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:05.4627425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:35:05.4628219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:35:05.6166285Z ok (3.124s)
2022-05-18T05:35:05.6166504Z 
2022-05-18T05:35:05.6166887Z ----------------------------------------------------------------------
2022-05-18T05:35:05.6167244Z Ran 1 test in 3.124s
2022-05-18T05:35:05.6167417Z 
2022-05-18T05:35:05.6167512Z OK
2022-05-18T05:35:05.6167630Z 
2022-05-18T05:35:05.6167763Z Generating XML reports...
2022-05-18T05:35:05.6225516Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053502.xml
2022-05-18T05:35:07.0495350Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:35:07.0510426Z 
2022-05-18T05:35:07.0510745Z Running tests...
2022-05-18T05:35:07.0511184Z ----------------------------------------------------------------------
2022-05-18T05:35:07.0533144Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T05:35:07.0533474Z 
2022-05-18T05:35:07.0533756Z ----------------------------------------------------------------------
2022-05-18T05:35:07.0534066Z Ran 1 test in 0.002s
2022-05-18T05:35:07.0534227Z 
2022-05-18T05:35:07.0534357Z OK (skipped=1)
2022-05-18T05:35:07.0534514Z 
2022-05-18T05:35:07.0534638Z Generating XML reports...
2022-05-18T05:35:07.0577190Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053507.xml
2022-05-18T05:35:08.3345897Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:35:08.3361249Z 
2022-05-18T05:35:08.3361711Z Running tests...
2022-05-18T05:35:08.3362205Z ----------------------------------------------------------------------
2022-05-18T05:35:08.3383335Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T05:35:08.3383697Z 
2022-05-18T05:35:08.3383975Z ----------------------------------------------------------------------
2022-05-18T05:35:08.3384312Z Ran 1 test in 0.002s
2022-05-18T05:35:08.3384476Z 
2022-05-18T05:35:08.3384585Z OK (skipped=1)
2022-05-18T05:35:08.3384743Z 
2022-05-18T05:35:08.3384863Z Generating XML reports...
2022-05-18T05:35:08.3427580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053508.xml
2022-05-18T05:35:09.6239320Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:35:09.6255358Z 
2022-05-18T05:35:09.6255680Z Running tests...
2022-05-18T05:35:09.6256132Z ----------------------------------------------------------------------
2022-05-18T05:35:11.2898213Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:35:11.3303534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91416
2022-05-18T05:35:11.3424573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91417
2022-05-18T05:35:12.5501205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:35:12.5502038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:35:12.5502850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:12.5503570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:12.5611098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:35:12.6516186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:35:13.8755111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64m6u0fr
2022-05-18T05:35:13.8755758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64m6u0fr/_remote_module_non_scriptable.py
2022-05-18T05:35:13.9680600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprs_i8842
2022-05-18T05:35:13.9681240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprs_i8842/_remote_module_non_scriptable.py
2022-05-18T05:35:14.6506331Z ok (5.025s)
2022-05-18T05:35:14.6506546Z 
2022-05-18T05:35:14.6507252Z ----------------------------------------------------------------------
2022-05-18T05:35:14.6507599Z Ran 1 test in 5.025s
2022-05-18T05:35:14.6507764Z 
2022-05-18T05:35:14.6507839Z OK
2022-05-18T05:35:14.6507974Z 
2022-05-18T05:35:14.6508108Z Generating XML reports...
2022-05-18T05:35:14.6565120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053509.xml
2022-05-18T05:35:16.1047912Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:35:16.1063555Z 
2022-05-18T05:35:16.1063812Z Running tests...
2022-05-18T05:35:16.1064229Z ----------------------------------------------------------------------
2022-05-18T05:35:17.7746358Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:35:17.8140747Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91537
2022-05-18T05:35:17.8258787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91538
2022-05-18T05:35:19.0652566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:35:19.0653111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:35:19.0653890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:19.0654578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:19.0662308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:35:19.0662823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:35:20.4338622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8pw33eea
2022-05-18T05:35:20.4339598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8pw33eea/_remote_module_non_scriptable.py
2022-05-18T05:35:20.4353179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7kaas_1t
2022-05-18T05:35:20.4356093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7kaas_1t/_remote_module_non_scriptable.py
2022-05-18T05:35:21.1342863Z ok (5.028s)
2022-05-18T05:35:21.1343095Z 
2022-05-18T05:35:21.1343462Z ----------------------------------------------------------------------
2022-05-18T05:35:21.1343803Z Ran 1 test in 5.028s
2022-05-18T05:35:21.1343969Z 
2022-05-18T05:35:21.1344069Z OK
2022-05-18T05:35:21.1344204Z 
2022-05-18T05:35:21.1344321Z Generating XML reports...
2022-05-18T05:35:21.1401776Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053516.xml
2022-05-18T05:35:22.5766999Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:35:22.5782226Z 
2022-05-18T05:35:22.5782457Z Running tests...
2022-05-18T05:35:22.5782898Z ----------------------------------------------------------------------
2022-05-18T05:35:24.2459639Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:35:24.2863987Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91658
2022-05-18T05:35:24.2984453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91659
2022-05-18T05:35:25.4770392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:35:25.4771012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:35:25.4771832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:25.4772544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:25.4879620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:35:25.5785296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:35:28.2079346Z ok (5.629s)
2022-05-18T05:35:28.2079732Z 
2022-05-18T05:35:28.2080410Z ----------------------------------------------------------------------
2022-05-18T05:35:28.2081041Z Ran 1 test in 5.630s
2022-05-18T05:35:28.2081322Z 
2022-05-18T05:35:28.2081483Z OK
2022-05-18T05:35:28.2081745Z 
2022-05-18T05:35:28.2081979Z Generating XML reports...
2022-05-18T05:35:28.2140874Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053522.xml
2022-05-18T05:35:29.6362648Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:35:29.6377392Z 
2022-05-18T05:35:29.6377799Z Running tests...
2022-05-18T05:35:29.6378302Z ----------------------------------------------------------------------
2022-05-18T05:35:31.2671439Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:35:31.3077110Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91776
2022-05-18T05:35:31.3196343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91777
2022-05-18T05:35:32.5196202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:35:32.5196753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:35:32.5197582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:32.5198291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:32.5304998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:35:32.6211221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:35:35.2289640Z ok (5.591s)
2022-05-18T05:35:35.2289853Z 
2022-05-18T05:35:35.2290527Z ----------------------------------------------------------------------
2022-05-18T05:35:35.2290853Z Ran 1 test in 5.591s
2022-05-18T05:35:35.2291023Z 
2022-05-18T05:35:35.2291119Z OK
2022-05-18T05:35:35.2291255Z 
2022-05-18T05:35:35.2291390Z Generating XML reports...
2022-05-18T05:35:35.2349991Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053529.xml
2022-05-18T05:35:36.6810114Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:35:36.6824757Z 
2022-05-18T05:35:36.6825265Z Running tests...
2022-05-18T05:35:36.6825779Z ----------------------------------------------------------------------
2022-05-18T05:35:38.3275222Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:35:38.3395075Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.657s)
2022-05-18T05:35:38.3395685Z 
2022-05-18T05:35:38.3395958Z ----------------------------------------------------------------------
2022-05-18T05:35:38.3396273Z Ran 1 test in 1.657s
2022-05-18T05:35:38.3396438Z 
2022-05-18T05:35:38.3396549Z OK (skipped=1)
2022-05-18T05:35:38.3396706Z 
2022-05-18T05:35:38.3396850Z Generating XML reports...
2022-05-18T05:35:38.3435625Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053536.xml
2022-05-18T05:35:39.7358525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:35:39.7373787Z 
2022-05-18T05:35:39.7374040Z Running tests...
2022-05-18T05:35:39.7374489Z ----------------------------------------------------------------------
2022-05-18T05:35:41.3955573Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:35:41.4070816Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.669s)
2022-05-18T05:35:41.4071411Z 
2022-05-18T05:35:41.4071708Z ----------------------------------------------------------------------
2022-05-18T05:35:41.4072038Z Ran 1 test in 1.670s
2022-05-18T05:35:41.4072202Z 
2022-05-18T05:35:41.4072308Z OK (skipped=1)
2022-05-18T05:35:41.4072456Z 
2022-05-18T05:35:41.4072579Z Generating XML reports...
2022-05-18T05:35:41.4109581Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053539.xml
2022-05-18T05:35:42.7987793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:35:42.8002888Z 
2022-05-18T05:35:42.8003187Z Running tests...
2022-05-18T05:35:42.8003616Z ----------------------------------------------------------------------
2022-05-18T05:35:44.4531101Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:35:44.4926413Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91966
2022-05-18T05:35:44.5045679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91967
2022-05-18T05:35:45.6740629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:35:45.6741255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:35:45.6742036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:45.6742741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:45.6850862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:35:45.7755687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:35:45.9093512Z skip: Need at least 4 CUDA devices (3.109s)
2022-05-18T05:35:45.9093911Z 
2022-05-18T05:35:45.9094536Z ----------------------------------------------------------------------
2022-05-18T05:35:45.9094898Z Ran 1 test in 3.109s
2022-05-18T05:35:45.9095062Z 
2022-05-18T05:35:45.9095171Z OK (skipped=1)
2022-05-18T05:35:45.9095341Z 
2022-05-18T05:35:45.9095468Z Generating XML reports...
2022-05-18T05:35:45.9152580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053542.xml
2022-05-18T05:35:47.3370421Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:35:47.3385515Z 
2022-05-18T05:35:47.3385724Z Running tests...
2022-05-18T05:35:47.3386151Z ----------------------------------------------------------------------
2022-05-18T05:35:49.0029799Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:35:49.0426220Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92081
2022-05-18T05:35:49.0547151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92082
2022-05-18T05:35:50.2630652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:35:50.2631463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:35:50.2632265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:50.2632957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:50.2640028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:35:50.2641199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:35:50.4593596Z skip: Need at least 4 CUDA devices (3.120s)
2022-05-18T05:35:50.4593830Z 
2022-05-18T05:35:50.4594228Z ----------------------------------------------------------------------
2022-05-18T05:35:50.4594565Z Ran 1 test in 3.121s
2022-05-18T05:35:50.4594728Z 
2022-05-18T05:35:50.4594847Z OK (skipped=1)
2022-05-18T05:35:50.4595004Z 
2022-05-18T05:35:50.4595112Z Generating XML reports...
2022-05-18T05:35:50.4652006Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053547.xml
2022-05-18T05:35:51.8941017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:35:51.8956577Z 
2022-05-18T05:35:51.8957046Z Running tests...
2022-05-18T05:35:51.8957545Z ----------------------------------------------------------------------
2022-05-18T05:35:53.5583067Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:35:53.5988287Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92196
2022-05-18T05:35:53.6108584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92197
2022-05-18T05:35:54.8213297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:35:54.8213888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:35:54.8214676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:54.8215377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:54.8223530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:35:54.8224071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:35:54.8432595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:35:54.8433313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:35:54.8434036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:35:54.8434743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:35:55.1160179Z ok (3.220s)
2022-05-18T05:35:55.1161240Z 
2022-05-18T05:35:55.1161650Z ----------------------------------------------------------------------
2022-05-18T05:35:55.1162001Z Ran 1 test in 3.220s
2022-05-18T05:35:55.1162160Z 
2022-05-18T05:35:55.1162253Z OK
2022-05-18T05:35:55.1162392Z 
2022-05-18T05:35:55.1162528Z Generating XML reports...
2022-05-18T05:35:55.1218589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053551.xml
2022-05-18T05:35:56.5503160Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:35:56.5518456Z 
2022-05-18T05:35:56.5518924Z Running tests...
2022-05-18T05:35:56.5519435Z ----------------------------------------------------------------------
2022-05-18T05:35:58.2103216Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:35:58.2511068Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92321
2022-05-18T05:35:58.2633582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92322
2022-05-18T05:35:59.4498484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:35:59.4499052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:35:59.4499842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:59.4500531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:35:59.4609493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:35:59.5512250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:35:59.5721459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:35:59.5721990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:35:59.5722728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:35:59.5723417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:35:59.7686315Z ok (3.217s)
2022-05-18T05:35:59.7686565Z 
2022-05-18T05:35:59.7686963Z ----------------------------------------------------------------------
2022-05-18T05:35:59.7687303Z Ran 1 test in 3.217s
2022-05-18T05:35:59.7687475Z 
2022-05-18T05:35:59.7687567Z OK
2022-05-18T05:35:59.7687711Z 
2022-05-18T05:35:59.7687828Z Generating XML reports...
2022-05-18T05:35:59.7746659Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053556.xml
2022-05-18T05:36:01.1997321Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:01.2012858Z 
2022-05-18T05:36:01.2013009Z Running tests...
2022-05-18T05:36:01.2013659Z ----------------------------------------------------------------------
2022-05-18T05:36:02.8743180Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:36:02.9152593Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92446
2022-05-18T05:36:02.9273997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92447
2022-05-18T05:36:04.1059748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:36:04.1060311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:36:04.1061104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:04.1061814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:04.1068702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:36:04.1069174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:36:04.1177500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:36:04.1178033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:36:04.1178733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:36:04.1179650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:36:04.3322635Z ok (3.131s)
2022-05-18T05:36:04.3323265Z 
2022-05-18T05:36:04.3324103Z ----------------------------------------------------------------------
2022-05-18T05:36:04.3324886Z Ran 1 test in 3.131s
2022-05-18T05:36:04.3325099Z 
2022-05-18T05:36:04.3325198Z OK
2022-05-18T05:36:04.3325337Z 
2022-05-18T05:36:04.3325454Z Generating XML reports...
2022-05-18T05:36:04.3383402Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053601.xml
2022-05-18T05:36:05.7629276Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:05.7644208Z 
2022-05-18T05:36:05.7644729Z Running tests...
2022-05-18T05:36:05.7645652Z ----------------------------------------------------------------------
2022-05-18T05:36:07.4239044Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:36:07.4634738Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92571
2022-05-18T05:36:07.4754116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92572
2022-05-18T05:36:08.6517751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:36:08.6518299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:36:08.6519092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:08.6519822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:08.6628231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:36:08.7530729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:36:08.7644935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:36:08.7645486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:36:08.7646232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:36:08.7646924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:36:08.9804726Z ok (3.216s)
2022-05-18T05:36:08.9804937Z 
2022-05-18T05:36:08.9805577Z ----------------------------------------------------------------------
2022-05-18T05:36:08.9805912Z Ran 1 test in 3.216s
2022-05-18T05:36:08.9806079Z 
2022-05-18T05:36:08.9806173Z OK
2022-05-18T05:36:08.9807314Z 
2022-05-18T05:36:08.9807675Z Generating XML reports...
2022-05-18T05:36:08.9873373Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053605.xml
2022-05-18T05:36:10.4184830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:10.4199807Z 
2022-05-18T05:36:10.4200077Z Running tests...
2022-05-18T05:36:10.4200524Z ----------------------------------------------------------------------
2022-05-18T05:36:12.0739787Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:36:12.1145966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92696
2022-05-18T05:36:12.1267552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92697
2022-05-18T05:36:13.3061122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:36:13.3061816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:36:13.3063084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:13.3064000Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:13.3070529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:36:13.3071033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:36:13.5316039Z skip: Skipped due to small world size. (3.111s)
2022-05-18T05:36:13.5316300Z 
2022-05-18T05:36:13.5316669Z ----------------------------------------------------------------------
2022-05-18T05:36:13.5317030Z Ran 1 test in 3.112s
2022-05-18T05:36:13.5317196Z 
2022-05-18T05:36:13.5317314Z OK (skipped=1)
2022-05-18T05:36:13.5317474Z 
2022-05-18T05:36:13.5317602Z Generating XML reports...
2022-05-18T05:36:13.5385511Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053610.xml
2022-05-18T05:36:14.9667212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:14.9682772Z 
2022-05-18T05:36:14.9683192Z Running tests...
2022-05-18T05:36:14.9683676Z ----------------------------------------------------------------------
2022-05-18T05:36:16.6354361Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:36:16.6762194Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92811
2022-05-18T05:36:16.6885036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92812
2022-05-18T05:36:17.9054645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:36:17.9055207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:36:17.9056023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:17.9056730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:17.9165889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:36:18.0069423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:36:18.1935126Z skip: Skipped due to small world size. (3.225s)
2022-05-18T05:36:18.1935370Z 
2022-05-18T05:36:18.1935718Z ----------------------------------------------------------------------
2022-05-18T05:36:18.1936255Z Ran 1 test in 3.225s
2022-05-18T05:36:18.1936445Z 
2022-05-18T05:36:18.1936565Z OK (skipped=1)
2022-05-18T05:36:18.1936724Z 
2022-05-18T05:36:18.1936854Z Generating XML reports...
2022-05-18T05:36:18.2004651Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053614.xml
2022-05-18T05:36:19.6317610Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:19.6332696Z 
2022-05-18T05:36:19.6332845Z Running tests...
2022-05-18T05:36:19.6333287Z ----------------------------------------------------------------------
2022-05-18T05:36:21.2896861Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:36:21.3301605Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92926
2022-05-18T05:36:21.3422297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92927
2022-05-18T05:36:22.5421076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:36:22.5421637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:36:22.5422713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:22.5423471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:22.5430345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:36:22.5430839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:36:22.7472875Z skip: Skipped due to small world size. (3.114s)
2022-05-18T05:36:22.7473133Z 
2022-05-18T05:36:22.7473501Z ----------------------------------------------------------------------
2022-05-18T05:36:22.7473840Z Ran 1 test in 3.114s
2022-05-18T05:36:22.7474022Z 
2022-05-18T05:36:22.7474135Z OK (skipped=1)
2022-05-18T05:36:22.7474274Z 
2022-05-18T05:36:22.7476871Z Generating XML reports...
2022-05-18T05:36:22.7531372Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053619.xml
2022-05-18T05:36:24.1359711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:24.1374740Z 
2022-05-18T05:36:24.1375314Z Running tests...
2022-05-18T05:36:24.1375818Z ----------------------------------------------------------------------
2022-05-18T05:36:25.8050740Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:36:25.8455346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93041
2022-05-18T05:36:25.8576696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93042
2022-05-18T05:36:27.0270781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:36:27.0271844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:36:27.0273227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:27.0274455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:27.0280024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:36:27.0281270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:36:27.2624281Z skip: Skipped due to small world size. (3.125s)
2022-05-18T05:36:27.2624521Z 
2022-05-18T05:36:27.2624901Z ----------------------------------------------------------------------
2022-05-18T05:36:27.2625242Z Ran 1 test in 3.125s
2022-05-18T05:36:27.2625415Z 
2022-05-18T05:36:27.2625685Z OK (skipped=1)
2022-05-18T05:36:27.2625838Z 
2022-05-18T05:36:27.2625971Z Generating XML reports...
2022-05-18T05:36:27.2683153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053624.xml
2022-05-18T05:36:28.6918274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:28.6933316Z 
2022-05-18T05:36:28.6933598Z Running tests...
2022-05-18T05:36:28.6934023Z ----------------------------------------------------------------------
2022-05-18T05:36:30.3484350Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:36:30.3888894Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93156
2022-05-18T05:36:30.4013527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93157
2022-05-18T05:36:31.5580799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:36:31.5581388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:36:31.5582192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:31.5583145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:31.5589203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:36:31.5590037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:36:31.8063834Z ok (3.113s)
2022-05-18T05:36:31.8064090Z 
2022-05-18T05:36:31.8064493Z ----------------------------------------------------------------------
2022-05-18T05:36:31.8064837Z Ran 1 test in 3.113s
2022-05-18T05:36:31.8065003Z 
2022-05-18T05:36:31.8065079Z OK
2022-05-18T05:36:31.8065215Z 
2022-05-18T05:36:31.8065366Z Generating XML reports...
2022-05-18T05:36:31.8122554Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053628.xml
2022-05-18T05:36:33.2487824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:33.2503252Z 
2022-05-18T05:36:33.2503672Z Running tests...
2022-05-18T05:36:33.2504162Z ----------------------------------------------------------------------
2022-05-18T05:36:34.9100179Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:36:34.9507301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93275
2022-05-18T05:36:34.9627750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93276
2022-05-18T05:36:36.1684720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:36:36.1685314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:36:36.1686102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:36.1686825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:36.1694509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:36:36.1695014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:36:36.3679927Z ok (3.117s)
2022-05-18T05:36:36.3680171Z 
2022-05-18T05:36:36.3680563Z ----------------------------------------------------------------------
2022-05-18T05:36:36.3680911Z Ran 1 test in 3.118s
2022-05-18T05:36:36.3681082Z 
2022-05-18T05:36:36.3681178Z OK
2022-05-18T05:36:36.3681294Z 
2022-05-18T05:36:36.3681440Z Generating XML reports...
2022-05-18T05:36:36.3739115Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053633.xml
2022-05-18T05:36:37.7555526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:37.7570668Z 
2022-05-18T05:36:37.7570865Z Running tests...
2022-05-18T05:36:37.7571312Z ----------------------------------------------------------------------
2022-05-18T05:36:37.7595853Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-05-18T05:36:37.7596190Z 
2022-05-18T05:36:37.7596471Z ----------------------------------------------------------------------
2022-05-18T05:36:37.7596807Z Ran 1 test in 0.003s
2022-05-18T05:36:37.7596975Z 
2022-05-18T05:36:37.7597085Z OK (skipped=1)
2022-05-18T05:36:37.7597242Z 
2022-05-18T05:36:37.7597348Z Generating XML reports...
2022-05-18T05:36:37.7639757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053637.xml
2022-05-18T05:36:38.9984572Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:38.9999766Z 
2022-05-18T05:36:39.0000153Z Running tests...
2022-05-18T05:36:39.0000864Z ----------------------------------------------------------------------
2022-05-18T05:36:40.6586728Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:36:40.6982485Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93429
2022-05-18T05:36:40.7101067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93430
2022-05-18T05:36:41.9203487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:36:41.9204095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:36:41.9204916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:41.9205628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:41.9212524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:36:41.9214493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:36:42.1150661Z ok (3.115s)
2022-05-18T05:36:42.1151017Z 
2022-05-18T05:36:42.1151408Z ----------------------------------------------------------------------
2022-05-18T05:36:42.1151732Z Ran 1 test in 3.115s
2022-05-18T05:36:42.1151901Z 
2022-05-18T05:36:42.1151996Z OK
2022-05-18T05:36:42.1152133Z 
2022-05-18T05:36:42.1152270Z Generating XML reports...
2022-05-18T05:36:42.1210128Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053638.xml
2022-05-18T05:36:43.5309282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:43.5323938Z 
2022-05-18T05:36:43.5324339Z Running tests...
2022-05-18T05:36:43.5324794Z ----------------------------------------------------------------------
2022-05-18T05:36:45.1350503Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:36:45.1746315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93548
2022-05-18T05:36:45.1867919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93549
2022-05-18T05:36:46.3876703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:36:46.3877279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:36:46.3878051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:46.3878965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:46.3886466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:36:46.3887537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:36:46.5916457Z ok (3.059s)
2022-05-18T05:36:46.5916655Z 
2022-05-18T05:36:46.5917054Z ----------------------------------------------------------------------
2022-05-18T05:36:46.5917395Z Ran 1 test in 3.059s
2022-05-18T05:36:46.5917560Z 
2022-05-18T05:36:46.5920099Z OK
2022-05-18T05:36:46.5920537Z 
2022-05-18T05:36:46.5920874Z Generating XML reports...
2022-05-18T05:36:46.5975060Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053643.xml
2022-05-18T05:36:48.0273365Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:48.0289110Z 
2022-05-18T05:36:48.0289329Z Running tests...
2022-05-18T05:36:48.0290069Z ----------------------------------------------------------------------
2022-05-18T05:36:48.0320278Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.003s)
2022-05-18T05:36:48.0321067Z 
2022-05-18T05:36:48.0321382Z ----------------------------------------------------------------------
2022-05-18T05:36:48.0321710Z Ran 1 test in 0.003s
2022-05-18T05:36:48.0321880Z 
2022-05-18T05:36:48.0321990Z OK (skipped=1)
2022-05-18T05:36:48.0322148Z 
2022-05-18T05:36:48.0322275Z Generating XML reports...
2022-05-18T05:36:48.0372693Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053648.xml
2022-05-18T05:36:49.3150980Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:49.3166216Z 
2022-05-18T05:36:49.3166619Z Running tests...
2022-05-18T05:36:49.3167126Z ----------------------------------------------------------------------
2022-05-18T05:36:49.3197145Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.003s)
2022-05-18T05:36:49.3197473Z 
2022-05-18T05:36:49.3197756Z ----------------------------------------------------------------------
2022-05-18T05:36:49.3198099Z Ran 1 test in 0.003s
2022-05-18T05:36:49.3198250Z 
2022-05-18T05:36:49.3198359Z OK (skipped=1)
2022-05-18T05:36:49.3198513Z 
2022-05-18T05:36:49.3198641Z Generating XML reports...
2022-05-18T05:36:49.3249239Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053649.xml
2022-05-18T05:36:50.6028471Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:50.6044078Z 
2022-05-18T05:36:50.6044320Z Running tests...
2022-05-18T05:36:50.6045067Z ----------------------------------------------------------------------
2022-05-18T05:36:52.2640946Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:36:52.3036854Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93737
2022-05-18T05:36:52.3155963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93738
2022-05-18T05:36:53.5236573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:36:53.5237137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:36:53.5237934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:53.5238662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:53.5246012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:36:53.5247251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:36:53.7206040Z ok (3.116s)
2022-05-18T05:36:53.7206243Z 
2022-05-18T05:36:53.7206640Z ----------------------------------------------------------------------
2022-05-18T05:36:53.7206992Z Ran 1 test in 3.116s
2022-05-18T05:36:53.7207158Z 
2022-05-18T05:36:53.7207257Z OK
2022-05-18T05:36:53.7207397Z 
2022-05-18T05:36:53.7207512Z Generating XML reports...
2022-05-18T05:36:53.7265381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053650.xml
2022-05-18T05:36:55.1464017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:55.1479641Z 
2022-05-18T05:36:55.1479911Z Running tests...
2022-05-18T05:36:55.1480335Z ----------------------------------------------------------------------
2022-05-18T05:36:56.8081577Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:36:56.8476967Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93856
2022-05-18T05:36:56.8596117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93857
2022-05-18T05:36:58.0016519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:36:58.0017099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:36:58.0017885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:58.0018596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:36:58.0125112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:36:58.1027837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:36:58.2647330Z ok (3.116s)
2022-05-18T05:36:58.2647557Z 
2022-05-18T05:36:58.2647931Z ----------------------------------------------------------------------
2022-05-18T05:36:58.2648273Z Ran 1 test in 3.117s
2022-05-18T05:36:58.2648440Z 
2022-05-18T05:36:58.2648536Z OK
2022-05-18T05:36:58.2648677Z 
2022-05-18T05:36:58.2648816Z Generating XML reports...
2022-05-18T05:36:58.2708382Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053655.xml
2022-05-18T05:36:59.6793973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:36:59.6808879Z 
2022-05-18T05:36:59.6809616Z Running tests...
2022-05-18T05:36:59.6810890Z ----------------------------------------------------------------------
2022-05-18T05:37:01.3064413Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:37:01.3465484Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93975
2022-05-18T05:37:01.3582776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93976
2022-05-18T05:37:02.5426650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:37:02.5427677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:37:02.5428500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:02.5429189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:02.5435553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:37:02.5436601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:37:02.7632210Z ok (3.082s)
2022-05-18T05:37:02.7632969Z 
2022-05-18T05:37:02.7633747Z ----------------------------------------------------------------------
2022-05-18T05:37:02.7634127Z Ran 1 test in 3.082s
2022-05-18T05:37:02.7634299Z 
2022-05-18T05:37:02.7634409Z OK
2022-05-18T05:37:02.7634555Z 
2022-05-18T05:37:02.7634679Z Generating XML reports...
2022-05-18T05:37:02.7690418Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053659.xml
2022-05-18T05:37:04.1977643Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:04.1992995Z 
2022-05-18T05:37:04.1993249Z Running tests...
2022-05-18T05:37:04.1993692Z ----------------------------------------------------------------------
2022-05-18T05:37:05.8718049Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:37:05.9125807Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94090
2022-05-18T05:37:05.9248813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94091
2022-05-18T05:37:07.1440466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:37:07.1441326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:37:07.1442136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:07.1442845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:07.1549451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:37:07.2453037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:37:07.4299376Z ok (3.230s)
2022-05-18T05:37:07.4299615Z 
2022-05-18T05:37:07.4300029Z ----------------------------------------------------------------------
2022-05-18T05:37:07.4300357Z Ran 1 test in 3.231s
2022-05-18T05:37:07.4300528Z 
2022-05-18T05:37:07.4300623Z OK
2022-05-18T05:37:07.4301863Z 
2022-05-18T05:37:07.4302279Z Generating XML reports...
2022-05-18T05:37:07.4358222Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053704.xml
2022-05-18T05:37:08.8483108Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:08.8497780Z 
2022-05-18T05:37:08.8498025Z Running tests...
2022-05-18T05:37:08.8498471Z ----------------------------------------------------------------------
2022-05-18T05:37:08.8519232Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T05:37:08.8519577Z 
2022-05-18T05:37:08.8519913Z ----------------------------------------------------------------------
2022-05-18T05:37:08.8520235Z Ran 1 test in 0.002s
2022-05-18T05:37:08.8520402Z 
2022-05-18T05:37:08.8520527Z OK (skipped=1)
2022-05-18T05:37:08.8520686Z 
2022-05-18T05:37:08.8520814Z Generating XML reports...
2022-05-18T05:37:08.8561505Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053708.xml
2022-05-18T05:37:10.1393994Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:10.1409487Z 
2022-05-18T05:37:10.1409824Z Running tests...
2022-05-18T05:37:10.1410551Z ----------------------------------------------------------------------
2022-05-18T05:37:10.1432605Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T05:37:10.1432923Z 
2022-05-18T05:37:10.1433211Z ----------------------------------------------------------------------
2022-05-18T05:37:10.1433555Z Ran 1 test in 0.002s
2022-05-18T05:37:10.1433700Z 
2022-05-18T05:37:10.1433814Z OK (skipped=1)
2022-05-18T05:37:10.1433975Z 
2022-05-18T05:37:10.1434396Z Generating XML reports...
2022-05-18T05:37:10.1478435Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053710.xml
2022-05-18T05:37:11.3833569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:11.3851094Z 
2022-05-18T05:37:11.3851555Z Running tests...
2022-05-18T05:37:11.3852407Z ----------------------------------------------------------------------
2022-05-18T05:37:13.0645848Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:37:13.1050327Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94279
2022-05-18T05:37:13.1169417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94280
2022-05-18T05:37:14.3179257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:37:14.3179809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:37:14.3180616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:14.3181626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:14.3289128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:37:14.4192289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:37:14.4304815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:37:14.4305338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:37:14.4306033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:37:14.4306721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:37:14.6221405Z ok (3.237s)
2022-05-18T05:37:14.6221699Z 
2022-05-18T05:37:14.6222218Z ----------------------------------------------------------------------
2022-05-18T05:37:14.6222567Z Ran 1 test in 3.237s
2022-05-18T05:37:14.6222735Z 
2022-05-18T05:37:14.6222837Z OK
2022-05-18T05:37:14.6222977Z 
2022-05-18T05:37:14.6223093Z Generating XML reports...
2022-05-18T05:37:14.6280160Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053711.xml
2022-05-18T05:37:16.0628651Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:16.0643977Z 
2022-05-18T05:37:16.0644483Z Running tests...
2022-05-18T05:37:16.0645136Z ----------------------------------------------------------------------
2022-05-18T05:37:17.7601780Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:37:17.8012439Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94404
2022-05-18T05:37:17.8134129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94405
2022-05-18T05:37:19.0343424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:37:19.0344126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:37:19.0345213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:19.0345926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:19.0352507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:37:19.0353437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:37:19.2182682Z skip: Skipped due to small world size. (3.153s)
2022-05-18T05:37:19.2182950Z 
2022-05-18T05:37:19.2183595Z ----------------------------------------------------------------------
2022-05-18T05:37:19.2183946Z Ran 1 test in 3.154s
2022-05-18T05:37:19.2184092Z 
2022-05-18T05:37:19.2184207Z OK (skipped=1)
2022-05-18T05:37:19.2184368Z 
2022-05-18T05:37:19.2184498Z Generating XML reports...
2022-05-18T05:37:19.2240634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053716.xml
2022-05-18T05:37:20.6581917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:20.6597770Z 
2022-05-18T05:37:20.6598291Z Running tests...
2022-05-18T05:37:20.6598796Z ----------------------------------------------------------------------
2022-05-18T05:37:22.3327180Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:37:22.3738663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94519
2022-05-18T05:37:22.3861256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94520
2022-05-18T05:37:23.6051081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:37:23.6051642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:37:23.6052446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:23.6053137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:23.6061057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:37:23.6061569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:37:23.7909660Z ok (3.131s)
2022-05-18T05:37:23.7909921Z 
2022-05-18T05:37:23.7910541Z ----------------------------------------------------------------------
2022-05-18T05:37:23.7910916Z Ran 1 test in 3.131s
2022-05-18T05:37:23.7911093Z 
2022-05-18T05:37:23.7911188Z OK
2022-05-18T05:37:23.7911331Z 
2022-05-18T05:37:23.7911451Z Generating XML reports...
2022-05-18T05:37:23.7967305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053720.xml
2022-05-18T05:37:25.2007276Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:25.2021747Z 
2022-05-18T05:37:25.2022014Z Running tests...
2022-05-18T05:37:25.2022455Z ----------------------------------------------------------------------
2022-05-18T05:37:26.8337003Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:37:26.8736960Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94634
2022-05-18T05:37:26.8854024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94635
2022-05-18T05:37:28.0999787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:37:28.1000354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:37:28.1001144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:28.1001851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:28.1009380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:37:28.1010180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:37:28.2905709Z ok (3.088s)
2022-05-18T05:37:28.2906022Z 
2022-05-18T05:37:28.2906409Z ----------------------------------------------------------------------
2022-05-18T05:37:28.2906758Z Ran 1 test in 3.088s
2022-05-18T05:37:28.2906938Z 
2022-05-18T05:37:28.2907036Z OK
2022-05-18T05:37:28.2907174Z 
2022-05-18T05:37:28.2907307Z Generating XML reports...
2022-05-18T05:37:28.2963170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053725.xml
2022-05-18T05:37:29.7444937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:29.7460868Z 
2022-05-18T05:37:29.7461619Z Running tests...
2022-05-18T05:37:29.7462501Z ----------------------------------------------------------------------
2022-05-18T05:37:31.4354538Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:37:31.4763304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94749
2022-05-18T05:37:31.4887577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94750
2022-05-18T05:37:32.6866871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:37:32.6867719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:37:32.6868529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:32.6869216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:32.6975550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:37:32.7878152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:37:32.9939689Z ok (3.247s)
2022-05-18T05:37:32.9939921Z 
2022-05-18T05:37:32.9940322Z ----------------------------------------------------------------------
2022-05-18T05:37:32.9940647Z Ran 1 test in 3.248s
2022-05-18T05:37:32.9940814Z 
2022-05-18T05:37:32.9940919Z OK
2022-05-18T05:37:32.9941066Z 
2022-05-18T05:37:32.9941196Z Generating XML reports...
2022-05-18T05:37:32.9997705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053729.xml
2022-05-18T05:37:34.4083795Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:34.4099539Z 
2022-05-18T05:37:34.4099774Z Running tests...
2022-05-18T05:37:34.4100346Z ----------------------------------------------------------------------
2022-05-18T05:37:36.0897019Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:37:36.1301423Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94864
2022-05-18T05:37:36.1422233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94865
2022-05-18T05:37:37.3036165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:37:37.3036758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:37:37.3037605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:37.3038288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:37.3045412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:37:37.3045911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:37:37.5471301Z ok (3.137s)
2022-05-18T05:37:37.5471523Z 
2022-05-18T05:37:37.5472159Z ----------------------------------------------------------------------
2022-05-18T05:37:37.5472533Z Ran 1 test in 3.137s
2022-05-18T05:37:37.5472699Z 
2022-05-18T05:37:37.5473712Z OK
2022-05-18T05:37:37.5474628Z 
2022-05-18T05:37:37.5474768Z Generating XML reports...
2022-05-18T05:37:37.5530951Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053734.xml
2022-05-18T05:37:38.9779230Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:38.9794818Z 
2022-05-18T05:37:38.9795275Z Running tests...
2022-05-18T05:37:38.9795932Z ----------------------------------------------------------------------
2022-05-18T05:37:40.6250580Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:37:40.6648792Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94983
2022-05-18T05:37:40.6767999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94984
2022-05-18T05:37:41.8788107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:37:41.8788679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:37:41.8789732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:41.8790448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:41.8897246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:37:41.9800100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:37:42.1818096Z ok (3.202s)
2022-05-18T05:37:42.1818289Z 
2022-05-18T05:37:42.1818866Z ----------------------------------------------------------------------
2022-05-18T05:37:42.1819242Z Ran 1 test in 3.202s
2022-05-18T05:37:42.1819415Z 
2022-05-18T05:37:42.1819514Z OK
2022-05-18T05:37:42.1819652Z 
2022-05-18T05:37:42.1819783Z Generating XML reports...
2022-05-18T05:37:42.1876738Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053738.xml
2022-05-18T05:37:43.6017214Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:43.6031830Z 
2022-05-18T05:37:43.6032119Z Running tests...
2022-05-18T05:37:43.6032576Z ----------------------------------------------------------------------
2022-05-18T05:37:45.2258659Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:37:45.2663044Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95102
2022-05-18T05:37:45.2781087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95103
2022-05-18T05:37:46.4419637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:37:46.4420207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:37:46.4421021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:46.4421710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:46.4429742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:37:46.4430236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:37:46.6828763Z ok (3.079s)
2022-05-18T05:37:46.6828948Z 
2022-05-18T05:37:46.6829355Z ----------------------------------------------------------------------
2022-05-18T05:37:46.6829713Z Ran 1 test in 3.080s
2022-05-18T05:37:46.6829883Z 
2022-05-18T05:37:46.6829981Z OK
2022-05-18T05:37:46.6830351Z 
2022-05-18T05:37:46.6830481Z Generating XML reports...
2022-05-18T05:37:46.6887631Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053743.xml
2022-05-18T05:37:48.1166719Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:48.1182415Z 
2022-05-18T05:37:48.1182989Z Running tests...
2022-05-18T05:37:48.1183618Z ----------------------------------------------------------------------
2022-05-18T05:37:48.1203681Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T05:37:48.1204262Z 
2022-05-18T05:37:48.1204867Z ----------------------------------------------------------------------
2022-05-18T05:37:48.1205377Z Ran 1 test in 0.002s
2022-05-18T05:37:48.1205545Z 
2022-05-18T05:37:48.1205636Z OK (skipped=1)
2022-05-18T05:37:48.1205794Z 
2022-05-18T05:37:48.1205923Z Generating XML reports...
2022-05-18T05:37:48.1249049Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053748.xml
2022-05-18T05:37:49.3730222Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:49.3745118Z 
2022-05-18T05:37:49.3745459Z Running tests...
2022-05-18T05:37:49.3745904Z ----------------------------------------------------------------------
2022-05-18T05:37:49.3765867Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T05:37:49.3766193Z 
2022-05-18T05:37:49.3766482Z ----------------------------------------------------------------------
2022-05-18T05:37:49.3766812Z Ran 1 test in 0.002s
2022-05-18T05:37:49.3766977Z 
2022-05-18T05:37:49.3767089Z OK (skipped=1)
2022-05-18T05:37:49.3767245Z 
2022-05-18T05:37:49.3767351Z Generating XML reports...
2022-05-18T05:37:49.3807821Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053749.xml
2022-05-18T05:37:50.6596849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:50.6612915Z 
2022-05-18T05:37:50.6613338Z Running tests...
2022-05-18T05:37:50.6613827Z ----------------------------------------------------------------------
2022-05-18T05:37:50.6637476Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T05:37:50.6637947Z 
2022-05-18T05:37:50.6638238Z ----------------------------------------------------------------------
2022-05-18T05:37:50.6638581Z Ran 1 test in 0.003s
2022-05-18T05:37:50.6638747Z 
2022-05-18T05:37:50.6638855Z OK (skipped=1)
2022-05-18T05:37:50.6638993Z 
2022-05-18T05:37:50.6639126Z Generating XML reports...
2022-05-18T05:37:50.6682275Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053750.xml
2022-05-18T05:37:51.9607241Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:51.9623002Z 
2022-05-18T05:37:51.9623319Z Running tests...
2022-05-18T05:37:51.9623743Z ----------------------------------------------------------------------
2022-05-18T05:37:53.6176232Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:37:53.6574010Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95326
2022-05-18T05:37:53.6692436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95327
2022-05-18T05:37:54.9130952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:37:54.9132137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:37:54.9132972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:54.9133876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:54.9140431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:37:54.9141300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:37:55.0739841Z ok (3.111s)
2022-05-18T05:37:55.0740183Z 
2022-05-18T05:37:55.0740881Z ----------------------------------------------------------------------
2022-05-18T05:37:55.0741510Z Ran 1 test in 3.112s
2022-05-18T05:37:55.0741678Z 
2022-05-18T05:37:55.0741779Z OK
2022-05-18T05:37:55.0741916Z 
2022-05-18T05:37:55.0742048Z Generating XML reports...
2022-05-18T05:37:55.0798799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053751.xml
2022-05-18T05:37:56.4977458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:37:56.4992740Z 
2022-05-18T05:37:56.4993162Z Running tests...
2022-05-18T05:37:56.4993652Z ----------------------------------------------------------------------
2022-05-18T05:37:58.1465238Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:37:58.1874441Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95445
2022-05-18T05:37:58.1997308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95446
2022-05-18T05:37:59.3747979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:37:59.3748548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:37:59.3749335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:59.3750046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:37:59.3858377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:37:59.4760895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:37:59.7047475Z ok (3.205s)
2022-05-18T05:37:59.7047694Z 
2022-05-18T05:37:59.7048075Z ----------------------------------------------------------------------
2022-05-18T05:37:59.7048420Z Ran 1 test in 3.205s
2022-05-18T05:37:59.7048585Z 
2022-05-18T05:37:59.7048677Z OK
2022-05-18T05:37:59.7048814Z 
2022-05-18T05:37:59.7048929Z Generating XML reports...
2022-05-18T05:37:59.7107067Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053756.xml
2022-05-18T05:38:01.1516574Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:38:01.1531463Z 
2022-05-18T05:38:01.1532105Z Running tests...
2022-05-18T05:38:01.1532632Z ----------------------------------------------------------------------
2022-05-18T05:38:02.8086804Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:38:02.8496804Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95560
2022-05-18T05:38:02.8623266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95561
2022-05-18T05:38:04.0328107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:38:04.0328675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:38:04.0329438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:04.0330621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:04.0438588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:38:04.1340676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:38:04.3674533Z ok (3.214s)
2022-05-18T05:38:04.3674754Z 
2022-05-18T05:38:04.3675545Z ----------------------------------------------------------------------
2022-05-18T05:38:04.3675925Z Ran 1 test in 3.214s
2022-05-18T05:38:04.3676076Z 
2022-05-18T05:38:04.3676173Z OK
2022-05-18T05:38:04.3676307Z 
2022-05-18T05:38:04.3676448Z Generating XML reports...
2022-05-18T05:38:04.3733267Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053801.xml
2022-05-18T05:38:05.7992822Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:38:05.8008554Z 
2022-05-18T05:38:05.8008775Z Running tests...
2022-05-18T05:38:05.8009250Z ----------------------------------------------------------------------
2022-05-18T05:38:07.4514740Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:38:07.4923709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95679
2022-05-18T05:38:07.5044951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95680
2022-05-18T05:38:08.6790857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:38:08.6791445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:38:08.6792240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:08.6792947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:08.6800537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:38:08.6801371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:38:08.9093435Z ok (3.108s)
2022-05-18T05:38:08.9093688Z 
2022-05-18T05:38:08.9094324Z ----------------------------------------------------------------------
2022-05-18T05:38:08.9094700Z Ran 1 test in 3.109s
2022-05-18T05:38:08.9094869Z 
2022-05-18T05:38:08.9094954Z OK
2022-05-18T05:38:08.9095091Z 
2022-05-18T05:38:08.9095231Z Generating XML reports...
2022-05-18T05:38:08.9152988Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053805.xml
2022-05-18T05:38:10.3166016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:38:10.3181040Z 
2022-05-18T05:38:10.3181458Z Running tests...
2022-05-18T05:38:10.3181970Z ----------------------------------------------------------------------
2022-05-18T05:38:11.9490914Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:38:11.9888568Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95798
2022-05-18T05:38:12.0007154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95799
2022-05-18T05:38:13.2186352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:38:13.2186909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:38:13.2187728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:13.2188431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:13.2194808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:38:13.2195520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:38:13.4053622Z ok (3.087s)
2022-05-18T05:38:13.4053978Z 
2022-05-18T05:38:13.4054391Z ----------------------------------------------------------------------
2022-05-18T05:38:13.4054735Z Ran 1 test in 3.087s
2022-05-18T05:38:13.4054882Z 
2022-05-18T05:38:13.4054976Z OK
2022-05-18T05:38:13.4055112Z 
2022-05-18T05:38:13.4055244Z Generating XML reports...
2022-05-18T05:38:13.4111261Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053810.xml
2022-05-18T05:38:14.8356784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:38:14.8371638Z 
2022-05-18T05:38:14.8371898Z Running tests...
2022-05-18T05:38:14.8372771Z ----------------------------------------------------------------------
2022-05-18T05:38:16.4993982Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:38:16.5402421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96003
2022-05-18T05:38:16.5523778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96004
2022-05-18T05:38:17.7120452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:38:17.7121402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:38:17.7122200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:17.7122907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:17.7128960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:38:17.7130240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:38:19.3599212Z ok (4.522s)
2022-05-18T05:38:19.3602531Z 
2022-05-18T05:38:19.3603481Z ----------------------------------------------------------------------
2022-05-18T05:38:19.3604154Z Ran 1 test in 4.523s
2022-05-18T05:38:19.3604437Z 
2022-05-18T05:38:19.3604601Z OK
2022-05-18T05:38:19.3604823Z 
2022-05-18T05:38:19.3605028Z Generating XML reports...
2022-05-18T05:38:19.3662471Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053814.xml
2022-05-18T05:38:20.8065810Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:38:20.8080959Z 
2022-05-18T05:38:20.8081418Z Running tests...
2022-05-18T05:38:20.8081929Z ----------------------------------------------------------------------
2022-05-18T05:38:22.4570584Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:38:22.4981030Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96210
2022-05-18T05:38:22.5102003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96211
2022-05-18T05:38:23.7148613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:38:23.7149165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:38:23.7149950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:23.7150656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:23.7157508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:38:23.7158372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:38:25.6182162Z ok (4.810s)
2022-05-18T05:38:25.6182419Z 
2022-05-18T05:38:25.6182826Z ----------------------------------------------------------------------
2022-05-18T05:38:25.6183149Z Ran 1 test in 4.810s
2022-05-18T05:38:25.6183325Z 
2022-05-18T05:38:25.6183418Z OK
2022-05-18T05:38:25.6183559Z 
2022-05-18T05:38:25.6183697Z Generating XML reports...
2022-05-18T05:38:25.6239804Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053820.xml
2022-05-18T05:38:27.0654749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:38:27.0669757Z 
2022-05-18T05:38:27.0669880Z Running tests...
2022-05-18T05:38:27.0670578Z ----------------------------------------------------------------------
2022-05-18T05:38:28.7253332Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:38:28.7657240Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96331
2022-05-18T05:38:28.7777406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96332
2022-05-18T05:38:29.9458328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:38:29.9459201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:38:29.9460001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:29.9460712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:29.9468010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:38:29.9468483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:38:29.9557342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpechrcdti
2022-05-18T05:38:29.9557890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2nwgka2
2022-05-18T05:38:29.9560094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpechrcdti/_remote_module_non_scriptable.py
2022-05-18T05:38:29.9560646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2nwgka2/_remote_module_non_scriptable.py
2022-05-18T05:38:30.1826511Z ok (3.115s)
2022-05-18T05:38:30.1826696Z 
2022-05-18T05:38:30.1827050Z ----------------------------------------------------------------------
2022-05-18T05:38:30.1827402Z Ran 1 test in 3.116s
2022-05-18T05:38:30.1827570Z 
2022-05-18T05:38:30.1827666Z OK
2022-05-18T05:38:30.1827804Z 
2022-05-18T05:38:30.1827937Z Generating XML reports...
2022-05-18T05:38:30.1884810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053827.xml
2022-05-18T05:38:31.5734786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:38:31.5750006Z 
2022-05-18T05:38:31.5750410Z Running tests...
2022-05-18T05:38:31.5750899Z ----------------------------------------------------------------------
2022-05-18T05:38:33.2236169Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:38:33.2642892Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96450
2022-05-18T05:38:33.2762623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96451
2022-05-18T05:38:34.4544534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:38:34.4545406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:38:34.4546200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:34.4547141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:34.4553945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:38:34.4554985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:38:35.7885204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_c5cf8t5
2022-05-18T05:38:35.7886096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_c5cf8t5/_remote_module_non_scriptable.py
2022-05-18T05:38:35.8032530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznf2bide
2022-05-18T05:38:35.8035339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznf2bide/_remote_module_non_scriptable.py
2022-05-18T05:38:36.0837091Z ok (4.508s)
2022-05-18T05:38:36.0837316Z 
2022-05-18T05:38:36.0837691Z ----------------------------------------------------------------------
2022-05-18T05:38:36.0838049Z Ran 1 test in 4.509s
2022-05-18T05:38:36.0838216Z 
2022-05-18T05:38:36.0838291Z OK
2022-05-18T05:38:36.0838429Z 
2022-05-18T05:38:36.0838565Z Generating XML reports...
2022-05-18T05:38:36.0895487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053831.xml
2022-05-18T05:38:37.5155704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:38:37.5171054Z 
2022-05-18T05:38:37.5171197Z Running tests...
2022-05-18T05:38:37.5171887Z ----------------------------------------------------------------------
2022-05-18T05:38:39.1466581Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:38:39.1866803Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96567
2022-05-18T05:38:39.1987779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96568
2022-05-18T05:38:40.3811939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:38:40.3812535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:38:40.3813349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:40.3814060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:40.3821717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:38:40.3822223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:38:41.7179219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6s68njcc
2022-05-18T05:38:41.7180216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6s68njcc/_remote_module_non_scriptable.py
2022-05-18T05:38:41.7254629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rd05_gr
2022-05-18T05:38:41.7257625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rd05_gr/_remote_module_non_scriptable.py
2022-05-18T05:38:42.0269631Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:38:42.0274716Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:38:42.3067060Z ok (4.789s)
2022-05-18T05:38:42.3067269Z 
2022-05-18T05:38:42.3067648Z ----------------------------------------------------------------------
2022-05-18T05:38:42.3067966Z Ran 1 test in 4.790s
2022-05-18T05:38:42.3068141Z 
2022-05-18T05:38:42.3068242Z OK
2022-05-18T05:38:42.3068378Z 
2022-05-18T05:38:42.3068513Z Generating XML reports...
2022-05-18T05:38:42.3125109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053837.xml
2022-05-18T05:38:43.7334074Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:38:43.7348470Z 
2022-05-18T05:38:43.7348822Z Running tests...
2022-05-18T05:38:43.7349300Z ----------------------------------------------------------------------
2022-05-18T05:38:45.3740143Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:38:45.4141357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96688
2022-05-18T05:38:45.4260487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96689
2022-05-18T05:38:46.6340289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:38:46.6340864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:38:46.6341655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:46.6342385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:46.6350434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:38:46.6350937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:38:46.6558481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:38:46.6558998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:38:46.6559670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:38:46.6560364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:38:46.6769005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:38:46.6769763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:38:46.6770628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:38:46.6771338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:38:48.0167726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zr0c6xu
2022-05-18T05:38:48.0168560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zr0c6xu/_remote_module_non_scriptable.py
2022-05-18T05:38:48.0278552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7vygbv6
2022-05-18T05:38:48.0281343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7vygbv6/_remote_module_non_scriptable.py
2022-05-18T05:38:53.3425912Z ok (9.607s)
2022-05-18T05:38:53.3426133Z 
2022-05-18T05:38:53.3426776Z ----------------------------------------------------------------------
2022-05-18T05:38:53.3427118Z Ran 1 test in 9.608s
2022-05-18T05:38:53.3427283Z 
2022-05-18T05:38:53.3427382Z OK
2022-05-18T05:38:53.3427516Z 
2022-05-18T05:38:53.3427662Z Generating XML reports...
2022-05-18T05:38:53.3484063Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053843.xml
2022-05-18T05:38:54.7875677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:38:54.7890771Z 
2022-05-18T05:38:54.7891131Z Running tests...
2022-05-18T05:38:54.7891600Z ----------------------------------------------------------------------
2022-05-18T05:38:56.4340997Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:38:56.4750850Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96817
2022-05-18T05:38:56.4872170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96818
2022-05-18T05:38:57.6479928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:38:57.6480739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:38:57.6481551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:57.6482237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:38:57.6590877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:38:57.7492916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:38:57.7608169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:38:57.7608720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:38:57.7609429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:38:57.7610405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:38:57.7919136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:38:57.7919701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:38:57.7920436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:38:57.7921140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:38:59.1192353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqx2km0i
2022-05-18T05:38:59.1193171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqx2km0i/_remote_module_non_scriptable.py
2022-05-18T05:38:59.1311626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplms84t2m
2022-05-18T05:38:59.1314210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplms84t2m/_remote_module_non_scriptable.py
2022-05-18T05:39:04.5045812Z ok (9.715s)
2022-05-18T05:39:04.5046035Z 
2022-05-18T05:39:04.5046672Z ----------------------------------------------------------------------
2022-05-18T05:39:04.5047028Z Ran 1 test in 9.715s
2022-05-18T05:39:04.5047194Z 
2022-05-18T05:39:04.5048385Z OK
2022-05-18T05:39:04.5048576Z 
2022-05-18T05:39:04.5049351Z Generating XML reports...
2022-05-18T05:39:04.5104164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053854.xml
2022-05-18T05:39:04.9177576Z Running distributed/optim/test_zero_redundancy_optimizer ... [2022-05-18 05:39:04.917186]
2022-05-18T05:39:04.9178392Z Executing ['/opt/conda/bin/python', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:39:04.917283]
2022-05-18T05:39:06.0929234Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer
2022-05-18T05:39:06.0948709Z 
2022-05-18T05:39:06.0949169Z Running tests...
2022-05-18T05:39:06.0949657Z ----------------------------------------------------------------------
2022-05-18T05:39:06.0972177Z   test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:07.7526759Z Check that ZeroRedundancyOptimizer properly handles adding a new ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:39:07.7654143Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67287 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.670s)
2022-05-18T05:39:07.7674684Z   test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:07.7960620Z Check the state consolidation mechanism and the state dict exposed ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96946
2022-05-18T05:39:07.8083594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96947
2022-05-18T05:39:08.9436246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:08.9438539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:08.9508684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:08.9512306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:08.9513121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:08.9541372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:12.0177232Z ok (4.252s)
2022-05-18T05:39:12.0193708Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:12.0355402Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97034
2022-05-18T05:39:12.0468754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97035
2022-05-18T05:39:13.2473266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:13.2475850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:13.3389627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:13.3394514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:13.3395603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:13.3488860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:14.7675343Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:14.7678303Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:15.0698820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:15.0699365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:15.1122978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:15.1123484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:15.4544597Z ok (3.437s)
2022-05-18T05:39:15.4561454Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:15.4726796Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97151
2022-05-18T05:39:15.4843006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97152
2022-05-18T05:39:16.6383258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:16.6384814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:16.6545862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:16.6550375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:16.6552087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:16.6589232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:18.0756206Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:18.0758966Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:18.3796784Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:18.3797624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:18.4217481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:18.4219786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:18.7914624Z ok (3.337s)
2022-05-18T05:39:18.7931749Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:18.8094507Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97268
2022-05-18T05:39:18.8208034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97269
2022-05-18T05:39:20.0227582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:20.0229028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:20.0278999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:20.0284136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:20.0285681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:20.0333368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:21.4644326Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:21.4646851Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:21.7696766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:21.7697334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:21.8151518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:21.8152513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:22.1279570Z ok (3.336s)
2022-05-18T05:39:22.1296294Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:22.1458185Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97385
2022-05-18T05:39:22.1570880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97386
2022-05-18T05:39:23.3491325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:23.3493798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:23.3646037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:23.3650304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:23.3651796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:23.3699009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:24.8130592Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:24.8132995Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:25.1188234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:25.1188818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:25.1659755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:25.1660537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:25.5657068Z ok (3.438s)
2022-05-18T05:39:25.5673600Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:25.5834769Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97502
2022-05-18T05:39:25.5951577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97503
2022-05-18T05:39:26.7613503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:26.7616557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:26.7636027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:26.7640476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:26.7641455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:26.7718794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:28.1943187Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:28.1945730Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:28.4937862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:28.4938766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:28.5313992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:28.5314496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:28.9026689Z ok (3.337s)
2022-05-18T05:39:28.9043125Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:28.9206119Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97619
2022-05-18T05:39:28.9318740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97620
2022-05-18T05:39:30.0671459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:30.0672791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:30.0678123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:30.0682590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:30.0684098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:30.0776829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:31.5016060Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:31.5018486Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:31.8002293Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:31.8002829Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:31.8390693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:31.8391210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:32.2392855Z ok (3.336s)
2022-05-18T05:39:32.2410019Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:32.2576258Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97736
2022-05-18T05:39:32.2692340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97737
2022-05-18T05:39:33.4192767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:33.4194567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:33.4201644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:33.4206315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:33.4207850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:33.4298254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:34.8368652Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:34.8372322Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:35.1427601Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:35.1428146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:35.1854270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:35.1854781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:35.5765197Z ok (3.337s)
2022-05-18T05:39:35.5783065Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:35.5948761Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97853
2022-05-18T05:39:35.6064444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97854
2022-05-18T05:39:36.7202420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:36.7202944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:36.7204884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:36.7205409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:36.7206202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:36.7207318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:38.1392612Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:38.1395034Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:38.4422759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:38.4423342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:38.4862558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:38.4863391Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:38.8135745Z ok (3.237s)
2022-05-18T05:39:38.8152733Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:38.8312954Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97970
2022-05-18T05:39:38.8426829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97971
2022-05-18T05:39:39.9478555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:39.9481067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:39.9604046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:39.9607889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:39.9609120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:39.9684848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:41.3893515Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:41.3896116Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:41.6937668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:41.6938225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:41.7390632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:41.7391144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:42.0499595Z ok (3.236s)
2022-05-18T05:39:42.0518034Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:42.0682539Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98087
2022-05-18T05:39:42.0796319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98088
2022-05-18T05:39:43.2333559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:43.2336022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:43.2390803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:43.2394776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:43.2395738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:43.2438507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:44.6547361Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:44.6550791Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:44.9502502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:44.9503040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:44.9948094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:44.9948605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:45.3869342Z ok (3.337s)
2022-05-18T05:39:45.3887129Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:45.4050395Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98204
2022-05-18T05:39:45.4163568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98205
2022-05-18T05:39:46.5148867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:46.5150787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:46.5280750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:46.5284539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:46.5285348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:46.5354641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:47.9466820Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:47.9468277Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:48.2527238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:48.2527793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:48.3016605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:48.3017124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:48.6232928Z ok (3.236s)
2022-05-18T05:39:48.6250556Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:48.6416013Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98321
2022-05-18T05:39:48.6530009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98322
2022-05-18T05:39:49.7861833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:49.7864675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:49.7969582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:49.7973414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:49.7974234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:49.8069142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:51.2195442Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:51.2197909Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:51.5230786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:51.5231332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:51.5730000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:51.5730730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:51.9601821Z ok (3.337s)
2022-05-18T05:39:51.9618488Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:51.9781401Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98438
2022-05-18T05:39:51.9895212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98439
2022-05-18T05:39:53.1387935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:53.1390173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:53.1443112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:53.1446674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:53.1447478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:53.1493567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:54.5785183Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:54.5788093Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:54.8787848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:54.8788392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:54.9202929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:54.9203415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:55.2969233Z ok (3.337s)
2022-05-18T05:39:55.2986788Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:55.3153838Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98555
2022-05-18T05:39:55.3273154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98556
2022-05-18T05:39:56.4804386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:56.4806682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:56.4886779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:56.4890525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:56.4891764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:56.4909188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:57.8968408Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:57.8972449Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:39:58.1980058Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:58.1980598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:58.2402496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:58.2403014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:39:58.5356926Z ok (3.239s)
2022-05-18T05:39:58.5374252Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:39:58.5538928Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98672
2022-05-18T05:39:58.5656177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98673
2022-05-18T05:39:59.7006510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:39:59.7009067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:39:59.7022611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:39:59.7026488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:39:59.7027422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:39:59.7111582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:01.1225747Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:40:01.1229336Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:40:01.4249656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:40:01.4250437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:40:01.4724054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:40:01.4724608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:40:01.8727457Z ok (3.337s)
2022-05-18T05:40:01.8744548Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:01.8913342Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98789
2022-05-18T05:40:01.9218843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98790
2022-05-18T05:40:03.1003103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:03.1005017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:03.1051821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:03.1056888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:40:03.1058639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:03.1108643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:04.5229548Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:40:04.5232949Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-05-18T05:40:04.8294906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:40:04.8295449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:40:04.8768283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:40:04.8768792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:40:05.2291161Z ok (3.356s)
2022-05-18T05:40:05.2348762Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:05.2515133Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98906
2022-05-18T05:40:05.2631547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98907
2022-05-18T05:40:06.4153987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:06.4155558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:06.4310994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:06.4315288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:40:06.4316127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:06.4360080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:07.7380973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfeir11cl
2022-05-18T05:40:07.7381585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfeir11cl/_remote_module_non_scriptable.py
2022-05-18T05:40:07.7626307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3zyfio2_
2022-05-18T05:40:07.7628700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3zyfio2_/_remote_module_non_scriptable.py
2022-05-18T05:40:08.1306258Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:40:08.1337395Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:40:08.4115084Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.4133734Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.4325568Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.4344028Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.4537228Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.4555888Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.4748727Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.4767539Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.4961357Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.4981078Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.5175752Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.5193615Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.5387391Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.5406276Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.5729398Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.5760310Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:08.9713539Z ok (3.742s)
2022-05-18T05:40:08.9761004Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:08.9928545Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98993
2022-05-18T05:40:09.0045381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98994
2022-05-18T05:40:10.1579455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:10.1581793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:10.1610539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:10.1614536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:40:10.1615771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:10.1684911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:11.4928697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjtk_x1f
2022-05-18T05:40:11.4929293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjtk_x1f/_remote_module_non_scriptable.py
2022-05-18T05:40:11.4935588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zlchnwn
2022-05-18T05:40:11.4938638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zlchnwn/_remote_module_non_scriptable.py
2022-05-18T05:40:11.8725530Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:40:11.8741614Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:40:12.1523306Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.1543572Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.1735252Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.1756150Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.1948294Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.1968649Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.2160030Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.2180895Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.2372733Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.2392823Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.2584156Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.2604242Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.2796199Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.2817485Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.3137948Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.3169332Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:12.7123254Z ok (3.741s)
2022-05-18T05:40:12.7170229Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:12.7343719Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99080
2022-05-18T05:40:12.7458164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99081
2022-05-18T05:40:13.9670142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:13.9672320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:40:13.9682101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:13.9686615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:13.9688093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:13.9776307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:15.2742166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7yyfo077
2022-05-18T05:40:15.2743315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7yyfo077/_remote_module_non_scriptable.py
2022-05-18T05:40:15.3118431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1jopdkni
2022-05-18T05:40:15.3120036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1jopdkni/_remote_module_non_scriptable.py
2022-05-18T05:40:15.6902430Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:40:15.6955419Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:40:15.9631369Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:15.9641594Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:15.9832437Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:15.9844211Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:16.0036810Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:16.0048650Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:16.0240837Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:16.0252526Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:16.0446016Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:16.0457503Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:16.0651423Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:16.0662242Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:16.0855181Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:16.0866982Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:16.1180710Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:16.1200490Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:16.4539146Z ok (3.741s)
2022-05-18T05:40:16.4584996Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:16.4758990Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99167
2022-05-18T05:40:16.4873405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99168
2022-05-18T05:40:17.6222828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:17.6225183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:40:17.6296006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:17.6300336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:17.6301714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:17.6328262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:18.9330061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmyua_j_
2022-05-18T05:40:18.9331535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmyua_j_/_remote_module_non_scriptable.py
2022-05-18T05:40:18.9722479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5jrdrlhz
2022-05-18T05:40:18.9725072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5jrdrlhz/_remote_module_non_scriptable.py
2022-05-18T05:40:19.3507562Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:40:19.3558997Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:40:19.6292198Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.6312156Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.6503904Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.6515821Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.6710237Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.6723385Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.6915990Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.6930195Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.7123845Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.7138363Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.7330680Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.7345595Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.7539265Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.7553330Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.7873770Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:19.7890960Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:20.0955172Z ok (3.641s)
2022-05-18T05:40:20.1000851Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:20.1166199Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99254
2022-05-18T05:40:20.1283802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99255
2022-05-18T05:40:21.3108148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:21.3110517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:40:21.3160897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:21.3164780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:21.3165594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:21.3213473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:22.6267788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvwxw4bs
2022-05-18T05:40:22.6268998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvwxw4bs/_remote_module_non_scriptable.py
2022-05-18T05:40:22.6309279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmi4k5uzz
2022-05-18T05:40:22.6311942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmi4k5uzz/_remote_module_non_scriptable.py
2022-05-18T05:40:23.0009706Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:40:23.0065767Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:40:23.2566762Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.2582794Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.2757202Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.2772948Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.2947916Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.2964268Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.3139546Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.3155703Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.3330537Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.3347754Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.3521857Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.3539283Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.3714524Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.3730726Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.3940428Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.3941212Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:23.7363723Z ok (3.641s)
2022-05-18T05:40:23.7410535Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:23.7574903Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99341
2022-05-18T05:40:23.7688729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99342
2022-05-18T05:40:24.9163134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:24.9165486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:24.9222181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:24.9225862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:40:24.9227151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:24.9268185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:26.2267240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_sy56jzl
2022-05-18T05:40:26.2267865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_sy56jzl/_remote_module_non_scriptable.py
2022-05-18T05:40:26.2614395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuadc53w4
2022-05-18T05:40:26.2616802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuadc53w4/_remote_module_non_scriptable.py
2022-05-18T05:40:26.6247459Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:40:26.6306499Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:40:26.8823019Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:26.8835178Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:26.9015037Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:26.9026996Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:26.9207082Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:26.9219119Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:26.9398468Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:26.9410819Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:26.9590734Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:26.9603552Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:26.9783174Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:26.9795100Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:26.9975223Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:26.9987205Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:27.0195572Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:27.0199175Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-05-18T05:40:27.3768216Z ok (3.640s)
2022-05-18T05:40:27.3783944Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:27.3946564Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99428
2022-05-18T05:40:27.4060089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99429
2022-05-18T05:40:28.5922870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:28.5926009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:28.5939418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:28.5943132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:40:28.5944202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:28.6028464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:31.3144766Z ok (3.937s)
2022-05-18T05:40:31.3176386Z   test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:31.3340887Z Check parity between constructing ZeRO with multiple parameter groups ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99516
2022-05-18T05:40:31.3454100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99517
2022-05-18T05:40:32.5158887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:32.5161019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:32.5337250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:32.5341001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:40:32.5342245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:32.5364692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:35.5546651Z ok (4.240s)
2022-05-18T05:40:35.5580446Z   test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:35.5743487Z Check that ZeroRedundancyOptimizer works with a non-default process ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99604
2022-05-18T05:40:35.5856844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99605
2022-05-18T05:40:36.7734039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:36.7735494Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-05-18T05:40:36.7832283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:36.7833978Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-05-18T05:40:36.9894471Z ok (1.435s)
2022-05-18T05:40:36.9905911Z   test_sharding (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:36.9910159Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67295 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-05-18T05:40:36.9929023Z   test_step (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:37.0090317Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99676
2022-05-18T05:40:37.0204961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99677
2022-05-18T05:40:38.1530842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:38.1533409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:38.1629871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:38.1634000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:40:38.1635105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:38.1635798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:40.1275741Z ok (3.136s)
2022-05-18T05:40:40.1300876Z   test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:40.1463735Z Check that ZeroRedundancyOptimizer properly exposes the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99763
2022-05-18T05:40:40.1580564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99764
2022-05-18T05:40:41.3119557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:41.3122314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:41.3202006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:41.3206545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:40:41.3207747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:41.3224819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:43.2649216Z ok (3.137s)
2022-05-18T05:40:43.2655287Z   test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:43.2818324Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99850
2022-05-18T05:40:43.2931196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99851
2022-05-18T05:40:44.4487308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:44.4708706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:44.4924333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:40:44.4924851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:44.4925633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:44.4926336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:44.5033986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8aghzyy
2022-05-18T05:40:44.5036965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8aghzyy/_remote_module_non_scriptable.py
2022-05-18T05:40:44.5123983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7lzrx9u
2022-05-18T05:40:44.5126615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7lzrx9u/_remote_module_non_scriptable.py
2022-05-18T05:40:44.5331210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:40:44.5331754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:40:44.5761256Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpj7lzrx9u'>
2022-05-18T05:40:44.5761721Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:40:44.5762318Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpl8aghzyy'>
2022-05-18T05:40:44.5762837Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:40:44.7972297Z ok (1.532s)
2022-05-18T05:40:44.7978220Z   test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:44.8139048Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99932
2022-05-18T05:40:44.8254480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99933
2022-05-18T05:40:45.9559903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:45.9567346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:45.9791171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:45.9800850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:40:45.9801964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:45.9872619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:40:47.2898113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxdk1f37
2022-05-18T05:40:47.2898921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxdk1f37/_remote_module_non_scriptable.py
2022-05-18T05:40:47.3161338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6qukgpa
2022-05-18T05:40:47.3164324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6qukgpa/_remote_module_non_scriptable.py
2022-05-18T05:40:48.6611560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:40:48.6612135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:40:48.7406797Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpyxdk1f37'>
2022-05-18T05:40:48.7407592Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:40:48.7408196Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpy6qukgpa'>
2022-05-18T05:40:48.7408652Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:40:49.1346822Z ok (4.337s)
2022-05-18T05:40:49.1354765Z   test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:49.1517234Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100020
2022-05-18T05:40:49.1632558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100021
2022-05-18T05:40:50.3306975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:50.3362595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:50.4669258Z skip: Need at least 4 CUDA devices (1.332s)
2022-05-18T05:40:50.4676833Z   test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T05:40:50.4842089Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100092
2022-05-18T05:40:50.4957206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100093
2022-05-18T05:40:51.5905938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:51.5997120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:40:51.7996256Z skip: Need at least 4 CUDA devices (1.333s)
2022-05-18T05:40:51.8020302Z   test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T05:40:51.8181376Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100164
2022-05-18T05:40:52.9745228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:52.9747739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:52.9749004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:40:53.1216004Z ok (1.322s)
2022-05-18T05:40:53.1231055Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T05:40:53.1393576Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100201
2022-05-18T05:40:54.2611547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:54.2614478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:54.2615294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:40:55.8451338Z ok (2.723s)
2022-05-18T05:40:55.8462875Z   test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T05:40:55.8622918Z Check that ZeroRedundancyOptimizer raises an exception if the input ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100245
2022-05-18T05:40:56.9994558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:56.9996927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:56.9997863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:40:57.1656730Z ok (1.320s)
2022-05-18T05:40:57.1688381Z   test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T05:40:57.1857663Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100282
2022-05-18T05:40:58.3171824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:40:58.3174923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:40:58.3175756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:40:59.8916987Z ok (2.726s)
2022-05-18T05:40:59.8931497Z   test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T05:40:59.9096794Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100326
2022-05-18T05:41:01.0254953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:01.0257547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:01.0258642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:41:02.6154306Z ok (2.724s)
2022-05-18T05:41:02.6168949Z   test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T05:41:02.6333773Z Check that the ``step(**kwargs)`` interface is properly exposed. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100370
2022-05-18T05:41:03.8123777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:03.8126610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:03.8127411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:41:05.3391650Z ok (2.724s)
2022-05-18T05:41:05.3402897Z   test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T05:41:05.3563143Z Check that the ``step()`` method (without closure) is handled as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100414
2022-05-18T05:41:06.5164158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:06.5166521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:06.5167324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:41:08.0622208Z ok (2.723s)
2022-05-18T05:41:08.0634434Z   test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T05:41:08.0795088Z Check that the ``zero_grad`` method is properly handled. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100458
2022-05-18T05:41:09.2626142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:09.2629028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:09.2629969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:41:09.4830515Z ok (1.421s)
2022-05-18T05:41:09.4830724Z 
2022-05-18T05:41:09.4833537Z ----------------------------------------------------------------------
2022-05-18T05:41:09.4834454Z Ran 42 tests in 123.388s
2022-05-18T05:41:09.4834638Z 
2022-05-18T05:41:09.4835110Z OK (skipped=4)
2022-05-18T05:41:09.4835484Z 
2022-05-18T05:41:09.4835784Z Generating XML reports...
2022-05-18T05:41:09.4925122Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220518053906.xml
2022-05-18T05:41:09.4938717Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220518053906.xml
2022-05-18T05:41:09.7774858Z Running distributed/fsdp/test_fsdp_optim_state ... [2022-05-18 05:41:09.776945]
2022-05-18T05:41:09.7775972Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:41:09.777084]
2022-05-18T05:41:10.6974279Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state
2022-05-18T05:41:10.6994199Z 
2022-05-18T05:41:10.6994473Z Running tests...
2022-05-18T05:41:10.6994916Z ----------------------------------------------------------------------
2022-05-18T05:41:10.7013496Z   test_full_optim_state_dict_nested_use_multiple_param_groups_False_rank0_only_False (__main__.TestFSDPOptimState)
2022-05-18T05:41:12.3718277Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:41:12.4134307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100534
2022-05-18T05:41:12.4258256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100535
2022-05-18T05:41:13.3456383Z dist init r=1, world=2
2022-05-18T05:41:13.3459869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:13.3473612Z dist init r=0, world=2
2022-05-18T05:41:13.3478677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:13.3479691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:13.3562640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:14.7528529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:14.7529246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:15.4331034Z ok (4.733s)
2022-05-18T05:41:15.4350546Z   test_full_optim_state_dict_nested_use_multiple_param_groups_False_rank0_only_True (__main__.TestFSDPOptimState)
2022-05-18T05:41:15.4514258Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100621
2022-05-18T05:41:15.4626375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100622
2022-05-18T05:41:16.3943409Z dist init r=1, world=2
2022-05-18T05:41:16.3946696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:16.4222471Z dist init r=0, world=2
2022-05-18T05:41:16.4227049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:16.4228107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:16.4251496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:17.7986902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:17.7987441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:18.4694475Z ok (3.036s)
2022-05-18T05:41:18.4712037Z   test_full_optim_state_dict_nested_use_multiple_param_groups_True_rank0_only_False (__main__.TestFSDPOptimState)
2022-05-18T05:41:18.4877942Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100708
2022-05-18T05:41:18.4993347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100709
2022-05-18T05:41:19.4238062Z dist init r=1, world=2
2022-05-18T05:41:19.4241316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:19.4250164Z dist init r=0, world=2
2022-05-18T05:41:19.4254589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:19.4255607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:19.4344143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:20.8122239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:20.8123240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:21.5059612Z ok (3.036s)
2022-05-18T05:41:21.5077621Z   test_full_optim_state_dict_nested_use_multiple_param_groups_True_rank0_only_True (__main__.TestFSDPOptimState)
2022-05-18T05:41:21.5240029Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100795
2022-05-18T05:41:21.5351835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100796
2022-05-18T05:41:22.4136840Z dist init r=0, world=2
2022-05-18T05:41:22.4140031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:22.4669740Z dist init r=1, world=2
2022-05-18T05:41:22.4674203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:22.4675384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:22.4749390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:23.8490867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:23.8491990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:24.5420047Z ok (3.036s)
2022-05-18T05:41:24.5436587Z   test_rekey_optim_state_dict_to_ids_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2022-05-18T05:41:24.5598903Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100882
2022-05-18T05:41:24.5710823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100883
2022-05-18T05:41:25.4925238Z dist init r=1, world=2
2022-05-18T05:41:25.4928857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:25.5299793Z dist init r=0, world=2
2022-05-18T05:41:25.5304400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:25.5305491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:25.5335425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:26.9048785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:26.9049331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:27.6779087Z ok (3.136s)
2022-05-18T05:41:27.6795283Z   test_rekey_optim_state_dict_to_ids_use_multiple_param_groups_True (__main__.TestFSDPOptimState)
2022-05-18T05:41:27.6957406Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100969
2022-05-18T05:41:27.7072015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100970
2022-05-18T05:41:28.6243961Z dist init r=0, world=2
2022-05-18T05:41:28.6247049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:28.6280165Z dist init r=1, world=2
2022-05-18T05:41:28.6284715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:28.6285952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:28.6349835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:30.0225883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:30.0226657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:30.7137541Z ok (3.036s)
2022-05-18T05:41:30.7157434Z   test_rekey_optim_state_dict_to_names_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2022-05-18T05:41:30.7329634Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101056
2022-05-18T05:41:30.7441706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101057
2022-05-18T05:41:31.6568758Z dist init r=0, world=2
2022-05-18T05:41:31.6571664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:31.6624242Z dist init r=1, world=2
2022-05-18T05:41:31.6629228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:31.6630342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:31.6674920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:33.0433644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:33.0434330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:33.7509413Z ok (3.037s)
2022-05-18T05:41:33.7519994Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:41:33.7684046Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101143
2022-05-18T05:41:33.7796879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101144
2022-05-18T05:41:34.7004921Z dist init r=1, world=2
2022-05-18T05:41:34.7008359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:34.7027472Z dist init r=0, world=2
2022-05-18T05:41:34.7031989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:34.7032992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:34.7111378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:36.0746509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:36.0747053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:36.3858582Z skip: Need at least 4 CUDA devices (2.635s)
2022-05-18T05:41:36.3868798Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:41:36.4031993Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101226
2022-05-18T05:41:36.4144355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101227
2022-05-18T05:41:37.3287586Z dist init r=1, world=2
2022-05-18T05:41:37.3290876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:37.3314325Z dist init r=0, world=2
2022-05-18T05:41:37.3319027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:37.3319835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:37.3393684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:38.7092887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:38.7093448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:39.0204244Z skip: Need at least 4 CUDA devices (2.634s)
2022-05-18T05:41:39.0214848Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:41:39.0379184Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101309
2022-05-18T05:41:39.0490455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101310
2022-05-18T05:41:40.0058779Z dist init r=0, world=2
2022-05-18T05:41:40.0062952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:40.0179233Z dist init r=1, world=2
2022-05-18T05:41:40.0183846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:40.0185005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:40.0266504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:41.4065084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:41.4066050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:41.6549023Z skip: Need at least 4 CUDA devices (2.634s)
2022-05-18T05:41:41.6558212Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:41:41.6721187Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101392
2022-05-18T05:41:41.6832305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101393
2022-05-18T05:41:42.5607446Z dist init r=0, world=2
2022-05-18T05:41:42.5611525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:42.6063069Z dist init r=1, world=2
2022-05-18T05:41:42.6067985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:42.6068952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:42.6119091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:43.9914040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:43.9914999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:44.2901556Z skip: Need at least 4 CUDA devices (2.635s)
2022-05-18T05:41:44.2911177Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:41:44.3073829Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101475
2022-05-18T05:41:44.3187181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101476
2022-05-18T05:41:45.2336695Z dist init r=1, world=2
2022-05-18T05:41:45.2340208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:45.2355535Z dist init r=0, world=2
2022-05-18T05:41:45.2360207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:45.2361502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:45.2443051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:46.6275021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:46.6275558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:46.9255660Z skip: Need at least 4 CUDA devices (2.635s)
2022-05-18T05:41:46.9265475Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:41:46.9429220Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101558
2022-05-18T05:41:46.9549335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101559
2022-05-18T05:41:47.8832071Z dist init r=0, world=2
2022-05-18T05:41:47.8832412Z dist init r=1, world=2
2022-05-18T05:41:47.8836099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:47.8836633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:47.8837643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:47.8838368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:49.2553474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:49.2554006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:49.5617983Z skip: Need at least 4 CUDA devices (2.636s)
2022-05-18T05:41:49.5628000Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:41:49.5789952Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101641
2022-05-18T05:41:49.5902512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101642
2022-05-18T05:41:50.5207299Z dist init r=1, world=2
2022-05-18T05:41:50.5210139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:50.5214994Z dist init r=0, world=2
2022-05-18T05:41:50.5220140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:50.5220947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:50.5312838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:51.9193485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:51.9194022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:52.1963972Z skip: Need at least 4 CUDA devices (2.634s)
2022-05-18T05:41:52.1974856Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:41:52.2139710Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101724
2022-05-18T05:41:52.2252371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101725
2022-05-18T05:41:53.1382038Z dist init r=1, world=2
2022-05-18T05:41:53.1385760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:53.1627758Z dist init r=0, world=2
2022-05-18T05:41:53.1632441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:53.1633249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:53.1691272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:54.5333099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:54.5333641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:54.8312120Z skip: Need at least 4 CUDA devices (2.635s)
2022-05-18T05:41:54.8318747Z   test_scatter_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-05-18T05:41:54.8483833Z Tests :meth:`scatter_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101807
2022-05-18T05:41:54.8597043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101808
2022-05-18T05:41:55.7775021Z dist init r=0, world=2
2022-05-18T05:41:55.7778236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:55.7864768Z dist init r=1, world=2
2022-05-18T05:41:55.7869681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:55.7870485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:55.7880866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:57.1576291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:41:57.1576834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:57.4655727Z skip: Need at least 4 CUDA devices (2.634s)
2022-05-18T05:41:57.4665174Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:41:57.4830964Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101890
2022-05-18T05:41:57.4943863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101891
2022-05-18T05:41:58.4103486Z dist init r=1, world=2
2022-05-18T05:41:58.4107289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:41:58.4116967Z dist init r=0, world=2
2022-05-18T05:41:58.4121650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:41:58.4122480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:58.4210434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:41:59.7971587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:41:59.7972449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:42:00.1003427Z skip: Need at least 4 CUDA devices (2.635s)
2022-05-18T05:42:00.1013996Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:42:00.1179261Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101973
2022-05-18T05:42:00.1295044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101974
2022-05-18T05:42:01.1067764Z dist init r=1, world=2
2022-05-18T05:42:01.1068094Z dist init r=0, world=2
2022-05-18T05:42:01.1071065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:42:01.1071965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:42:01.1072817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:01.1073536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:02.4864701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:42:02.4865253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:42:02.7355539Z skip: Need at least 4 CUDA devices (2.635s)
2022-05-18T05:42:02.7365148Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:42:02.7531167Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102056
2022-05-18T05:42:02.7645013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102057
2022-05-18T05:42:03.6884422Z dist init r=0, world=2
2022-05-18T05:42:03.6888029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:42:03.6902926Z dist init r=1, world=2
2022-05-18T05:42:03.6907899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:42:03.6909319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:03.6991235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:05.0714869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:42:05.0715415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:42:05.3705828Z skip: Need at least 4 CUDA devices (2.635s)
2022-05-18T05:42:05.3715763Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:42:05.3878859Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102139
2022-05-18T05:42:05.3992282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102140
2022-05-18T05:42:06.3206611Z dist init r=0, world=2
2022-05-18T05:42:06.3210123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:42:06.3222476Z dist init r=1, world=2
2022-05-18T05:42:06.3227604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:42:06.3228442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:06.3313161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:07.7085259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:42:07.7085777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:42:08.0051449Z skip: Need at least 4 CUDA devices (2.634s)
2022-05-18T05:42:08.0061721Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:42:08.0228594Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102222
2022-05-18T05:42:08.0340646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102223
2022-05-18T05:42:08.9575548Z dist init r=0, world=2
2022-05-18T05:42:08.9579508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:42:09.0044354Z dist init r=1, world=2
2022-05-18T05:42:09.0049202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:42:09.0050277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:09.0087084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:10.3789806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:42:10.3790365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:42:10.6401054Z skip: Need at least 4 CUDA devices (2.635s)
2022-05-18T05:42:10.6410606Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:42:10.6575364Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102305
2022-05-18T05:42:10.6686042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102306
2022-05-18T05:42:11.5842343Z dist init r=0, world=2
2022-05-18T05:42:11.5845948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:42:11.5955036Z dist init r=1, world=2
2022-05-18T05:42:11.5960100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:42:11.5961368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:11.6050397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:12.9856440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:42:12.9856980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:42:13.2745840Z skip: Need at least 4 CUDA devices (2.634s)
2022-05-18T05:42:13.2757244Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:42:13.2921799Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102388
2022-05-18T05:42:13.3033333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102389
2022-05-18T05:42:14.2230657Z dist init r=1, world=2
2022-05-18T05:42:14.2234191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:42:14.2651689Z dist init r=0, world=2
2022-05-18T05:42:14.2656371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:42:14.2657201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:14.2742268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:15.6643603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:42:15.6644154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:42:15.9094482Z skip: Need at least 4 CUDA devices (2.635s)
2022-05-18T05:42:15.9104353Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:42:15.9268722Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102471
2022-05-18T05:42:15.9382773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102472
2022-05-18T05:42:16.8590034Z dist init r=0, world=2
2022-05-18T05:42:16.8593022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:42:16.8699455Z dist init r=1, world=2
2022-05-18T05:42:16.8704199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:42:16.8705284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:16.8797484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:18.2328106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:42:18.2329123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:42:18.5440881Z skip: Need at least 4 CUDA devices (2.634s)
2022-05-18T05:42:18.5447519Z   test_shard_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-05-18T05:42:18.5611733Z Tests :meth:`shard_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102554
2022-05-18T05:42:18.5724366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102555
2022-05-18T05:42:19.4875513Z dist init r=1, world=2
2022-05-18T05:42:19.4878512Z dist init r=0, world=2
2022-05-18T05:42:19.4879021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:42:19.4883492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:42:19.4884337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:19.4982039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:20.8742098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:42:20.8742628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:42:21.1792329Z skip: Need at least 4 CUDA devices (2.635s)
2022-05-18T05:42:21.1810862Z   test_shard_full_optim_state_dict_unmanaged_params_add_to_fsdp_module_False (__main__.TestFSDPOptimState)
2022-05-18T05:42:21.1974620Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102637
2022-05-18T05:42:21.2086945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102638
2022-05-18T05:42:22.0897045Z dist init r=1, world=2
2022-05-18T05:42:22.0900089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:42:22.1264258Z dist init r=0, world=2
2022-05-18T05:42:22.1269269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:42:22.1270071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:22.1306605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:23.5351033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:42:23.5351583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:42:24.2156010Z ok (3.036s)
2022-05-18T05:42:24.2176250Z   test_shard_full_optim_state_dict_unmanaged_params_add_to_fsdp_module_True (__main__.TestFSDPOptimState)
2022-05-18T05:42:24.2339037Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102724
2022-05-18T05:42:24.2454606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102725
2022-05-18T05:42:25.2022930Z dist init r=1, world=2
2022-05-18T05:42:25.2026524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:42:25.2029971Z dist init r=0, world=2
2022-05-18T05:42:25.2035976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:42:25.2037352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:25.2130774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:42:26.5681453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:42:26.5682581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:42:27.2532406Z ok (3.037s)
2022-05-18T05:42:27.2532770Z 
2022-05-18T05:42:27.2533572Z ----------------------------------------------------------------------
2022-05-18T05:42:27.2534085Z Ran 27 tests in 76.554s
2022-05-18T05:42:27.2534326Z 
2022-05-18T05:42:27.2534574Z OK (skipped=18)
2022-05-18T05:42:27.2534919Z 
2022-05-18T05:42:27.2539012Z Generating XML reports...
2022-05-18T05:42:27.2617924Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20220518054110.xml
2022-05-18T05:42:27.5368506Z Running distributed/test_store ... [2022-05-18 05:42:27.536337]
2022-05-18T05:42:27.5369968Z Executing ['/opt/conda/bin/python', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:42:27.536448]
2022-05-18T05:42:28.4288428Z test_compare_set (__main__.FileStoreTest)
2022-05-18T05:42:28.4289155Z test_set_get (__main__.FileStoreTest)
2022-05-18T05:42:28.4290330Z test_compare_set (__main__.HashStoreTest)
2022-05-18T05:42:28.4290898Z test_set_get (__main__.HashStoreTest)
2022-05-18T05:42:28.4291270Z test_compare_set (__main__.PrefixFileStoreTest)
2022-05-18T05:42:28.4291618Z test_set_get (__main__.PrefixFileStoreTest)
2022-05-18T05:42:28.4291960Z test_compare_set (__main__.PrefixTCPStoreTest)
2022-05-18T05:42:28.4292294Z test_set_get (__main__.PrefixTCPStoreTest)
2022-05-18T05:42:28.4292616Z test_set_get (__main__.PythonStoreTest)
2022-05-18T05:42:28.4292943Z test_nominal (__main__.RendezvousEnvTest)
2022-05-18T05:42:28.4293278Z test_common_errors (__main__.RendezvousFileTest)
2022-05-18T05:42:28.4293731Z test_nominal (__main__.RendezvousFileTest)
2022-05-18T05:42:28.4294168Z test_common_errors (__main__.RendezvousTCPTest)
2022-05-18T05:42:28.4294508Z test_dns_timeout (__main__.RendezvousTCPTest)
2022-05-18T05:42:28.4294816Z test_nominal (__main__.RendezvousTCPTest)
2022-05-18T05:42:28.4295489Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest)
2022-05-18T05:42:28.4295840Z test_unknown_handler (__main__.RendezvousTest)
2022-05-18T05:42:28.4296166Z test_address_already_in_use (__main__.TCPStoreTest)
2022-05-18T05:42:28.4296503Z test_compare_set (__main__.TCPStoreTest)
2022-05-18T05:42:28.4296856Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest)
2022-05-18T05:42:28.4297216Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest)
2022-05-18T05:42:28.4297732Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest)
2022-05-18T05:42:28.4298382Z test_multitenancy (__main__.TCPStoreTest)
2022-05-18T05:42:28.4299013Z test_numkeys_delkeys (__main__.TCPStoreTest)
2022-05-18T05:42:28.4299690Z test_set_get (__main__.TCPStoreTest)
2022-05-18T05:42:29.3046693Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:42:29.3063719Z 
2022-05-18T05:42:29.3064153Z Running tests...
2022-05-18T05:42:29.3065052Z ----------------------------------------------------------------------
2022-05-18T05:42:30.9476652Z   test_compare_set (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:42:30.9632238Z ok (1.657s)
2022-05-18T05:42:30.9634003Z 
2022-05-18T05:42:30.9634519Z ----------------------------------------------------------------------
2022-05-18T05:42:30.9634867Z Ran 1 test in 1.657s
2022-05-18T05:42:30.9635036Z 
2022-05-18T05:42:30.9635137Z OK
2022-05-18T05:42:30.9635275Z 
2022-05-18T05:42:30.9635405Z Generating XML reports...
2022-05-18T05:42:30.9666857Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518054229.xml
2022-05-18T05:42:32.0730741Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:42:32.0744887Z 
2022-05-18T05:42:32.0745152Z Running tests...
2022-05-18T05:42:32.0745758Z ----------------------------------------------------------------------
2022-05-18T05:42:33.6921431Z   test_set_get (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:42:33.7063108Z ok (1.632s)
2022-05-18T05:42:33.7064706Z 
2022-05-18T05:42:33.7065722Z ----------------------------------------------------------------------
2022-05-18T05:42:33.7066118Z Ran 1 test in 1.632s
2022-05-18T05:42:33.7066267Z 
2022-05-18T05:42:33.7066363Z OK
2022-05-18T05:42:33.7066509Z 
2022-05-18T05:42:33.7066647Z Generating XML reports...
2022-05-18T05:42:33.7097346Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518054232.xml
2022-05-18T05:42:34.8335648Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:42:34.8350931Z 
2022-05-18T05:42:34.8351175Z Running tests...
2022-05-18T05:42:34.8352032Z ----------------------------------------------------------------------
2022-05-18T05:42:36.4812028Z   test_compare_set (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:42:36.4965616Z ok (1.661s)
2022-05-18T05:42:36.4966832Z 
2022-05-18T05:42:36.4967326Z ----------------------------------------------------------------------
2022-05-18T05:42:36.4967671Z Ran 1 test in 1.662s
2022-05-18T05:42:36.4967846Z 
2022-05-18T05:42:36.4967942Z OK
2022-05-18T05:42:36.4968086Z 
2022-05-18T05:42:36.4968198Z Generating XML reports...
2022-05-18T05:42:36.5008313Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518054234.xml
2022-05-18T05:42:37.6013251Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:42:37.6028972Z 
2022-05-18T05:42:37.6029501Z Running tests...
2022-05-18T05:42:37.6030016Z ----------------------------------------------------------------------
2022-05-18T05:42:39.2660774Z   test_set_get (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:42:39.2799547Z ok (1.677s)
2022-05-18T05:42:39.2800131Z 
2022-05-18T05:42:39.2800497Z ----------------------------------------------------------------------
2022-05-18T05:42:39.2801078Z Ran 1 test in 1.677s
2022-05-18T05:42:39.2801250Z 
2022-05-18T05:42:39.2801346Z OK
2022-05-18T05:42:39.2801487Z 
2022-05-18T05:42:39.2801621Z Generating XML reports...
2022-05-18T05:42:39.2842000Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518054237.xml
2022-05-18T05:42:40.4218965Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:42:40.4234140Z 
2022-05-18T05:42:40.4234436Z Running tests...
2022-05-18T05:42:40.4234879Z ----------------------------------------------------------------------
2022-05-18T05:42:42.0744666Z   test_compare_set (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:42:42.0905506Z ok (1.667s)
2022-05-18T05:42:42.0907909Z 
2022-05-18T05:42:42.0908469Z ----------------------------------------------------------------------
2022-05-18T05:42:42.0908798Z Ran 1 test in 1.667s
2022-05-18T05:42:42.0908975Z 
2022-05-18T05:42:42.0909073Z OK
2022-05-18T05:42:42.0909226Z 
2022-05-18T05:42:42.0909355Z Generating XML reports...
2022-05-18T05:42:42.0949550Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518054240.xml
2022-05-18T05:42:43.2333075Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:42:43.2348725Z 
2022-05-18T05:42:43.2349194Z Running tests...
2022-05-18T05:42:43.2349686Z ----------------------------------------------------------------------
2022-05-18T05:42:44.9124646Z   test_set_get (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:42:44.9273843Z ok (1.692s)
2022-05-18T05:42:44.9276511Z 
2022-05-18T05:42:44.9277221Z ----------------------------------------------------------------------
2022-05-18T05:42:44.9277606Z Ran 1 test in 1.693s
2022-05-18T05:42:44.9277780Z 
2022-05-18T05:42:44.9277865Z OK
2022-05-18T05:42:44.9278006Z 
2022-05-18T05:42:44.9278138Z Generating XML reports...
2022-05-18T05:42:44.9317790Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518054243.xml
2022-05-18T05:42:46.0744707Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:42:46.0759789Z 
2022-05-18T05:42:46.0759937Z Running tests...
2022-05-18T05:42:46.0760661Z ----------------------------------------------------------------------
2022-05-18T05:42:47.7321629Z   test_compare_set (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:42:47.7495366Z ok (1.673s)
2022-05-18T05:42:47.7499625Z 
2022-05-18T05:42:47.7500269Z ----------------------------------------------------------------------
2022-05-18T05:42:47.7500656Z Ran 1 test in 1.674s
2022-05-18T05:42:47.7500830Z 
2022-05-18T05:42:47.7500935Z OK
2022-05-18T05:42:47.7501378Z 
2022-05-18T05:42:47.7501524Z Generating XML reports...
2022-05-18T05:42:47.7541310Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518054246.xml
2022-05-18T05:42:48.8823574Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:42:48.8839209Z 
2022-05-18T05:42:48.8839539Z Running tests...
2022-05-18T05:42:48.8839972Z ----------------------------------------------------------------------
2022-05-18T05:42:50.5163234Z   test_set_get (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:42:50.5327073Z ok (1.649s)
2022-05-18T05:42:50.5330570Z 
2022-05-18T05:42:50.5331179Z ----------------------------------------------------------------------
2022-05-18T05:42:50.5331523Z Ran 1 test in 1.649s
2022-05-18T05:42:50.5331679Z 
2022-05-18T05:42:50.5331775Z OK
2022-05-18T05:42:50.5331914Z 
2022-05-18T05:42:50.5332043Z Generating XML reports...
2022-05-18T05:42:50.5364784Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518054248.xml
2022-05-18T05:42:51.6633579Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:42:51.6648660Z 
2022-05-18T05:42:51.6649132Z Running tests...
2022-05-18T05:42:51.6649890Z ----------------------------------------------------------------------
2022-05-18T05:42:53.3194719Z   test_set_get (__main__.PythonStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:42:53.3316927Z ok (1.667s)
2022-05-18T05:42:53.3317414Z 
2022-05-18T05:42:53.3317984Z ----------------------------------------------------------------------
2022-05-18T05:42:53.3318515Z Ran 1 test in 1.667s
2022-05-18T05:42:53.3318700Z 
2022-05-18T05:42:53.3318801Z OK
2022-05-18T05:42:53.3318942Z 
2022-05-18T05:42:53.3319076Z Generating XML reports...
2022-05-18T05:42:53.3351539Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220518054251.xml
2022-05-18T05:42:54.4705933Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:42:54.4721307Z 
2022-05-18T05:42:54.4721454Z Running tests...
2022-05-18T05:42:54.4722572Z ----------------------------------------------------------------------
2022-05-18T05:42:56.1534463Z   test_nominal (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:42:56.1673517Z ok (1.695s)
2022-05-18T05:42:56.1674208Z 
2022-05-18T05:42:56.1674613Z ----------------------------------------------------------------------
2022-05-18T05:42:56.1674946Z Ran 1 test in 1.695s
2022-05-18T05:42:56.1675111Z 
2022-05-18T05:42:56.1675206Z OK
2022-05-18T05:42:56.1675340Z 
2022-05-18T05:42:56.1675473Z Generating XML reports...
2022-05-18T05:42:56.1707917Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220518054254.xml
2022-05-18T05:42:57.2969214Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:42:57.2985392Z 
2022-05-18T05:42:57.2985902Z Running tests...
2022-05-18T05:42:57.2986403Z ----------------------------------------------------------------------
2022-05-18T05:42:58.9631318Z   test_common_errors (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:42:58.9778504Z ok (1.679s)
2022-05-18T05:42:58.9779328Z 
2022-05-18T05:42:58.9779718Z ----------------------------------------------------------------------
2022-05-18T05:42:58.9780065Z Ran 1 test in 1.679s
2022-05-18T05:42:58.9780231Z 
2022-05-18T05:42:58.9780304Z OK
2022-05-18T05:42:58.9780443Z 
2022-05-18T05:42:58.9780577Z Generating XML reports...
2022-05-18T05:42:58.9821742Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518054257.xml
2022-05-18T05:43:00.1322571Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:00.1338180Z 
2022-05-18T05:43:00.1338474Z Running tests...
2022-05-18T05:43:00.1339215Z ----------------------------------------------------------------------
2022-05-18T05:43:01.7965407Z   test_nominal (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:01.8104913Z ok (1.676s)
2022-05-18T05:43:01.8105611Z 
2022-05-18T05:43:01.8106124Z ----------------------------------------------------------------------
2022-05-18T05:43:01.8106505Z Ran 1 test in 1.677s
2022-05-18T05:43:01.8106685Z 
2022-05-18T05:43:01.8106781Z OK
2022-05-18T05:43:01.8106902Z 
2022-05-18T05:43:01.8107036Z Generating XML reports...
2022-05-18T05:43:01.8140323Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518054300.xml
2022-05-18T05:43:02.9664559Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:02.9679813Z 
2022-05-18T05:43:02.9680164Z Running tests...
2022-05-18T05:43:02.9680940Z ----------------------------------------------------------------------
2022-05-18T05:43:04.6442038Z   test_common_errors (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:04.6569518Z ok (1.689s)
2022-05-18T05:43:04.6570102Z 
2022-05-18T05:43:04.6570962Z ----------------------------------------------------------------------
2022-05-18T05:43:04.6571654Z Ran 1 test in 1.689s
2022-05-18T05:43:04.6571818Z 
2022-05-18T05:43:04.6571913Z OK
2022-05-18T05:43:04.6572051Z 
2022-05-18T05:43:04.6572189Z Generating XML reports...
2022-05-18T05:43:04.6604015Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518054302.xml
2022-05-18T05:43:05.7966661Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:05.7982687Z 
2022-05-18T05:43:05.7983196Z Running tests...
2022-05-18T05:43:05.7983687Z ----------------------------------------------------------------------
2022-05-18T05:43:07.4704002Z   test_dns_timeout (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:07.4925549Z [W socket.cpp:558] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known).
2022-05-18T05:43:07.4926089Z [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456).
2022-05-18T05:43:07.4929224Z ok (1.694s)
2022-05-18T05:43:07.4930439Z 
2022-05-18T05:43:07.4930763Z ----------------------------------------------------------------------
2022-05-18T05:43:07.4931103Z Ran 1 test in 1.695s
2022-05-18T05:43:07.4931275Z 
2022-05-18T05:43:07.4931351Z OK
2022-05-18T05:43:07.4931485Z 
2022-05-18T05:43:07.4931614Z Generating XML reports...
2022-05-18T05:43:07.4964341Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518054305.xml
2022-05-18T05:43:08.5990818Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:08.6007013Z 
2022-05-18T05:43:08.6007269Z Running tests...
2022-05-18T05:43:08.6007721Z ----------------------------------------------------------------------
2022-05-18T05:43:10.2605767Z   test_nominal (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:10.2745748Z ok (1.674s)
2022-05-18T05:43:10.2746239Z 
2022-05-18T05:43:10.2746877Z ----------------------------------------------------------------------
2022-05-18T05:43:10.2747238Z Ran 1 test in 1.674s
2022-05-18T05:43:10.2747417Z 
2022-05-18T05:43:10.2747512Z OK
2022-05-18T05:43:10.2747656Z 
2022-05-18T05:43:10.2747879Z Generating XML reports...
2022-05-18T05:43:10.2780641Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518054308.xml
2022-05-18T05:43:11.4000178Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:11.4015309Z 
2022-05-18T05:43:11.4015637Z Running tests...
2022-05-18T05:43:11.4016367Z ----------------------------------------------------------------------
2022-05-18T05:43:13.0530132Z   test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:23.0733798Z ok (11.672s)
2022-05-18T05:43:23.0733971Z 
2022-05-18T05:43:23.0734398Z ----------------------------------------------------------------------
2022-05-18T05:43:23.0734746Z Ran 1 test in 11.672s
2022-05-18T05:43:23.0734913Z 
2022-05-18T05:43:23.0735007Z OK
2022-05-18T05:43:23.0735141Z 
2022-05-18T05:43:23.0735256Z Generating XML reports...
2022-05-18T05:43:23.0770395Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518054311.xml
2022-05-18T05:43:24.2257187Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:24.2272074Z 
2022-05-18T05:43:24.2272555Z Running tests...
2022-05-18T05:43:24.2273046Z ----------------------------------------------------------------------
2022-05-18T05:43:25.8862943Z   test_unknown_handler (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:25.8987289Z ok (1.671s)
2022-05-18T05:43:25.8987782Z 
2022-05-18T05:43:25.8988151Z ----------------------------------------------------------------------
2022-05-18T05:43:25.8988488Z Ran 1 test in 1.672s
2022-05-18T05:43:25.8988963Z 
2022-05-18T05:43:25.8989058Z OK
2022-05-18T05:43:25.8989195Z 
2022-05-18T05:43:25.8989325Z Generating XML reports...
2022-05-18T05:43:25.9021632Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220518054324.xml
2022-05-18T05:43:26.9960756Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:26.9976316Z 
2022-05-18T05:43:26.9976697Z Running tests...
2022-05-18T05:43:26.9977187Z ----------------------------------------------------------------------
2022-05-18T05:43:28.6513838Z   test_address_already_in_use (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:28.6653108Z [W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:41117 (errno: 98 - Address already in use).
2022-05-18T05:43:28.6671822Z [W socket.cpp:401] [c10d] The server socket has failed to bind to 0.0.0.0:41117 (errno: 98 - Address already in use).
2022-05-18T05:43:28.6672285Z [E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address.
2022-05-18T05:43:28.6677308Z ok (1.670s)
2022-05-18T05:43:28.6678345Z 
2022-05-18T05:43:28.6678644Z ----------------------------------------------------------------------
2022-05-18T05:43:28.6678995Z Ran 1 test in 1.670s
2022-05-18T05:43:28.6679161Z 
2022-05-18T05:43:28.6679236Z OK
2022-05-18T05:43:28.6679367Z 
2022-05-18T05:43:28.6679495Z Generating XML reports...
2022-05-18T05:43:28.6713514Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054326.xml
2022-05-18T05:43:29.7971693Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:29.7986872Z 
2022-05-18T05:43:29.7987312Z Running tests...
2022-05-18T05:43:29.7987813Z ----------------------------------------------------------------------
2022-05-18T05:43:31.4675701Z   test_compare_set (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:31.4843369Z ok (1.685s)
2022-05-18T05:43:31.4843669Z 
2022-05-18T05:43:31.4844053Z ----------------------------------------------------------------------
2022-05-18T05:43:31.4844385Z Ran 1 test in 1.686s
2022-05-18T05:43:31.4844557Z 
2022-05-18T05:43:31.4844650Z OK
2022-05-18T05:43:31.4844767Z 
2022-05-18T05:43:31.4844894Z Generating XML reports...
2022-05-18T05:43:31.4877127Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054329.xml
2022-05-18T05:43:32.6116327Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:32.6131866Z 
2022-05-18T05:43:32.6132217Z Running tests...
2022-05-18T05:43:32.6132724Z ----------------------------------------------------------------------
2022-05-18T05:43:34.2648438Z   test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:34.2787970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:43:34.2788785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:43:34.3529439Z ok (1.740s)
2022-05-18T05:43:34.3530076Z 
2022-05-18T05:43:34.3530682Z ----------------------------------------------------------------------
2022-05-18T05:43:34.3531036Z Ran 1 test in 1.740s
2022-05-18T05:43:34.3531207Z 
2022-05-18T05:43:34.3531296Z OK
2022-05-18T05:43:34.3531444Z 
2022-05-18T05:43:34.3531572Z Generating XML reports...
2022-05-18T05:43:34.3563467Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054332.xml
2022-05-18T05:43:35.4873582Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:35.4890144Z 
2022-05-18T05:43:35.4890587Z Running tests...
2022-05-18T05:43:35.4891081Z ----------------------------------------------------------------------
2022-05-18T05:43:37.1460263Z   test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:37.1662752Z ok (1.677s)
2022-05-18T05:43:37.1663512Z 
2022-05-18T05:43:37.1663904Z ----------------------------------------------------------------------
2022-05-18T05:43:37.1664253Z Ran 1 test in 1.677s
2022-05-18T05:43:37.1664420Z 
2022-05-18T05:43:37.1664515Z OK
2022-05-18T05:43:37.1664651Z 
2022-05-18T05:43:37.1664764Z Generating XML reports...
2022-05-18T05:43:37.1698661Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054335.xml
2022-05-18T05:43:38.3224677Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:38.3239528Z 
2022-05-18T05:43:38.3239818Z Running tests...
2022-05-18T05:43:38.3240269Z ----------------------------------------------------------------------
2022-05-18T05:43:39.9983869Z   test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:40.0130993Z ok (1.689s)
2022-05-18T05:43:40.0133123Z 
2022-05-18T05:43:40.0133787Z ----------------------------------------------------------------------
2022-05-18T05:43:40.0134152Z Ran 1 test in 1.689s
2022-05-18T05:43:40.0134301Z 
2022-05-18T05:43:40.0134404Z OK
2022-05-18T05:43:40.0134538Z 
2022-05-18T05:43:40.0134670Z Generating XML reports...
2022-05-18T05:43:40.0166411Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054338.xml
2022-05-18T05:43:41.1364435Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:41.1379894Z 
2022-05-18T05:43:41.1380393Z Running tests...
2022-05-18T05:43:41.1380893Z ----------------------------------------------------------------------
2022-05-18T05:43:42.8021646Z   test_multitenancy (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:42.8164919Z ok (1.678s)
2022-05-18T05:43:42.8165227Z 
2022-05-18T05:43:42.8165698Z ----------------------------------------------------------------------
2022-05-18T05:43:42.8166056Z Ran 1 test in 1.679s
2022-05-18T05:43:42.8166221Z 
2022-05-18T05:43:42.8166323Z OK
2022-05-18T05:43:42.8166458Z 
2022-05-18T05:43:42.8166586Z Generating XML reports...
2022-05-18T05:43:42.8200573Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054341.xml
2022-05-18T05:43:43.9480829Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:43.9495456Z 
2022-05-18T05:43:43.9495600Z Running tests...
2022-05-18T05:43:43.9496308Z ----------------------------------------------------------------------
2022-05-18T05:43:45.5974474Z   test_numkeys_delkeys (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:47.6187860Z ok (3.669s)
2022-05-18T05:43:47.6188575Z 
2022-05-18T05:43:47.6189223Z ----------------------------------------------------------------------
2022-05-18T05:43:47.6189866Z Ran 1 test in 3.669s
2022-05-18T05:43:47.6190182Z 
2022-05-18T05:43:47.6190359Z OK
2022-05-18T05:43:47.6190620Z 
2022-05-18T05:43:47.6190868Z Generating XML reports...
2022-05-18T05:43:47.6225645Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054343.xml
2022-05-18T05:43:48.7659594Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:43:48.7674553Z 
2022-05-18T05:43:48.7674989Z Running tests...
2022-05-18T05:43:48.7675472Z ----------------------------------------------------------------------
2022-05-18T05:43:50.4230435Z   test_set_get (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:50.4397399Z ok (1.672s)
2022-05-18T05:43:50.4398090Z 
2022-05-18T05:43:50.4398503Z ----------------------------------------------------------------------
2022-05-18T05:43:50.4398847Z Ran 1 test in 1.672s
2022-05-18T05:43:50.4398995Z 
2022-05-18T05:43:50.4399087Z OK
2022-05-18T05:43:50.4399225Z 
2022-05-18T05:43:50.4399351Z Generating XML reports...
2022-05-18T05:43:50.4433231Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054348.xml
2022-05-18T05:43:50.8275541Z Running distributed/test_pg_wrapper ... [2022-05-18 05:43:50.827026]
2022-05-18T05:43:50.8276275Z Executing ['/opt/conda/bin/python', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:43:50.827139]
2022-05-18T05:43:51.7257119Z <unittest.suite.TestSuite tests=[]>
2022-05-18T05:43:51.7257831Z <unittest.suite.TestSuite tests=[]>
2022-05-18T05:43:51.7259806Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-05-18T05:43:51.7262684Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T05:43:51.7263530Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T05:43:51.7264391Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T05:43:51.7265152Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T05:43:51.7265893Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T05:43:51.7266580Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T05:43:51.7267234Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T05:43:51.7267944Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T05:43:51.7268663Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T05:43:51.7270250Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-05-18T05:43:51.7272022Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T05:43:51.7272684Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T05:43:51.7273389Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T05:43:51.7274076Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T05:43:51.7274747Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T05:43:52.6198927Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:43:52.6215497Z 
2022-05-18T05:43:52.6215918Z Running tests...
2022-05-18T05:43:52.6216390Z ----------------------------------------------------------------------
2022-05-18T05:43:54.2866336Z   test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:54.3267979Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103898
2022-05-18T05:43:54.3388161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103899
2022-05-18T05:43:54.3511024Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103900
2022-05-18T05:43:54.3636096Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103901
2022-05-18T05:43:55.3281710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:43:55.3330127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:43:55.3352695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:43:55.3353206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:43:55.3542309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:43:55.3643933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:43:55.3667027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T05:43:55.3667584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T05:43:55.3668632Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:43:55.3669587Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:43:55.3746115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:43:55.3746818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:43:55.4512804Z [E ProcessGroupGloo.cpp:2791] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T05:43:55.4525860Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T05:43:55.4628804Z [E ProcessGroupGloo.cpp:136] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-05-18T05:43:55.4731233Z [E ProcessGroupGloo.cpp:136] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-05-18T05:43:55.7684006Z ok (3.147s)
2022-05-18T05:43:55.7684237Z 
2022-05-18T05:43:55.7684847Z ----------------------------------------------------------------------
2022-05-18T05:43:55.7685214Z Ran 1 test in 3.147s
2022-05-18T05:43:55.7685517Z 
2022-05-18T05:43:55.7685663Z OK
2022-05-18T05:43:55.7685887Z 
2022-05-18T05:43:55.7686026Z Generating XML reports...
2022-05-18T05:43:55.7731389Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054352.xml
2022-05-18T05:43:56.9148429Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:43:56.9163556Z 
2022-05-18T05:43:56.9164061Z Running tests...
2022-05-18T05:43:56.9164573Z ----------------------------------------------------------------------
2022-05-18T05:43:58.5815540Z   test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:43:58.6217941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104115
2022-05-18T05:43:58.6339612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104116
2022-05-18T05:43:58.6464942Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104117
2022-05-18T05:43:58.6597961Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104118
2022-05-18T05:43:59.6192461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:43:59.6298468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:43:59.6341729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:43:59.6881337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:43:59.7058436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:43:59.7058969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T05:43:59.7161432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:43:59.7162425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T05:43:59.7163810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:43:59.7165151Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:43:59.7166566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:43:59.7167958Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:44:00.0643324Z ok (3.148s)
2022-05-18T05:44:00.0643514Z 
2022-05-18T05:44:00.0643915Z ----------------------------------------------------------------------
2022-05-18T05:44:00.0644265Z Ran 1 test in 3.148s
2022-05-18T05:44:00.0644429Z 
2022-05-18T05:44:00.0644528Z OK
2022-05-18T05:44:00.0644664Z 
2022-05-18T05:44:00.0644782Z Generating XML reports...
2022-05-18T05:44:00.0696449Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054356.xml
2022-05-18T05:44:01.2182749Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:44:01.2197150Z 
2022-05-18T05:44:01.2197592Z Running tests...
2022-05-18T05:44:01.2198081Z ----------------------------------------------------------------------
2022-05-18T05:44:02.8468735Z   test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:44:02.8861916Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104332
2022-05-18T05:44:02.8980329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104333
2022-05-18T05:44:02.9099941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104334
2022-05-18T05:44:02.9224189Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104335
2022-05-18T05:44:03.8316562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:44:03.8436868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:44:03.8447228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:44:03.8923795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:44:04.0265419Z skip: Need at least 4 CUDA devices (2.806s)
2022-05-18T05:44:04.0265944Z 
2022-05-18T05:44:04.0266669Z ----------------------------------------------------------------------
2022-05-18T05:44:04.0267040Z Ran 1 test in 2.807s
2022-05-18T05:44:04.0267189Z 
2022-05-18T05:44:04.0267301Z OK (skipped=1)
2022-05-18T05:44:04.0267458Z 
2022-05-18T05:44:04.0267588Z Generating XML reports...
2022-05-18T05:44:04.0310890Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054401.xml
2022-05-18T05:44:05.1943582Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:44:05.1958360Z 
2022-05-18T05:44:05.1958793Z Running tests...
2022-05-18T05:44:05.1959317Z ----------------------------------------------------------------------
2022-05-18T05:44:06.8533030Z   test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:44:06.8937904Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104513
2022-05-18T05:44:06.9058978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104514
2022-05-18T05:44:06.9181779Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104515
2022-05-18T05:44:06.9309752Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104516
2022-05-18T05:44:07.8871997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:44:07.8885960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:44:07.8902912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:44:07.8930941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:44:08.1352233Z skip: Need at least 4 CUDA devices (2.939s)
2022-05-18T05:44:08.1352710Z 
2022-05-18T05:44:08.1353363Z ----------------------------------------------------------------------
2022-05-18T05:44:08.1353994Z Ran 1 test in 2.939s
2022-05-18T05:44:08.1354262Z 
2022-05-18T05:44:08.1354462Z OK (skipped=1)
2022-05-18T05:44:08.1354741Z 
2022-05-18T05:44:08.1354985Z Generating XML reports...
2022-05-18T05:44:08.1398448Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054405.xml
2022-05-18T05:44:09.2987407Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:44:09.3002115Z 
2022-05-18T05:44:09.3002365Z Running tests...
2022-05-18T05:44:09.3002822Z ----------------------------------------------------------------------
2022-05-18T05:44:10.9579964Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:44:10.9982729Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104694
2022-05-18T05:44:11.0102442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104695
2022-05-18T05:44:11.0229084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104696
2022-05-18T05:44:11.0358466Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104697
2022-05-18T05:44:11.9195367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:44:11.9600738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:44:11.9808427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:44:11.9952507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:44:12.0526853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:44:12.0626668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:44:12.0729246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T05:44:12.0729757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T05:44:12.0730809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:44:12.0731495Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:44:12.0732204Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:44:12.0831396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:44:12.1351649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:44:12.1455330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:44:12.1555452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-05-18T05:44:12.1555958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T05:44:12.1556655Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T05:44:12.1557337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T05:44:12.1558037Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T05:44:12.1558740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T05:44:12.5407752Z ok (3.240s)
2022-05-18T05:44:12.5407975Z 
2022-05-18T05:44:12.5408377Z ----------------------------------------------------------------------
2022-05-18T05:44:12.5408720Z Ran 1 test in 3.241s
2022-05-18T05:44:12.5408885Z 
2022-05-18T05:44:12.5408979Z OK
2022-05-18T05:44:12.5409096Z 
2022-05-18T05:44:12.5409228Z Generating XML reports...
2022-05-18T05:44:12.5452906Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054409.xml
2022-05-18T05:44:13.7080779Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:44:13.7095118Z 
2022-05-18T05:44:13.7095424Z Running tests...
2022-05-18T05:44:13.7095871Z ----------------------------------------------------------------------
2022-05-18T05:44:15.3377101Z   test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:44:15.3772141Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104923
2022-05-18T05:44:15.3889358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104924
2022-05-18T05:44:15.4011598Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104925
2022-05-18T05:44:15.4136095Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104926
2022-05-18T05:44:16.3515308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:44:16.3843712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:44:16.3976792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:44:16.4057291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:44:16.4165885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:44:16.4333988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T05:44:16.4334512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:44:16.4335004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T05:44:16.4335795Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:44:16.4336504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:44:16.4337207Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:44:16.4370239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:44:16.8181852Z ok (3.108s)
2022-05-18T05:44:16.8182106Z 
2022-05-18T05:44:16.8182506Z ----------------------------------------------------------------------
2022-05-18T05:44:16.8182850Z Ran 1 test in 3.109s
2022-05-18T05:44:16.8183021Z 
2022-05-18T05:44:16.8183118Z OK
2022-05-18T05:44:16.8183258Z 
2022-05-18T05:44:16.8183376Z Generating XML reports...
2022-05-18T05:44:16.8226475Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054413.xml
2022-05-18T05:44:17.9909900Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:44:17.9925821Z 
2022-05-18T05:44:17.9925963Z Running tests...
2022-05-18T05:44:17.9926824Z ----------------------------------------------------------------------
2022-05-18T05:44:19.6503321Z   test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:44:19.6897113Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105140
2022-05-18T05:44:19.7015223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105141
2022-05-18T05:44:19.7138376Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105142
2022-05-18T05:44:19.7262249Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105143
2022-05-18T05:44:20.5885508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:44:20.6164291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:44:20.6687518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:44:20.6872240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:44:20.8304686Z skip: Need at least 4 CUDA devices (2.838s)
2022-05-18T05:44:20.8305584Z 
2022-05-18T05:44:20.8305988Z ----------------------------------------------------------------------
2022-05-18T05:44:20.8306358Z Ran 1 test in 2.838s
2022-05-18T05:44:20.8306523Z 
2022-05-18T05:44:20.8306615Z OK (skipped=1)
2022-05-18T05:44:20.8306777Z 
2022-05-18T05:44:20.8306907Z Generating XML reports...
2022-05-18T05:44:20.8357531Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054417.xml
2022-05-18T05:44:22.0022084Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:44:22.0039729Z 
2022-05-18T05:44:22.0040224Z Running tests...
2022-05-18T05:44:22.0040732Z ----------------------------------------------------------------------
2022-05-18T05:44:23.6571648Z   test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:44:23.6966451Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105321
2022-05-18T05:44:23.7084773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105322
2022-05-18T05:44:23.7208316Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105323
2022-05-18T05:44:23.7336099Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105324
2022-05-18T05:44:24.6232557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:44:24.6547332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:44:24.6788593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:44:24.6809094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:44:24.8376820Z skip: Need at least 4 CUDA devices (2.834s)
2022-05-18T05:44:24.8377247Z 
2022-05-18T05:44:24.8377613Z ----------------------------------------------------------------------
2022-05-18T05:44:24.8377956Z Ran 1 test in 2.834s
2022-05-18T05:44:24.8378344Z 
2022-05-18T05:44:24.8378464Z OK (skipped=1)
2022-05-18T05:44:24.8378631Z 
2022-05-18T05:44:24.8378762Z Generating XML reports...
2022-05-18T05:44:24.8423574Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054422.xml
2022-05-18T05:44:26.0087166Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:44:26.0103214Z 
2022-05-18T05:44:26.0103369Z Running tests...
2022-05-18T05:44:26.0103824Z ----------------------------------------------------------------------
2022-05-18T05:44:27.6606514Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:44:27.7012022Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105502
2022-05-18T05:44:27.7134526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105503
2022-05-18T05:44:27.7259744Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105504
2022-05-18T05:44:27.7387252Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105505
2022-05-18T05:44:28.6195146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:44:28.6588949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:44:28.6708058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:44:28.6793418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:44:28.7437257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:44:28.7540286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:44:28.7642516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T05:44:28.7643061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T05:44:28.7643831Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:44:28.7644539Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:44:28.7743675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:44:28.7744378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:44:28.8466619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:44:28.8668123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T05:44:28.8668657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:44:28.8669725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-05-18T05:44:28.8670434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T05:44:28.8671191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T05:44:28.8672014Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T05:44:28.8771453Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T05:44:29.2438616Z ok (3.233s)
2022-05-18T05:44:29.2438814Z 
2022-05-18T05:44:29.2439197Z ----------------------------------------------------------------------
2022-05-18T05:44:29.2439866Z Ran 1 test in 3.233s
2022-05-18T05:44:29.2440038Z 
2022-05-18T05:44:29.2440133Z OK
2022-05-18T05:44:29.2440279Z 
2022-05-18T05:44:29.2440415Z Generating XML reports...
2022-05-18T05:44:29.2485632Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054426.xml
2022-05-18T05:44:30.4249410Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:44:30.4264397Z 
2022-05-18T05:44:30.4264902Z Running tests...
2022-05-18T05:44:30.4265388Z ----------------------------------------------------------------------
2022-05-18T05:44:32.0533788Z   test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:44:32.0930690Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105731
2022-05-18T05:44:32.1048241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105732
2022-05-18T05:44:33.0307046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:44:33.0309400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:44:33.0516735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:44:33.0520164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:44:33.0521189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:44:33.0617299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:44:33.0829336Z [E ProcessGroupGloo.cpp:2791] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T05:44:33.0829843Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T05:44:33.3085924Z ok (2.882s)
2022-05-18T05:44:33.3086133Z 
2022-05-18T05:44:33.3086717Z ----------------------------------------------------------------------
2022-05-18T05:44:33.3087107Z Ran 1 test in 2.882s
2022-05-18T05:44:33.3087278Z 
2022-05-18T05:44:33.3087374Z OK
2022-05-18T05:44:33.3087513Z 
2022-05-18T05:44:33.3087653Z Generating XML reports...
2022-05-18T05:44:33.3130938Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518054430.xml
2022-05-18T05:44:34.4755918Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:44:34.4771010Z 
2022-05-18T05:44:34.4771503Z Running tests...
2022-05-18T05:44:34.4772251Z ----------------------------------------------------------------------
2022-05-18T05:44:36.1345034Z   test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:44:36.1737525Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105850
2022-05-18T05:44:36.1855603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105851
2022-05-18T05:44:37.0341893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:44:37.0344552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:44:37.0359145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:44:37.0362754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:44:37.0363899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:44:37.0447221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:44:38.7930457Z ok (4.316s)
2022-05-18T05:44:38.7931122Z 
2022-05-18T05:44:38.7931524Z ----------------------------------------------------------------------
2022-05-18T05:44:38.7931993Z Ran 1 test in 4.316s
2022-05-18T05:44:38.7932280Z 
2022-05-18T05:44:38.7932376Z OK
2022-05-18T05:44:38.7932512Z 
2022-05-18T05:44:38.7932628Z Generating XML reports...
2022-05-18T05:44:38.7975669Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518054434.xml
2022-05-18T05:44:39.9825950Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:44:39.9840924Z 
2022-05-18T05:44:39.9841391Z Running tests...
2022-05-18T05:44:39.9841886Z ----------------------------------------------------------------------
2022-05-18T05:44:41.6423477Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:44:41.6818797Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105985
2022-05-18T05:44:41.6936861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105986
2022-05-18T05:44:42.5847388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:44:42.6304230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:44:42.6463319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:44:42.6463921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:44:42.6465012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:44:42.6466029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:44:42.6570838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:44:42.6571364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:44:42.6572033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:44:42.6572729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:44:44.4002499Z ok (4.416s)
2022-05-18T05:44:44.4002877Z 
2022-05-18T05:44:44.4003637Z ----------------------------------------------------------------------
2022-05-18T05:44:44.4004140Z Ran 1 test in 4.416s
2022-05-18T05:44:44.4004313Z 
2022-05-18T05:44:44.4004390Z OK
2022-05-18T05:44:44.4004530Z 
2022-05-18T05:44:44.4004966Z Generating XML reports...
2022-05-18T05:44:44.4047749Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518054439.xml
2022-05-18T05:44:45.5812159Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:44:45.5827790Z 
2022-05-18T05:44:45.5828137Z Running tests...
2022-05-18T05:44:45.5828648Z ----------------------------------------------------------------------
2022-05-18T05:44:47.2494909Z   test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:44:47.2892509Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106126
2022-05-18T05:44:47.3010806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106127
2022-05-18T05:44:48.1880835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:44:48.1883109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:44:48.1891204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:44:48.1894688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:44:48.1896088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:44:48.1987527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:44:51.0096611Z ok (5.427s)
2022-05-18T05:44:51.0096836Z 
2022-05-18T05:44:51.0097230Z ----------------------------------------------------------------------
2022-05-18T05:44:51.0097570Z Ran 1 test in 5.427s
2022-05-18T05:44:51.0097717Z 
2022-05-18T05:44:51.0097823Z OK
2022-05-18T05:44:51.0097959Z 
2022-05-18T05:44:51.0098095Z Generating XML reports...
2022-05-18T05:44:51.0141817Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518054445.xml
2022-05-18T05:44:52.1978820Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T05:44:52.1993395Z 
2022-05-18T05:44:52.1993734Z Running tests...
2022-05-18T05:44:52.1994454Z ----------------------------------------------------------------------
2022-05-18T05:44:53.8709357Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:44:53.9114421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106262
2022-05-18T05:44:53.9235640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106263
2022-05-18T05:44:54.8264434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:44:54.8621295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:44:54.8779226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:44:54.8779868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:44:54.8780659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:44:54.8781358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:44:54.8888177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:44:54.8888728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:44:54.8889420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:44:54.8890703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:44:57.6328649Z ok (5.433s)
2022-05-18T05:44:57.6328864Z 
2022-05-18T05:44:57.6329277Z ----------------------------------------------------------------------
2022-05-18T05:44:57.6329874Z Ran 1 test in 5.433s
2022-05-18T05:44:57.6330026Z 
2022-05-18T05:44:57.6330117Z OK
2022-05-18T05:44:57.6330257Z 
2022-05-18T05:44:57.6330391Z Generating XML reports...
2022-05-18T05:44:57.6373112Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518054452.xml
2022-05-18T05:44:58.0487898Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2022-05-18 05:44:58.048097]
2022-05-18T05:44:58.0540013Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:44:58.048214]
2022-05-18T05:44:58.9815673Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm
2022-05-18T05:44:58.9834730Z 
2022-05-18T05:44:58.9835033Z Running tests...
2022-05-18T05:44:58.9835471Z ----------------------------------------------------------------------
2022-05-18T05:44:58.9844929Z   test_fsdp_calc_grad_norm_error_norm_type_1_3 (__main__.TestCalcuGradNorm)
2022-05-18T05:45:00.6484495Z Test the abnormal cases of grad norm cal API. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:45:00.6902945Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106404
2022-05-18T05:45:00.7030490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106405
2022-05-18T05:45:01.6390229Z dist init r=1, world=2
2022-05-18T05:45:01.6393846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:01.6475068Z dist init r=0, world=2
2022-05-18T05:45:01.6480026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:01.6480987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:01.6496857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:03.0511968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:03.0512526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:03.7105436Z ok (4.727s)
2022-05-18T05:45:03.7115012Z   test_fsdp_calc_grad_norm_error_norm_type_2_5 (__main__.TestCalcuGradNorm)
2022-05-18T05:45:03.7278712Z Test the abnormal cases of grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106491
2022-05-18T05:45:03.7392258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106492
2022-05-18T05:45:04.6570158Z dist init r=0, world=2
2022-05-18T05:45:04.6573379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:04.6577874Z dist init r=1, world=2
2022-05-18T05:45:04.6582385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:04.6583503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:04.6676706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:06.0267773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:06.0268321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:06.6459537Z ok (2.935s)
2022-05-18T05:45:06.6469402Z   test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-05-18T05:45:06.6633928Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106578
2022-05-18T05:45:06.6745210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106579
2022-05-18T05:45:07.5850540Z dist init r=0, world=2
2022-05-18T05:45:07.5853504Z dist init r=1, world=2
2022-05-18T05:45:07.5853902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:07.5857787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:07.5858661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:07.5957433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:08.9739389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:08.9739975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:09.5810641Z ok (2.935s)
2022-05-18T05:45:09.5820966Z   test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-05-18T05:45:09.5985013Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106665
2022-05-18T05:45:09.6096568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106666
2022-05-18T05:45:10.5271650Z dist init r=0, world=2
2022-05-18T05:45:10.5275036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:10.5277936Z dist init r=1, world=2
2022-05-18T05:45:10.5282530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:10.5283348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:10.5377865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:11.8967709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:11.8968274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:12.5160932Z ok (2.935s)
2022-05-18T05:45:12.5170751Z   test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-05-18T05:45:12.5335307Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106752
2022-05-18T05:45:12.5445879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106753
2022-05-18T05:45:13.4607591Z dist init r=0, world=2
2022-05-18T05:45:13.4610195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:13.4616138Z dist init r=1, world=2
2022-05-18T05:45:13.4620859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:13.4621975Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:13.4713627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:14.8415260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:14.8415785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:15.1484545Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:15.1485619Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:45:15.1563493Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:15.1564238Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:45:15.4511417Z ok (2.935s)
2022-05-18T05:45:15.4522420Z   test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-05-18T05:45:15.4691375Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106839
2022-05-18T05:45:15.4804479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106840
2022-05-18T05:45:16.3983355Z dist init r=0, world=2
2022-05-18T05:45:16.3986634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:16.4134710Z dist init r=1, world=2
2022-05-18T05:45:16.4139795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:16.4141433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:16.4191677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:17.7954488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:17.7955488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:18.1127095Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:18.1127908Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:45:18.1131066Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:18.1131821Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:45:18.3871815Z ok (2.936s)
2022-05-18T05:45:18.3881162Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-05-18T05:45:18.4044463Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106926
2022-05-18T05:45:18.4155978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106927
2022-05-18T05:45:19.3291581Z dist init r=0, world=2
2022-05-18T05:45:19.3295095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:19.3308166Z dist init r=1, world=2
2022-05-18T05:45:19.3312795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:19.3313922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:19.3397992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:20.7226607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:20.7227143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:21.0244158Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:21.0245051Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:21.0297903Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:21.0298579Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:21.0344805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:21.0345489Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:21.0346378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:21.0347204Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:21.6226763Z ok (3.235s)
2022-05-18T05:45:21.6235043Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-05-18T05:45:21.6400063Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107013
2022-05-18T05:45:21.6511368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107014
2022-05-18T05:45:22.5602234Z dist init r=1, world=2
2022-05-18T05:45:22.5605663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:22.5726341Z dist init r=0, world=2
2022-05-18T05:45:22.5731275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:22.5732067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:22.5810105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:23.9615904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:23.9616471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:24.2644763Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:24.2645503Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:24.2710018Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:24.2710671Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:24.2768498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:24.2769453Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:24.2771259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:24.2771943Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:24.8582990Z ok (3.235s)
2022-05-18T05:45:24.8591244Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-05-18T05:45:24.8755162Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107100
2022-05-18T05:45:24.8868996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107101
2022-05-18T05:45:25.7920984Z dist init r=0, world=2
2022-05-18T05:45:25.7923929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:25.7958460Z dist init r=1, world=2
2022-05-18T05:45:25.7963317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:25.7964363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:25.8027321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:27.1781822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:27.1782350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:27.4870933Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:27.4871680Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:27.4872369Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:27.4873006Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:27.4939268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:27.4939958Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:27.4940869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:27.4941539Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:28.0939249Z ok (3.235s)
2022-05-18T05:45:28.0947366Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-05-18T05:45:28.1111044Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107187
2022-05-18T05:45:28.1223182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107188
2022-05-18T05:45:29.0680004Z dist init r=1, world=2
2022-05-18T05:45:29.0683032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:29.0699981Z dist init r=0, world=2
2022-05-18T05:45:29.0704326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:29.0705543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:29.0785916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:30.4427225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:30.4427766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:30.7453619Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:30.7454378Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:30.7456737Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:30.7457711Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:30.7540715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:30.7541393Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:30.7542304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:30.7542971Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:31.3294789Z ok (3.235s)
2022-05-18T05:45:31.3302959Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-05-18T05:45:31.3467175Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107274
2022-05-18T05:45:31.3578045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107275
2022-05-18T05:45:32.2719051Z dist init r=1, world=2
2022-05-18T05:45:32.2719369Z dist init r=0, world=2
2022-05-18T05:45:32.2722779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:32.2723368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:32.2724173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:32.2724891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:33.6389079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:33.6389605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:33.9386587Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:33.9387457Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:33.9417349Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:33.9418111Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:33.9464065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:33.9464756Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:33.9465638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:33.9466303Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:33.9496926Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:33.9497818Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:45:33.9498782Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:33.9499495Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:45:34.2645138Z ok (2.935s)
2022-05-18T05:45:34.2653852Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-05-18T05:45:34.2818592Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107361
2022-05-18T05:45:34.2929156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107362
2022-05-18T05:45:35.2045564Z dist init r=0, world=2
2022-05-18T05:45:35.2048296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:35.2066365Z dist init r=1, world=2
2022-05-18T05:45:35.2071135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:35.2072261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:35.2151807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:36.6040309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:36.6040873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:36.9025702Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:36.9026443Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:36.9095843Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:36.9096540Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:36.9150264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:36.9150970Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:36.9151872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:36.9152526Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:36.9181466Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:36.9182370Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:45:36.9183350Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:36.9184062Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:45:37.1994496Z ok (2.935s)
2022-05-18T05:45:37.2002985Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-05-18T05:45:37.2170172Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107448
2022-05-18T05:45:37.2282341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107449
2022-05-18T05:45:38.1385383Z dist init r=0, world=2
2022-05-18T05:45:38.1388489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:38.1491821Z dist init r=1, world=2
2022-05-18T05:45:38.1497198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:38.1497996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:38.1592539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:39.5348643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:39.5349668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:39.8380867Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:39.8382171Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:39.8383480Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:39.8384756Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:39.8454221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:39.8455552Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:39.8457275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:39.8458518Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:39.8486968Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:39.8488395Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:45:39.8490632Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:39.8492238Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:45:40.1349882Z ok (2.935s)
2022-05-18T05:45:40.1357835Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-05-18T05:45:40.1522818Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107535
2022-05-18T05:45:40.1634700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107536
2022-05-18T05:45:41.0753898Z dist init r=0, world=2
2022-05-18T05:45:41.0756890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:41.0784714Z dist init r=1, world=2
2022-05-18T05:45:41.0789701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:41.0790840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:41.0859872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:42.4715013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:42.4715581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:42.7733492Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:42.7734231Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:42.7791578Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:42.7792215Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:45:42.7874195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:42.7874874Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:42.7876086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:42.7876782Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:45:42.7906590Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:42.7907332Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:45:42.7909848Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:45:42.7910568Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:45:43.0702441Z ok (2.935s)
2022-05-18T05:45:43.0702804Z 
2022-05-18T05:45:43.0703463Z ----------------------------------------------------------------------
2022-05-18T05:45:43.0704090Z Ran 14 tests in 44.087s
2022-05-18T05:45:43.0704379Z 
2022-05-18T05:45:43.0704663Z OK
2022-05-18T05:45:43.0704914Z 
2022-05-18T05:45:43.0705141Z Generating XML reports...
2022-05-18T05:45:43.0756409Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220518054458.xml
2022-05-18T05:45:43.0770426Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220518054458.xml
2022-05-18T05:45:43.3525809Z Running distributed/fsdp/test_fsdp_grad_acc ... [2022-05-18 05:45:43.352062]
2022-05-18T05:45:43.3526552Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:45:43.352180]
2022-05-18T05:45:44.2873004Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc
2022-05-18T05:45:44.2891130Z 
2022-05-18T05:45:44.2891273Z Running tests...
2022-05-18T05:45:44.2892130Z ----------------------------------------------------------------------
2022-05-18T05:45:44.2908955Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc)
2022-05-18T05:45:45.9378523Z Tests gradient accumulation. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:45:45.9787846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107659
2022-05-18T05:45:45.9908911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107660
2022-05-18T05:45:46.9419368Z dist init r=0, world=2
2022-05-18T05:45:46.9423068Z dist init r=1, world=2
2022-05-18T05:45:46.9430374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:46.9432020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:46.9435988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:46.9437271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:48.3620634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:48.3621651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:48.3934516Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:45:48.3935797Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:45:48.3968259Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:45:48.3969923Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:45:49.0982511Z ok (4.809s)
2022-05-18T05:45:49.0998778Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc)
2022-05-18T05:45:49.1161903Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107746
2022-05-18T05:45:49.1274624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107747
2022-05-18T05:45:50.0460216Z dist init r=1, world=2
2022-05-18T05:45:50.0470797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:50.0798027Z dist init r=0, world=2
2022-05-18T05:45:50.0808472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:50.0809950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:50.0877706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:51.4604694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:51.4605254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:51.4928238Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:45:51.4929010Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:45:51.4930715Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:45:51.4931359Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:45:52.2344284Z ok (3.136s)
2022-05-18T05:45:52.2359428Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_None (__main__.TestGradAcc)
2022-05-18T05:45:52.2524652Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107833
2022-05-18T05:45:52.2635335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107834
2022-05-18T05:45:53.1806485Z dist init r=0, world=2
2022-05-18T05:45:53.1817487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:53.2044784Z dist init r=1, world=2
2022-05-18T05:45:53.2056176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:53.2057496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:53.2123552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:54.6015047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:54.6015602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:54.6330722Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:45:54.6331371Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:45:54.6332227Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:45:54.6332876Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:45:55.3707003Z ok (3.136s)
2022-05-18T05:45:55.3722767Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc)
2022-05-18T05:45:55.3886636Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107920
2022-05-18T05:45:55.3997771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107921
2022-05-18T05:45:56.3095223Z dist init r=0, world=2
2022-05-18T05:45:56.3104252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:56.3146670Z dist init r=1, world=2
2022-05-18T05:45:56.3158485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:56.3159351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:56.3207646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:57.6877841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:45:57.6878359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:45:58.0059655Z ok (2.635s)
2022-05-18T05:45:58.0075191Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc)
2022-05-18T05:45:58.0239131Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108003
2022-05-18T05:45:58.0353048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108004
2022-05-18T05:45:58.9822556Z dist init r=0, world=2
2022-05-18T05:45:58.9832654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:45:58.9918717Z dist init r=1, world=2
2022-05-18T05:45:58.9929889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:45:58.9931313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:45:58.9935293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:00.3697764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:00.3698709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:00.6413177Z ok (2.635s)
2022-05-18T05:46:00.6429301Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_None (__main__.TestGradAcc)
2022-05-18T05:46:00.6592096Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108086
2022-05-18T05:46:00.6707194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108087
2022-05-18T05:46:01.5898246Z dist init r=0, world=2
2022-05-18T05:46:01.5908219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:01.6486761Z dist init r=1, world=2
2022-05-18T05:46:01.6498856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:01.6500191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:01.6517433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:03.0238015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:03.0238874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:03.2768074Z ok (2.635s)
2022-05-18T05:46:03.2784512Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc)
2022-05-18T05:46:03.2949979Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108169
2022-05-18T05:46:03.3062225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108170
2022-05-18T05:46:04.2232555Z dist init r=0, world=2
2022-05-18T05:46:04.2241838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:04.2258940Z dist init r=1, world=2
2022-05-18T05:46:04.2270351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:04.2271521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:04.2344700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:05.5980988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:05.5981522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:05.6287542Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:05.6288226Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:05.6323628Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:05.6324303Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:06.4132947Z ok (3.136s)
2022-05-18T05:46:06.4148900Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc)
2022-05-18T05:46:06.4314319Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108256
2022-05-18T05:46:06.4427531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108257
2022-05-18T05:46:07.3593167Z dist init r=1, world=2
2022-05-18T05:46:07.3603555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:07.3720021Z dist init r=0, world=2
2022-05-18T05:46:07.3731260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:07.3732886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:07.3808031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:08.7693880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:08.7694415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:08.8006902Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:08.8007917Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:08.8009481Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:08.8011004Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:09.5496983Z ok (3.136s)
2022-05-18T05:46:09.5513144Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_None (__main__.TestGradAcc)
2022-05-18T05:46:09.5677113Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108343
2022-05-18T05:46:09.5788717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108344
2022-05-18T05:46:10.4923595Z dist init r=0, world=2
2022-05-18T05:46:10.4923917Z dist init r=1, world=2
2022-05-18T05:46:10.4933542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:10.4934652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:10.4935588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:10.4936808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:11.8738792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:11.8739333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:11.9093097Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:11.9093801Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:11.9094662Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:11.9095486Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:12.6858582Z ok (3.136s)
2022-05-18T05:46:12.6874315Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc)
2022-05-18T05:46:12.7041723Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108430
2022-05-18T05:46:12.7156380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108431
2022-05-18T05:46:13.6305919Z dist init r=1, world=2
2022-05-18T05:46:13.6316499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:13.6651943Z dist init r=0, world=2
2022-05-18T05:46:13.6663045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:13.6663855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:13.6722985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:15.0490261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:15.0491053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:15.3217013Z ok (2.636s)
2022-05-18T05:46:15.3232620Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc)
2022-05-18T05:46:15.3394957Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108513
2022-05-18T05:46:15.3506531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108514
2022-05-18T05:46:16.2621074Z dist init r=0, world=2
2022-05-18T05:46:16.2630861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:16.2746640Z dist init r=1, world=2
2022-05-18T05:46:16.2758156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:16.2759161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:16.2834933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:17.6397759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:17.6398296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:17.9567691Z ok (2.635s)
2022-05-18T05:46:17.9584380Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_None (__main__.TestGradAcc)
2022-05-18T05:46:17.9747529Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108596
2022-05-18T05:46:17.9865703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108597
2022-05-18T05:46:18.9075060Z dist init r=1, world=2
2022-05-18T05:46:18.9085459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:18.9288362Z dist init r=0, world=2
2022-05-18T05:46:18.9299206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:18.9300389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:18.9390935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:20.3175653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:20.3176501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:20.5926247Z ok (2.636s)
2022-05-18T05:46:20.5926467Z 
2022-05-18T05:46:20.5926843Z ----------------------------------------------------------------------
2022-05-18T05:46:20.5927207Z Ran 12 tests in 36.303s
2022-05-18T05:46:20.5929752Z 
2022-05-18T05:46:20.5930308Z OK
2022-05-18T05:46:20.5930483Z 
2022-05-18T05:46:20.5932102Z Generating XML reports...
2022-05-18T05:46:20.5990102Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20220518054544.xml
2022-05-18T05:46:20.8770898Z Running distributed/fsdp/test_fsdp_freezing_weights ... [2022-05-18 05:46:20.876562]
2022-05-18T05:46:20.8771681Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_freezing_weights.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:46:20.876675]
2022-05-18T05:46:21.8102077Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights
2022-05-18T05:46:21.8126436Z 
2022-05-18T05:46:21.8126762Z Running tests...
2022-05-18T05:46:21.8127201Z ----------------------------------------------------------------------
2022-05-18T05:46:23.4488994Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:46:23.4904852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108716
2022-05-18T05:46:23.5026725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108717
2022-05-18T05:46:24.4164649Z dist init r=1, world=2
2022-05-18T05:46:24.4167741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:24.4184069Z dist init r=0, world=2
2022-05-18T05:46:24.4189045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:24.4190105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:24.4270954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:25.8058939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:25.8059493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:26.9920555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:26.9921126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:27.5116542Z ok (5.699s)
2022-05-18T05:46:27.5296599Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108803
2022-05-18T05:46:27.5414007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108804
2022-05-18T05:46:28.4493868Z dist init r=1, world=2
2022-05-18T05:46:28.4497237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:28.4856886Z dist init r=0, world=2
2022-05-18T05:46:28.4861629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:28.4862436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:28.4904095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:29.8792085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:29.8793539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:31.0418028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:31.0419039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:31.0529060Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:31.0530793Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:31.0532471Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:31.0533696Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:31.5500858Z ok (4.038s)
2022-05-18T05:46:31.5682571Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108890
2022-05-18T05:46:31.5805388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108891
2022-05-18T05:46:32.5015848Z dist init r=1, world=2
2022-05-18T05:46:32.5019008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:32.5392798Z dist init r=0, world=2
2022-05-18T05:46:32.5397620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:32.5398419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:32.5425357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:33.9146394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:33.9147332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:35.0269026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:35.0271271Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:35.4889021Z ok (3.939s)
2022-05-18T05:46:35.5066806Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108977
2022-05-18T05:46:35.5180252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108978
2022-05-18T05:46:36.4421534Z dist init r=0, world=2
2022-05-18T05:46:36.4421829Z dist init r=1, world=2
2022-05-18T05:46:36.4425103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:36.4425638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:36.4426440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:36.4427129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:37.8323245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:37.8323788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:38.9433510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:38.9436810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:38.9515544Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:38.9516232Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:38.9517086Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:38.9517723Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:39.4264800Z ok (3.937s)
2022-05-18T05:46:39.4443016Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109064
2022-05-18T05:46:39.4561739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109065
2022-05-18T05:46:40.3683577Z dist init r=0, world=2
2022-05-18T05:46:40.3687120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:40.3937945Z dist init r=1, world=2
2022-05-18T05:46:40.3942521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:40.3943314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:40.3992646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:41.7652078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:41.7652650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:42.9557437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:42.9557985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:43.5650086Z ok (4.138s)
2022-05-18T05:46:43.5827442Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109151
2022-05-18T05:46:43.5940816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109152
2022-05-18T05:46:44.5046231Z dist init r=0, world=2
2022-05-18T05:46:44.5049379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:44.5098237Z dist init r=1, world=2
2022-05-18T05:46:44.5102890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:44.5103976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:44.5152388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:45.9017604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:45.9018147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:47.0995747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:47.1283248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:47.1284578Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:47.1285284Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:47.1286136Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:47.1286784Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:47.7028843Z ok (4.138s)
2022-05-18T05:46:47.7204983Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109238
2022-05-18T05:46:47.7317136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109239
2022-05-18T05:46:48.6432564Z dist init r=1, world=2
2022-05-18T05:46:48.6435899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:48.6861349Z dist init r=0, world=2
2022-05-18T05:46:48.6865995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:48.6866790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:48.6944041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:50.0550930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:50.0551484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:51.1507556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:51.1508154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:51.6403258Z ok (3.937s)
2022-05-18T05:46:51.6580700Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109325
2022-05-18T05:46:51.6692512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109326
2022-05-18T05:46:52.5931284Z dist init r=0, world=2
2022-05-18T05:46:52.5934804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:52.6128670Z dist init r=1, world=2
2022-05-18T05:46:52.6133842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:52.6135004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:52.6139005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:53.9912740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:46:53.9913421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:46:55.1016662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:55.1017886Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:46:55.1128154Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:55.1129169Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:55.1130758Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:46:55.1131417Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:46:55.5779689Z ok (3.937s)
2022-05-18T05:46:55.5779930Z 
2022-05-18T05:46:55.5780490Z ----------------------------------------------------------------------
2022-05-18T05:46:55.5780841Z Ran 8 tests in 33.765s
2022-05-18T05:46:55.5781007Z 
2022-05-18T05:46:55.5781101Z OK
2022-05-18T05:46:55.5782800Z 
2022-05-18T05:46:55.5783107Z Generating XML reports...
2022-05-18T05:46:55.5834848Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20220518054621.xml
2022-05-18T05:46:55.8574821Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2022-05-18 05:46:55.857006]
2022-05-18T05:46:55.8575585Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:46:55.857130]
2022-05-18T05:46:56.7906854Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler
2022-05-18T05:46:56.7925488Z 
2022-05-18T05:46:56.7925845Z Running tests...
2022-05-18T05:46:56.7926358Z ----------------------------------------------------------------------
2022-05-18T05:46:58.4603260Z   test_grad_scaling (__main__.TestShardGradScaler) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:46:58.4744105Z ok (1.682s)
2022-05-18T05:46:58.4767714Z   test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... ok (0.002s)
2022-05-18T05:46:58.4841405Z   test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... ok (0.007s)
2022-05-18T05:46:58.5158550Z   test_scaler_enabled_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109464
2022-05-18T05:46:58.5283748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109465
2022-05-18T05:46:59.4395440Z dist init r=1, world=2
2022-05-18T05:46:59.4398736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:46:59.4426508Z dist init r=0, world=2
2022-05-18T05:46:59.4431168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:46:59.4432068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:46:59.4501545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:00.8407821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:00.8408376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:01.1499380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:01.1499958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:01.1541590Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:01.1542292Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:01.1543134Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:01.1544072Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:01.2128227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:01.2128955Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:01.2130874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:01.2131549Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:01.2193033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:01.2193574Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:01.6355920Z ok (3.151s)
2022-05-18T05:47:01.6539122Z   test_scaler_enabled_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109551
2022-05-18T05:47:01.6659105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109552
2022-05-18T05:47:02.5819617Z dist init r=0, world=2
2022-05-18T05:47:02.5822993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:02.5861491Z dist init r=1, world=2
2022-05-18T05:47:02.5866282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:02.5867100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:02.5925845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:03.9668092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:03.9668663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:04.2758271Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:04.2758824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:04.2795830Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:04.2796524Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:04.2797389Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:04.2798044Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:04.3305818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:04.3306494Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:04.3307668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:04.3308620Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:04.3405676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:04.3406195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:04.6729451Z ok (3.037s)
2022-05-18T05:47:04.6908292Z   test_scaler_enabled_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109638
2022-05-18T05:47:04.7027541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109639
2022-05-18T05:47:05.6097316Z dist init r=0, world=2
2022-05-18T05:47:05.6100508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:05.6566809Z dist init r=1, world=2
2022-05-18T05:47:05.6571420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:05.6572793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:05.6608414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:07.0253650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:07.0254192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:07.3375678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:07.3376207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:07.3418864Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:07.3419559Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:07.3420448Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:07.3421099Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:07.4000002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:07.4000680Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:07.4004242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:07.4004930Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:07.4067051Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:07.4067550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:07.8099802Z ok (3.137s)
2022-05-18T05:47:07.8278239Z   test_scaler_enabled_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109725
2022-05-18T05:47:07.8392749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109726
2022-05-18T05:47:08.7618485Z dist init r=0, world=2
2022-05-18T05:47:08.7622230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:08.7638175Z dist init r=1, world=2
2022-05-18T05:47:08.7642700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:08.7644786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:08.7724810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:10.1557603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:10.1558145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:10.4708295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:10.4708850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:10.4747563Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:10.4748552Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:10.4749394Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:10.4750037Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:10.5264507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:10.5265205Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:10.5267318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:10.5267995Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:10.5368268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:10.5368770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:10.8459749Z ok (3.036s)
2022-05-18T05:47:10.8634393Z   test_scaler_enabled_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109812
2022-05-18T05:47:10.8749655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109813
2022-05-18T05:47:11.7900419Z dist init r=1, world=2
2022-05-18T05:47:11.7903679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:11.8158378Z dist init r=0, world=2
2022-05-18T05:47:11.8163265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:11.8164102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:11.8209150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:13.1862852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:13.1863401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:13.5017275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:13.5017841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:13.5059208Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:13.5059904Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:13.5060764Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:13.5061388Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:13.5191861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:13.5192397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:13.5274236Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:47:13.5276748Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:47:13.6015649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:13.6016352Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:13.6019783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:13.6020497Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:13.6082061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:13.6082590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:13.9819125Z ok (3.136s)
2022-05-18T05:47:13.9994517Z   test_scaler_enabled_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109899
2022-05-18T05:47:14.0107970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109900
2022-05-18T05:47:14.9248668Z dist init r=1, world=2
2022-05-18T05:47:14.9252110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:14.9349782Z dist init r=0, world=2
2022-05-18T05:47:14.9354327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:14.9355154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:14.9355844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:16.3169419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:16.3170530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:16.6300253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:16.6300843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:16.6340237Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:16.6340916Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:16.6341767Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:16.6342400Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:16.6470612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:16.6472001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:16.6553150Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:47:16.6558342Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:47:16.7311491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:16.7312177Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:16.7316082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:16.7316769Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:16.7417068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:16.7418092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:17.1177541Z ok (3.136s)
2022-05-18T05:47:17.1352496Z   test_scaler_enabled_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109986
2022-05-18T05:47:17.1464815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109987
2022-05-18T05:47:18.1125798Z dist init r=1, world=2
2022-05-18T05:47:18.1129199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:18.1133401Z dist init r=0, world=2
2022-05-18T05:47:18.1138007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:18.1138827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:18.1232321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:19.5058151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:19.5058698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:19.8218284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:19.8218822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:19.8260203Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:19.8260879Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:19.8261759Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:19.8262412Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:19.8391458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:19.8391988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:19.8472915Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:47:19.8474919Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:47:19.9180299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:19.9181004Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:19.9183573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:19.9184255Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:19.9245452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:19.9245955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:20.3536161Z ok (3.236s)
2022-05-18T05:47:20.3711063Z   test_scaler_enabled_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110073
2022-05-18T05:47:20.3824068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110074
2022-05-18T05:47:21.3565959Z dist init r=1, world=2
2022-05-18T05:47:21.3569089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:21.3596947Z dist init r=0, world=2
2022-05-18T05:47:21.3601721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:21.3602730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:21.3672001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:22.7517886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:22.7518451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:23.0683827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:23.0684366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:23.0722441Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:23.0723145Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:23.0724025Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:23.0724970Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:23.0852717Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:23.0853232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:23.0934764Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:47:23.0936889Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:47:23.1680509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:23.1681371Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:23.1683062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-05-18T05:47:23.1683742Z   warnings.warn(msg, FutureWarning)
2022-05-18T05:47:23.1785173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:23.1785679Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:47:23.5894059Z ok (3.236s)
2022-05-18T05:47:23.5894287Z 
2022-05-18T05:47:23.5894672Z ----------------------------------------------------------------------
2022-05-18T05:47:23.5895014Z Ran 11 tests in 26.797s
2022-05-18T05:47:23.5895165Z 
2022-05-18T05:47:23.5895257Z OK
2022-05-18T05:47:23.5895391Z 
2022-05-18T05:47:23.5895526Z Generating XML reports...
2022-05-18T05:47:23.5944908Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20220518054656.xml
2022-05-18T05:47:23.5955877Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20220518054656.xml
2022-05-18T05:47:23.8708475Z Running distributed/fsdp/test_fsdp_exec_order ... [2022-05-18 05:47:23.870273]
2022-05-18T05:47:23.8709240Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:47:23.870386]
2022-05-18T05:47:24.7696077Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order
2022-05-18T05:47:24.7720316Z 
2022-05-18T05:47:24.7720603Z Running tests...
2022-05-18T05:47:24.7721051Z ----------------------------------------------------------------------
2022-05-18T05:47:24.7732372Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-05-18T05:47:26.4283561Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:47:26.4704954Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110197
2022-05-18T05:47:26.4830697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110198
2022-05-18T05:47:27.4501046Z dist init r=0, world=2
2022-05-18T05:47:27.4504428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:27.4658174Z dist init r=1, world=2
2022-05-18T05:47:27.4663060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:27.4663828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:27.4708770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:28.8709523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:28.8710050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:28.8919101Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:28.8919943Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:28.8920807Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:28.8921432Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:29.4903992Z ok (4.718s)
2022-05-18T05:47:29.4914288Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-05-18T05:47:29.5079536Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110280
2022-05-18T05:47:29.5191293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110281
2022-05-18T05:47:30.4393203Z dist init r=1, world=2
2022-05-18T05:47:30.4396792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:30.4412613Z dist init r=0, world=2
2022-05-18T05:47:30.4417620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:30.4418823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:30.4500060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:31.8441283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:31.8442153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:31.8677806Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:31.8678592Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:31.8679456Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:31.8680080Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:32.5258659Z ok (3.035s)
2022-05-18T05:47:32.5278458Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-05-18T05:47:32.5447666Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110363
2022-05-18T05:47:32.5570058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110364
2022-05-18T05:47:33.5209681Z dist init r=0, world=2
2022-05-18T05:47:33.5213423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:33.5312707Z dist init r=1, world=2
2022-05-18T05:47:33.5317464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:33.5318292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:33.5417568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:34.9377555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:34.9378090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:34.9596757Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:34.9597469Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:34.9598308Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:34.9598954Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:35.2561628Z [['layer0.weight', 'layer0.bias'], ['layer1.weight']] <class 'list'>
2022-05-18T05:47:35.2562121Z [['layer2.0.weight', 'layer2.2.weight']] <class 'list'>
2022-05-18T05:47:35.5637247Z ok (3.038s)
2022-05-18T05:47:35.5657584Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-05-18T05:47:35.5822155Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110450
2022-05-18T05:47:35.5933857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110451
2022-05-18T05:47:36.5213084Z dist init r=1, world=2
2022-05-18T05:47:36.5216442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:36.5235548Z dist init r=0, world=2
2022-05-18T05:47:36.5240187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:36.5241214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:36.5319517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:37.9034860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:37.9035401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:37.9278799Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:37.9279499Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:37.9280366Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:37.9281005Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:38.2347287Z [['layer0.weight', 'layer0.bias'], ['layer1.weight']] <class 'list'>
2022-05-18T05:47:38.2347765Z [['layer2.0.weight', 'layer2.2.weight']] <class 'list'>
2022-05-18T05:47:38.5000455Z ok (2.936s)
2022-05-18T05:47:38.5021250Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-05-18T05:47:38.5186383Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110537
2022-05-18T05:47:38.5299517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110538
2022-05-18T05:47:39.4269402Z dist init r=0, world=2
2022-05-18T05:47:39.4273024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:39.4670413Z dist init r=1, world=2
2022-05-18T05:47:39.4674723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:39.4675521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:39.4679177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:40.8342553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:40.8343096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:40.8557060Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:40.8557758Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:40.8592153Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:40.8592826Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:41.1549341Z [['layer0.weight', 'layer0.bias'], ['layer1.weight']] <class 'list'>
2022-05-18T05:47:41.1549830Z [['layer2.0.weight', 'layer2.2.weight']] <class 'list'>
2022-05-18T05:47:41.4365734Z ok (2.936s)
2022-05-18T05:47:41.4386876Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-05-18T05:47:41.4551114Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110624
2022-05-18T05:47:41.4666442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110625
2022-05-18T05:47:42.3805424Z dist init r=0, world=2
2022-05-18T05:47:42.3808871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:42.4207874Z dist init r=1, world=2
2022-05-18T05:47:42.4213155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:42.4214245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:42.4215293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:43.8089244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:43.8090079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:43.8316557Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:43.8317530Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:43.8318395Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:43.8319036Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:44.1373635Z [['layer0.weight', 'layer0.bias'], ['layer1.weight']] <class 'list'>
2022-05-18T05:47:44.1374145Z [['layer2.0.weight', 'layer2.2.weight']] <class 'list'>
2022-05-18T05:47:44.4733041Z ok (3.037s)
2022-05-18T05:47:44.4916732Z   test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110711
2022-05-18T05:47:44.5028703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110712
2022-05-18T05:47:45.4164106Z dist init r=0, world=2
2022-05-18T05:47:45.4167637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:45.4537947Z dist init r=1, world=2
2022-05-18T05:47:45.4543483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:45.4544869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:45.4574472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:46.8331478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:46.8332440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:46.8557385Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:46.8558054Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:46.8593131Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:46.8594454Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:47.5096028Z ok (3.036s)
2022-05-18T05:47:47.5276959Z   test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110798
2022-05-18T05:47:47.5389131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110799
2022-05-18T05:47:48.4557774Z dist init r=1, world=2
2022-05-18T05:47:48.4560662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:47:48.4933041Z dist init r=0, world=2
2022-05-18T05:47:48.4937936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:48.4939095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:48.4966584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:47:49.8856566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:47:49.8857099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:49.9077879Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:49.9078883Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:49.9079740Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:49.9080382Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:47:50.5457996Z ok (3.036s)
2022-05-18T05:47:50.5458220Z 
2022-05-18T05:47:50.5458634Z ----------------------------------------------------------------------
2022-05-18T05:47:50.5458977Z Ran 8 tests in 25.774s
2022-05-18T05:47:50.5459125Z 
2022-05-18T05:47:50.5459221Z OK
2022-05-18T05:47:50.5459366Z 
2022-05-18T05:47:50.5464211Z Generating XML reports...
2022-05-18T05:47:50.5513189Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20220518054724.xml
2022-05-18T05:47:50.8277565Z Running distributed/fsdp/test_fsdp_overlap ... [2022-05-18 05:47:50.827227]
2022-05-18T05:47:50.8278305Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:47:50.827339]
2022-05-18T05:47:51.7586880Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap
2022-05-18T05:47:51.7612073Z 
2022-05-18T05:47:51.7612395Z Running tests...
2022-05-18T05:47:51.7612845Z ----------------------------------------------------------------------
2022-05-18T05:47:53.4444211Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:47:53.4862706Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110922
2022-05-18T05:47:54.4089301Z dist init r=0, world=1
2022-05-18T05:47:54.4092942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:47:54.4093813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:47:55.7349738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:47:55.8239424Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:47:55.8240410Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:48:03.8562464Z 
2022-05-18T05:48:03.8562982Z rank0:
2022-05-18T05:48:03.8563721Z   e1: {'cpu_iter': 0.002217621700000283, 'cpu_wait': 3.704290000010602e-05, 'gpu_compute': 0.07444160021841525, 'gpu_total': 1.357855987548828}
2022-05-18T05:48:03.8564326Z   e2: {'cpu_iter': 0.00407261580000009, 'cpu_wait': 3.543060000019693e-05, 'gpu_compute': 0.26491840146481993, 'gpu_total': 2.1884608030319215}
2022-05-18T05:48:03.8564913Z   e3: {'cpu_iter': 0.0023079194999999332, 'cpu_wait': 0.1873428689999999, 'gpu_compute': 189.06942253112794, 'gpu_total': 189.56471405029296}
2022-05-18T05:48:03.8565499Z   e4: {'cpu_iter': 0.0041753047999998575, 'cpu_wait': 0.18553358079999976, 'gpu_compute': 189.07795829772948, 'gpu_total': 189.70594177246093}
2022-05-18T05:48:04.1037039Z ok (12.342s)
2022-05-18T05:48:04.1048665Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-05-18T05:48:04.1050054Z 
2022-05-18T05:48:04.1050358Z ----------------------------------------------------------------------
2022-05-18T05:48:04.1050677Z Ran 2 tests in 12.344s
2022-05-18T05:48:04.1050845Z 
2022-05-18T05:48:04.1050954Z OK (skipped=1)
2022-05-18T05:48:04.1051110Z 
2022-05-18T05:48:04.1051236Z Generating XML reports...
2022-05-18T05:48:04.1099436Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220518054751.xml
2022-05-18T05:48:04.1105012Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220518054751.xml
2022-05-18T05:48:04.3886146Z Running distributed/elastic/multiprocessing/api_test ... [2022-05-18 05:48:04.388138]
2022-05-18T05:48:04.3886910Z Executing ['/opt/conda/bin/python', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:48:04.388240]
2022-05-18T05:48:05.3856196Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test
2022-05-18T05:48:05.3879399Z 
2022-05-18T05:48:05.3879860Z Running tests...
2022-05-18T05:48:05.3880334Z ----------------------------------------------------------------------
2022-05-18T05:48:07.0726381Z   test_get_failures (__main__.RunProcResultsTest) ... ok (1.684s)
2022-05-18T05:48:07.0737798Z   test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s)
2022-05-18T05:48:07.0758212Z   test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.002s)
2022-05-18T05:48:07.1336189Z   test_binary (__main__.StartProcessesListTest) ... hello stdout from 0
2022-05-18T05:48:07.1336561Z hello stderr from 0
2022-05-18T05:48:07.1556872Z hello stdout from 1
2022-05-18T05:48:07.1557167Z hello stderr from 1
2022-05-18T05:48:07.2312996Z ok (0.155s)
2022-05-18T05:48:07.3088214Z   test_binary_exit (__main__.StartProcessesListTest) ... bar stdout from 1
2022-05-18T05:48:07.3088624Z bar stderr from 1
2022-05-18T05:48:07.3826129Z failed (exitcode: 138) local_rank: 0 (pid: 111004) of binary: distributed/elastic/multiprocessing/bin/echo1.py
2022-05-18T05:48:07.3839162Z ok (0.152s)
2022-05-18T05:48:07.4102128Z   test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.026s)
2022-05-18T05:48:07.4618196Z   test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last):
2022-05-18T05:48:07.4618715Z   File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-05-18T05:48:07.4619078Z     raise RuntimeError(f"raised from {rank}")
2022-05-18T05:48:07.4619387Z RuntimeError: raised from 0
2022-05-18T05:48:07.4839508Z bar from 1
2022-05-18T05:48:07.5586044Z failed (exitcode: 1) local_rank: 0 (pid: 111007) of binary: distributed/elastic/multiprocessing/bin/echo2.py
2022-05-18T05:48:07.5596339Z ok (0.149s)
2022-05-18T05:48:07.6352167Z   test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1
2022-05-18T05:48:07.7101502Z [trainer1]:world stderr from 1
2022-05-18T05:48:07.7101820Z [trainer0]:hello stdout from 0
2022-05-18T05:48:08.7127966Z ok (1.153s)
2022-05-18T05:48:09.7263116Z   test_function (__main__.StartProcessesListTest) ... hello stdout from 1
2022-05-18T05:48:09.7263512Z hello stderr from 1
2022-05-18T05:48:09.7295917Z hello stdout from 0
2022-05-18T05:48:09.7296162Z hello stderr from 0
2022-05-18T05:48:09.8846420Z Closing process 111015 via signal SIGTERM
2022-05-18T05:48:09.8901059Z ok (1.177s)
2022-05-18T05:48:11.4916583Z   test_function_large_ret_val (__main__.StartProcessesListTest) ... Closing process 111084 via signal SIGTERM
2022-05-18T05:48:11.4917057Z Closing process 111085 via signal SIGTERM
2022-05-18T05:48:11.4917372Z Closing process 111086 via signal SIGTERM
2022-05-18T05:48:11.4951183Z ok (1.605s)
2022-05-18T05:48:11.4978290Z   test_function_raise (__main__.StartProcessesListTest)
2022-05-18T05:48:12.6407463Z run 2x copies of echo2, raise an exception on the first ... failed (exitcode: 1) local_rank: 0 (pid: 111224) of fn: echo2 (start_method: spawn)
2022-05-18T05:48:12.6408252Z Traceback (most recent call last):
2022-05-18T05:48:12.6408934Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-05-18T05:48:12.6409371Z     self._pc.join(-1)
2022-05-18T05:48:12.6410504Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-05-18T05:48:12.6410985Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-05-18T05:48:12.6411447Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-05-18T05:48:12.6411723Z 
2022-05-18T05:48:12.6411953Z -- Process 0 terminated with the following error:
2022-05-18T05:48:12.6412271Z Traceback (most recent call last):
2022-05-18T05:48:12.6412765Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-05-18T05:48:12.6413120Z     fn(i, *args)
2022-05-18T05:48:12.6413607Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap
2022-05-18T05:48:12.6414003Z     ret = record(fn)(*args_)
2022-05-18T05:48:12.6414540Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
2022-05-18T05:48:12.6414960Z     return f(*args, **kwargs)
2022-05-18T05:48:12.6415339Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-05-18T05:48:12.6415723Z     raise RuntimeError(msg)
2022-05-18T05:48:12.6415988Z RuntimeError: hello
2022-05-18T05:48:12.6416154Z 
2022-05-18T05:48:12.6422819Z ok (1.147s)
2022-05-18T05:48:12.6450909Z   test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.003s)
2022-05-18T05:48:12.6468202Z   test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.002s)
2022-05-18T05:48:12.7087044Z   test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 111294 via signal SIGTERM
2022-05-18T05:48:12.7195893Z ok (0.073s)
2022-05-18T05:48:12.7243670Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-05-18T05:48:12.7244164Z Traceback (most recent call last):
2022-05-18T05:48:12.7244708Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-05-18T05:48:12.7245123Z     self._pc.join(-1)
2022-05-18T05:48:12.7245458Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__
2022-05-18T05:48:12.7245815Z     return _mock_self._mock_call(*args, **kwargs)
2022-05-18T05:48:12.7246466Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call
2022-05-18T05:48:12.7246808Z     raise effect
2022-05-18T05:48:12.7247181Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-05-18T05:48:12.7251125Z ok (0.005s)
2022-05-18T05:48:14.8636189Z   test_pcontext_wait (__main__.StartProcessesListTest) ... ok (2.138s)
2022-05-18T05:48:14.8927585Z   test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 111330 closing signal SIGTERM
2022-05-18T05:48:14.8945000Z ok (0.031s)
2022-05-18T05:48:14.8969680Z   test_to_map (__main__.StartProcessesListTest) ... ok (0.002s)
2022-05-18T05:48:14.8978597Z   test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s)
2022-05-18T05:48:15.8869045Z   test_void_function (__main__.StartProcessesListTest) ... world
2022-05-18T05:48:15.9375545Z hello
2022-05-18T05:48:16.1507157Z Closing process 111332 via signal SIGTERM
2022-05-18T05:48:16.1518409Z ok (1.254s)
2022-05-18T05:48:16.1546569Z   test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s)
2022-05-18T05:48:16.2344751Z   test_binary_exit (__main__.StartProcessesTest) ... bar stdout from 1
2022-05-18T05:48:16.2345112Z bar stderr from 1
2022-05-18T05:48:16.3079746Z failed (exitcode: 138) local_rank: 0 (pid: 111401) of binary: distributed/elastic/multiprocessing/bin/echo1.py
2022-05-18T05:48:16.3091324Z ok (0.154s)
2022-05-18T05:48:16.3353335Z   test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.026s)
2022-05-18T05:48:16.3862928Z   test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last):
2022-05-18T05:48:16.3863397Z   File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-05-18T05:48:16.3863783Z     raise RuntimeError(f"raised from {rank}")
2022-05-18T05:48:16.3864090Z RuntimeError: raised from 0
2022-05-18T05:48:16.4094063Z bar from 1
2022-05-18T05:48:16.4833804Z failed (exitcode: 1) local_rank: 0 (pid: 111404) of binary: distributed/elastic/multiprocessing/bin/echo2.py
2022-05-18T05:48:16.4843946Z ok (0.149s)
2022-05-18T05:48:18.0840640Z   test_function_large_ret_val (__main__.StartProcessesTest) ... Closing process 111406 via signal SIGTERM
2022-05-18T05:48:18.0841088Z Closing process 111407 via signal SIGTERM
2022-05-18T05:48:18.0841417Z Closing process 111409 via signal SIGTERM
2022-05-18T05:48:18.0912224Z ok (1.607s)
2022-05-18T05:48:18.0935437Z   test_function_raise (__main__.StartProcessesTest)
2022-05-18T05:48:19.2366413Z run 2x copies of echo2, raise an exception on the first ... failed (exitcode: 1) local_rank: 0 (pid: 111546) of fn: echo2 (start_method: spawn)
2022-05-18T05:48:19.2366899Z Traceback (most recent call last):
2022-05-18T05:48:19.2367567Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-05-18T05:48:19.2368010Z     self._pc.join(-1)
2022-05-18T05:48:19.2368525Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-05-18T05:48:19.2369024Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-05-18T05:48:19.2369472Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-05-18T05:48:19.2370423Z 
2022-05-18T05:48:19.2370677Z -- Process 0 terminated with the following error:
2022-05-18T05:48:19.2371023Z Traceback (most recent call last):
2022-05-18T05:48:19.2371506Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-05-18T05:48:19.2371866Z     fn(i, *args)
2022-05-18T05:48:19.2372374Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap
2022-05-18T05:48:19.2372753Z     ret = record(fn)(*args_)
2022-05-18T05:48:19.2373298Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
2022-05-18T05:48:19.2373712Z     return f(*args, **kwargs)
2022-05-18T05:48:19.2374109Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-05-18T05:48:19.2374744Z     raise RuntimeError(msg)
2022-05-18T05:48:19.2375042Z RuntimeError: hello
2022-05-18T05:48:19.2375210Z 
2022-05-18T05:48:19.2383412Z ok (1.147s)
2022-05-18T05:48:19.2412616Z   test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.003s)
2022-05-18T05:48:19.2430581Z   test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.002s)
2022-05-18T05:48:19.3053673Z   test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 111616 via signal SIGTERM
2022-05-18T05:48:19.3167616Z ok (0.074s)
2022-05-18T05:48:19.3208403Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-05-18T05:48:19.3209045Z Traceback (most recent call last):
2022-05-18T05:48:19.3210017Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-05-18T05:48:19.3210715Z     self._pc.join(-1)
2022-05-18T05:48:19.3211065Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__
2022-05-18T05:48:19.3211431Z     return _mock_self._mock_call(*args, **kwargs)
2022-05-18T05:48:19.3212015Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call
2022-05-18T05:48:19.3212592Z     raise effect
2022-05-18T05:48:19.3213008Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-05-18T05:48:19.3216335Z ok (0.005s)
2022-05-18T05:48:21.4605179Z   test_pcontext_wait (__main__.StartProcessesTest) ... ok (2.139s)
2022-05-18T05:48:21.4920580Z   test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 111652 closing signal SIGTERM
2022-05-18T05:48:21.4938430Z ok (0.033s)
2022-05-18T05:48:21.4961855Z   test_to_map (__main__.StartProcessesTest) ... ok (0.002s)
2022-05-18T05:48:21.4970977Z   test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s)
2022-05-18T05:48:22.5060836Z   test_void_function (__main__.StartProcessesTest) ... world
2022-05-18T05:48:22.5497751Z hello
2022-05-18T05:48:22.7491833Z Closing process 111654 via signal SIGTERM
2022-05-18T05:48:22.7537677Z ok (1.257s)
2022-05-18T05:48:22.7558197Z   test_from_str_bad_input (__main__.StdTest) ... ok (0.002s)
2022-05-18T05:48:22.7571593Z   test_from_value (__main__.StdTest) ... ok (0.001s)
2022-05-18T05:48:22.7582076Z   test_from_value_map (__main__.StdTest) ... ok (0.001s)
2022-05-18T05:48:22.7582830Z 
2022-05-18T05:48:22.7583325Z ----------------------------------------------------------------------
2022-05-18T05:48:22.7583915Z Ran 38 tests in 17.370s
2022-05-18T05:48:22.7584101Z 
2022-05-18T05:48:22.7584196Z OK
2022-05-18T05:48:22.7584335Z 
2022-05-18T05:48:22.7584469Z Generating XML reports...
2022-05-18T05:48:22.7635620Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220518054805.xml
2022-05-18T05:48:22.7663740Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220518054805.xml
2022-05-18T05:48:22.7684611Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220518054805.xml
2022-05-18T05:48:22.7690842Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220518054805.xml
2022-05-18T05:48:23.0524334Z Running distributed/_shard/sharded_tensor/ops/test_matrix_ops ... [2022-05-18 05:48:23.051917]
2022-05-18T05:48:23.0525136Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:48:23.052022]
2022-05-18T05:48:23.9504336Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops
2022-05-18T05:48:23.9530821Z 
2022-05-18T05:48:23.9531170Z Running tests...
2022-05-18T05:48:23.9531646Z ----------------------------------------------------------------------
2022-05-18T05:48:25.6037782Z   test_sharded_tensor_contiguous (__main__.TestShardedTensorMatrixOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:48:25.6440937Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111760
2022-05-18T05:48:25.6566029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111761
2022-05-18T05:48:25.6694203Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 111762
2022-05-18T05:48:25.6826029Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 111763
2022-05-18T05:48:26.5981755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:26.6125733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:48:26.6215932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:26.6437377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:48:26.7867997Z skip: Need at least 4 CUDA devices (2.833s)
2022-05-18T05:48:26.8055792Z   test_sharded_tensor_layer_norm (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111904
2022-05-18T05:48:26.8168530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111905
2022-05-18T05:48:26.8292552Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 111906
2022-05-18T05:48:26.8421874Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 111907
2022-05-18T05:48:27.7885482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:27.8460464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:48:27.8515778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:27.9029017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:48:28.0461395Z skip: Need at least 4 CUDA devices (1.259s)
2022-05-18T05:48:28.0649418Z   test_sharded_tensor_layer_norm_error (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112048
2022-05-18T05:48:28.0764128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112049
2022-05-18T05:48:28.0891628Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 112050
2022-05-18T05:48:28.1022943Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 112051
2022-05-18T05:48:28.9900434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:29.0153325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:48:29.0274194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:29.0311154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:48:29.2060590Z skip: Need at least 4 CUDA devices (1.160s)
2022-05-18T05:48:29.2236472Z   test_sharded_tensor_masked_fill (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112192
2022-05-18T05:48:29.2349947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112193
2022-05-18T05:48:29.2475733Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 112194
2022-05-18T05:48:29.2608927Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 112195
2022-05-18T05:48:30.2160353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:30.2325514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:48:30.2538455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:30.2667417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:48:30.4648433Z skip: Need at least 4 CUDA devices (1.259s)
2022-05-18T05:48:30.4831075Z   test_sharded_tensor_masked_fill_error (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112336
2022-05-18T05:48:30.4945176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112337
2022-05-18T05:48:30.5071580Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 112338
2022-05-18T05:48:30.5203314Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 112339
2022-05-18T05:48:31.4860388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:31.5074469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:31.5281856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:48:31.5347787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:48:31.7244383Z skip: Need at least 4 CUDA devices (1.259s)
2022-05-18T05:48:31.7424296Z   test_sharded_tensor_softmax (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112480
2022-05-18T05:48:31.7538765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112481
2022-05-18T05:48:31.7664780Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 112482
2022-05-18T05:48:31.7794796Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 112483
2022-05-18T05:48:32.6829670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:32.7108883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:48:32.7427755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:32.7566049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:48:32.9831888Z skip: Need at least 4 CUDA devices (1.259s)
2022-05-18T05:48:33.0016215Z   test_sharded_tensor_transpose (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112624
2022-05-18T05:48:33.0129155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112625
2022-05-18T05:48:33.0255893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 112626
2022-05-18T05:48:33.0390413Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 112627
2022-05-18T05:48:33.9199412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:33.9497918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:48:33.9708553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:48:33.9900235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:34.1426443Z skip: Need at least 4 CUDA devices (1.159s)
2022-05-18T05:48:34.1605434Z   test_sharded_tensor_transpose_error (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112768
2022-05-18T05:48:34.1718910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112769
2022-05-18T05:48:34.1845724Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 112770
2022-05-18T05:48:34.1977808Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 112771
2022-05-18T05:48:35.1405670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:35.1406997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:35.1416809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:48:35.1913874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:48:35.4015745Z skip: Need at least 4 CUDA devices (1.259s)
2022-05-18T05:48:35.4198910Z   test_sharded_tensor_type_as (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112912
2022-05-18T05:48:35.4312882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112913
2022-05-18T05:48:35.4440689Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 112914
2022-05-18T05:48:35.4572928Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 112915
2022-05-18T05:48:36.4495594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:36.4584131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:36.4779561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:48:36.4799906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:48:36.6615305Z skip: Need at least 4 CUDA devices (1.260s)
2022-05-18T05:48:36.6798159Z   test_sharded_tensor_view (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113056
2022-05-18T05:48:36.6912353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113057
2022-05-18T05:48:36.7039170Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 113058
2022-05-18T05:48:36.7169899Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 113059
2022-05-18T05:48:37.6101743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:48:37.6429323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:37.6621457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:37.6640469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:48:37.8205753Z skip: Need at least 4 CUDA devices (1.159s)
2022-05-18T05:48:37.8388914Z   test_sharded_tensor_view_error (__main__.TestShardedTensorMatrixOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113200
2022-05-18T05:48:37.8516898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113201
2022-05-18T05:48:37.8645753Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 113202
2022-05-18T05:48:37.8780853Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 113203
2022-05-18T05:48:38.8114657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:38.8526413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:48:38.8958751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:48:38.9000454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:39.0820528Z skip: Need at least 4 CUDA devices (1.261s)
2022-05-18T05:48:39.0820796Z 
2022-05-18T05:48:39.0821194Z ----------------------------------------------------------------------
2022-05-18T05:48:39.0821531Z Ran 11 tests in 15.129s
2022-05-18T05:48:39.0821699Z 
2022-05-18T05:48:39.0821792Z OK (skipped=11)
2022-05-18T05:48:39.0821962Z 
2022-05-18T05:48:39.0822091Z Generating XML reports...
2022-05-18T05:48:39.0895839Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20220518054823.xml
2022-05-18T05:48:39.3725130Z Running distributed/fsdp/test_fsdp_memory ... [2022-05-18 05:48:39.371984]
2022-05-18T05:48:39.3726446Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:48:39.372098]
2022-05-18T05:48:40.2829995Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory
2022-05-18T05:48:40.2854607Z 
2022-05-18T05:48:40.2855113Z Running tests...
2022-05-18T05:48:40.2855592Z ----------------------------------------------------------------------
2022-05-18T05:48:41.9536042Z   test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:48:41.9947315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113381
2022-05-18T05:48:42.0071064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113382
2022-05-18T05:48:42.8983553Z dist init r=0, world=2
2022-05-18T05:48:42.8986601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:48:42.9464435Z dist init r=1, world=2
2022-05-18T05:48:42.9469268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:48:42.9470322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:48:42.9494033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:48:44.3391416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:44.3392069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:44.3756482Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:48:44.3757234Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:48:44.3764662Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:48:44.3765322Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:48:48.2218443Z ok (7.936s)
2022-05-18T05:48:48.2406342Z   test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113468
2022-05-18T05:48:48.2521020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113469
2022-05-18T05:48:49.1638600Z dist init r=0, world=2
2022-05-18T05:48:49.1641959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:48:49.1667676Z dist init r=1, world=2
2022-05-18T05:48:49.1672952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:48:49.1674228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:48:49.1744588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:48:50.5615341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:50.5615885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:50.5980291Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:48:50.5980995Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:48:50.5981876Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:48:50.5982519Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:48:54.2639140Z ok (6.042s)
2022-05-18T05:48:54.2639386Z 
2022-05-18T05:48:54.2640008Z ----------------------------------------------------------------------
2022-05-18T05:48:54.2640362Z Ran 2 tests in 13.978s
2022-05-18T05:48:54.2640539Z 
2022-05-18T05:48:54.2640633Z OK
2022-05-18T05:48:54.2640769Z 
2022-05-18T05:48:54.2640887Z Generating XML reports...
2022-05-18T05:48:54.2688452Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20220518054840.xml
2022-05-18T05:48:54.5465444Z Running distributed/fsdp/test_fsdp_ignored_modules ... [2022-05-18 05:48:54.546058]
2022-05-18T05:48:54.5466225Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_ignored_modules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:48:54.546172]
2022-05-18T05:48:55.4620526Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules
2022-05-18T05:48:55.4642872Z 
2022-05-18T05:48:55.4643278Z Running tests...
2022-05-18T05:48:55.4643773Z ----------------------------------------------------------------------
2022-05-18T05:48:55.4653071Z   test_ignored_modules_invalid (__main__.TestFSDPIgnoredModules)
2022-05-18T05:48:57.0799231Z Tests that passing an FSDP module as an ignored module or the ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:48:57.1210355Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113592
2022-05-18T05:48:57.1333586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113593
2022-05-18T05:48:58.0466721Z dist init r=0, world=2
2022-05-18T05:48:58.0469732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:48:58.0533936Z dist init r=1, world=2
2022-05-18T05:48:58.0538540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:48:58.0539550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:48:58.0572657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:48:59.4269952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:48:59.4270504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:48:59.4490733Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:48:59.4491436Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:48:59.4492289Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:48:59.4492932Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:48:59.7398474Z ok (4.275s)
2022-05-18T05:48:59.7415016Z   test_ignored_modules_nested (__main__.TestFSDPIgnoredModules)
2022-05-18T05:48:59.7580034Z Tests that passing a module with nested FSDP modules does not ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113675
2022-05-18T05:48:59.7689666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113676
2022-05-18T05:49:00.6800667Z dist init r=1, world=2
2022-05-18T05:49:00.6804182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:49:00.6851046Z dist init r=0, world=2
2022-05-18T05:49:00.6855925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:49:00.6857016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:49:00.6907053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:49:02.0829752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:02.0830323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:02.6756173Z ok (2.936s)
2022-05-18T05:49:02.6773488Z   test_ignored_modules_transformer (__main__.TestFSDPIgnoredModules)
2022-05-18T05:49:02.6939218Z Tests that ignored modules' parameters are not flattened for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113762
2022-05-18T05:49:02.7049735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113763
2022-05-18T05:49:03.6252070Z dist init r=1, world=2
2022-05-18T05:49:03.6255879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:49:03.6625750Z dist init r=0, world=2
2022-05-18T05:49:03.6629762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:49:03.6630999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:49:03.6662565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:49:05.0362353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:05.0363413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:05.0679131Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:49:05.0680433Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:49:05.0682076Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:49:05.0683308Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:49:06.1125549Z ok (3.437s)
2022-05-18T05:49:06.1125764Z 
2022-05-18T05:49:06.1126162Z ----------------------------------------------------------------------
2022-05-18T05:49:06.1126519Z Ran 3 tests in 10.648s
2022-05-18T05:49:06.1126684Z 
2022-05-18T05:49:06.1128744Z OK
2022-05-18T05:49:06.1129058Z 
2022-05-18T05:49:06.1129229Z Generating XML reports...
2022-05-18T05:49:06.1175279Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20220518054855.xml
2022-05-18T05:49:06.3940300Z Running distributed/elastic/timer/local_timer_example ... [2022-05-18 05:49:06.393508]
2022-05-18T05:49:06.3941099Z Executing ['/opt/conda/bin/python', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:49:06.393616]
2022-05-18T05:49:07.2878828Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example
2022-05-18T05:49:07.2894639Z 
2022-05-18T05:49:07.2894960Z Running tests...
2022-05-18T05:49:07.2895411Z ----------------------------------------------------------------------
2022-05-18T05:49:08.9427439Z   test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2022-05-18 05:49:08,942 driver: init
2022-05-18T05:49:08.9730397Z [INFO] 2022-05-18 05:49:08,972 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-05-18T05:49:08.9731184Z [INFO] 2022-05-18 05:49:08,972 api: Starting watchdog thread...
2022-05-18T05:49:10.0984516Z [INFO] 2022-05-18 05:49:10,097 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:10.1173938Z [INFO] 2022-05-18 05:49:10,116 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:10.1384423Z [INFO] 2022-05-18 05:49:10,137 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:10.1459434Z [INFO] 2022-05-18 05:49:10,145 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:10.1546143Z [INFO] 2022-05-18 05:49:10,154 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:10.1582250Z [INFO] 2022-05-18 05:49:10,157 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:10.1806023Z [INFO] 2022-05-18 05:49:10,180 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:10.1875331Z [INFO] 2022-05-18 05:49:10,187 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:11.1596748Z [INFO] 2022-05-18 05:49:11,158 api: Reaping worker_id=[113887]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T05:49:11.1597786Z [INFO] 2022-05-18 05:49:11,159 api: Successfully reaped worker=[113887]
2022-05-18T05:49:11.2007020Z [INFO] 2022-05-18 05:49:11,200 api: Reaping worker_id=[113889]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T05:49:11.2008874Z [INFO] 2022-05-18 05:49:11,200 api: Successfully reaped worker=[113889]
2022-05-18T05:49:11.2112596Z [INFO] 2022-05-18 05:49:11,210 api: Reaping worker_id=[113891]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T05:49:11.2114111Z [INFO] 2022-05-18 05:49:11,211 api: Successfully reaped worker=[113891]
2022-05-18T05:49:11.2217298Z [INFO] 2022-05-18 05:49:11,221 api: Reaping worker_id=[113895]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T05:49:11.2223503Z [INFO] 2022-05-18 05:49:11,221 api: Successfully reaped worker=[113895]
2022-05-18T05:49:11.2286034Z [INFO] 2022-05-18 05:49:11,228 api: Stopping LocalTimerServer
2022-05-18T05:49:11.2286485Z [INFO] 2022-05-18 05:49:11,228 api: Stopping watchdog thread...
2022-05-18T05:49:11.2330134Z ok (3.943s)
2022-05-18T05:49:11.2353147Z   test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2022-05-18 05:49:11,235 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-05-18T05:49:11.2353703Z [INFO] 2022-05-18 05:49:11,235 api: Starting watchdog thread...
2022-05-18T05:49:12.3501203Z [INFO] 2022-05-18 05:49:12,349 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:12.3853994Z [INFO] 2022-05-18 05:49:12,384 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:12.3941535Z [INFO] 2022-05-18 05:49:12,393 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:12.4314323Z [INFO] 2022-05-18 05:49:12,430 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:12.4333434Z [INFO] 2022-05-18 05:49:12,432 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:12.4562029Z [INFO] 2022-05-18 05:49:12,455 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:12.4606522Z [INFO] 2022-05-18 05:49:12,460 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:12.4729831Z [INFO] 2022-05-18 05:49:12,472 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:14.2967485Z [INFO] 2022-05-18 05:49:14,296 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:14.3432951Z [INFO] 2022-05-18 05:49:14,342 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:14.3634148Z [INFO] 2022-05-18 05:49:14,362 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:14.3915154Z [INFO] 2022-05-18 05:49:14,390 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:14.3978984Z [INFO] 2022-05-18 05:49:14,397 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:14.3994921Z [INFO] 2022-05-18 05:49:14,399 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:14.4004916Z [INFO] 2022-05-18 05:49:14,400 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:14.4148834Z [INFO] 2022-05-18 05:49:14,414 api: Timer client configured to: LocalTimerClient
2022-05-18T05:49:15.3571065Z [INFO] 2022-05-18 05:49:15,356 api: Reaping worker_id=[114465]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T05:49:15.3576616Z [INFO] 2022-05-18 05:49:15,356 api: Successfully reaped worker=[114465]
2022-05-18T05:49:15.3980244Z [INFO] 2022-05-18 05:49:15,397 api: Reaping worker_id=[114469]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T05:49:15.3981208Z [INFO] 2022-05-18 05:49:15,397 api: Successfully reaped worker=[114469]
2022-05-18T05:49:15.4119602Z [INFO] 2022-05-18 05:49:15,411 api: Stopping LocalTimerServer
2022-05-18T05:49:15.4120373Z [INFO] 2022-05-18 05:49:15,411 api: Stopping watchdog thread...
2022-05-18T05:49:15.4184228Z [INFO] 2022-05-18 05:49:15,418 api: Reaping worker_id=[114468]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T05:49:15.4185146Z [INFO] 2022-05-18 05:49:15,418 local_timer: Process with pid=114468 does not exist. Skipping
2022-05-18T05:49:15.4185817Z [INFO] 2022-05-18 05:49:15,418 api: Successfully reaped worker=[114468]
2022-05-18T05:49:15.4190607Z ok (4.186s)
2022-05-18T05:49:15.4193416Z 
2022-05-18T05:49:15.4193996Z ----------------------------------------------------------------------
2022-05-18T05:49:15.4194373Z Ran 2 tests in 8.130s
2022-05-18T05:49:15.4194545Z 
2022-05-18T05:49:15.4194652Z OK
2022-05-18T05:49:15.4194772Z 
2022-05-18T05:49:15.4194913Z Generating XML reports...
2022-05-18T05:49:15.4237588Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220518054907.xml
2022-05-18T05:49:15.6938111Z Running distributed/fsdp/test_fsdp_input ... [2022-05-18 05:49:15.693295]
2022-05-18T05:49:15.6938865Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:49:15.693406]
2022-05-18T05:49:16.6263639Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input
2022-05-18T05:49:16.6287092Z 
2022-05-18T05:49:16.6287532Z Running tests...
2022-05-18T05:49:16.6288027Z ----------------------------------------------------------------------
2022-05-18T05:49:16.6307245Z   test_input_type_dict (__main__.TestInput)
2022-05-18T05:49:18.2949548Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:49:18.3367622Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114789
2022-05-18T05:49:19.2589609Z dist init r=0, world=1
2022-05-18T05:49:19.2593115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:49:19.2593956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:49:20.5626970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:20.5843765Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:49:20.5844464Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:49:21.1431133Z ok (4.514s)
2022-05-18T05:49:21.1449538Z   test_input_type_list (__main__.TestInput)
2022-05-18T05:49:21.1614532Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114833
2022-05-18T05:49:22.0732179Z dist init r=0, world=1
2022-05-18T05:49:22.0735139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:49:22.0736332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:49:23.3637946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:23.3843752Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:49:23.3844467Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:49:23.9671203Z ok (2.824s)
2022-05-18T05:49:23.9671413Z 
2022-05-18T05:49:23.9671803Z ----------------------------------------------------------------------
2022-05-18T05:49:23.9672387Z Ran 2 tests in 7.338s
2022-05-18T05:49:23.9672553Z 
2022-05-18T05:49:23.9675040Z OK
2022-05-18T05:49:23.9675249Z 
2022-05-18T05:49:23.9675637Z Generating XML reports...
2022-05-18T05:49:23.9719655Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20220518054916.xml
2022-05-18T05:49:24.2472124Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2022-05-18 05:49:24.246622]
2022-05-18T05:49:24.2473089Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:49:24.246725]
2022-05-18T05:49:25.1662040Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops
2022-05-18T05:49:25.1677923Z 
2022-05-18T05:49:25.1678494Z Running tests...
2022-05-18T05:49:25.1679389Z ----------------------------------------------------------------------
2022-05-18T05:49:26.8193324Z   test_clone (__main__.TestTensorOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:49:26.8606619Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114914
2022-05-18T05:49:26.8733637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114915
2022-05-18T05:49:26.8866544Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 114916
2022-05-18T05:49:26.9002114Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 114917
2022-05-18T05:49:27.7964798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:27.8638735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:49:27.8718527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:27.8856106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:49:28.1044386Z skip: Need at least 4 CUDA devices (2.936s)
2022-05-18T05:49:28.1226694Z   test_deep_copy (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115058
2022-05-18T05:49:28.1342317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115059
2022-05-18T05:49:28.1470564Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 115060
2022-05-18T05:49:28.1605423Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 115061
2022-05-18T05:49:29.1049811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:29.1385361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:29.1386292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:49:29.1454853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:49:29.3642827Z skip: Need at least 4 CUDA devices (1.260s)
2022-05-18T05:49:29.3822631Z   test_detach (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115202
2022-05-18T05:49:29.3939055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115203
2022-05-18T05:49:29.4066455Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 115204
2022-05-18T05:49:29.4198558Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 115205
2022-05-18T05:49:30.3790946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:30.3832900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:49:30.3833444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:49:30.3910039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:30.6235732Z skip: Need at least 4 CUDA devices (1.259s)
2022-05-18T05:49:30.6412938Z   test_set_requires_grad (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115346
2022-05-18T05:49:30.6529229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115347
2022-05-18T05:49:30.6656645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 115348
2022-05-18T05:49:30.6788103Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 115349
2022-05-18T05:49:31.6435553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:31.6671353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:31.6821024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:49:31.7202424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:49:31.8828734Z skip: Need at least 4 CUDA devices (1.259s)
2022-05-18T05:49:31.8828953Z 
2022-05-18T05:49:31.8829344Z ----------------------------------------------------------------------
2022-05-18T05:49:31.8829693Z Ran 4 tests in 6.715s
2022-05-18T05:49:31.8829862Z 
2022-05-18T05:49:31.8829973Z OK (skipped=4)
2022-05-18T05:49:31.8830130Z 
2022-05-18T05:49:31.8830256Z Generating XML reports...
2022-05-18T05:49:31.8894854Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20220518054925.xml
2022-05-18T05:49:32.1675828Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-05-18 05:49:32.167068]
2022-05-18T05:49:32.1676642Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:49:32.167174]
2022-05-18T05:49:33.0560370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71kx92pa
2022-05-18T05:49:33.0561602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71kx92pa/_remote_module_non_scriptable.py
2022-05-18T05:49:33.0728537Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec
2022-05-18T05:49:33.0749082Z 
2022-05-18T05:49:33.0749575Z Running tests...
2022-05-18T05:49:33.0750490Z ----------------------------------------------------------------------
2022-05-18T05:49:34.7365556Z   test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:49:34.7778118Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115527
2022-05-18T05:49:34.7903733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115528
2022-05-18T05:49:34.8034885Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 115529
2022-05-18T05:49:34.8168532Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 115530
2022-05-18T05:49:35.6986259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbtljy2r
2022-05-18T05:49:35.6987167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbtljy2r/_remote_module_non_scriptable.py
2022-05-18T05:49:35.7048705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkge9jw0m
2022-05-18T05:49:35.7051073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkge9jw0m/_remote_module_non_scriptable.py
2022-05-18T05:49:35.7122763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:35.7185735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:35.7239087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54ilb4zj
2022-05-18T05:49:35.7241366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54ilb4zj/_remote_module_non_scriptable.py
2022-05-18T05:49:35.7385525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:49:35.7555166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr0l20_76
2022-05-18T05:49:35.7556557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr0l20_76/_remote_module_non_scriptable.py
2022-05-18T05:49:35.7697142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:49:35.9208661Z ok (2.846s)
2022-05-18T05:49:35.9222818Z   test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec)
2022-05-18T05:49:35.9389117Z Test custom spec can be invoked from the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115671
2022-05-18T05:49:35.9500857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115672
2022-05-18T05:49:35.9622926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 115673
2022-05-18T05:49:35.9750677Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 115674
2022-05-18T05:49:36.9160543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9j2neqzx
2022-05-18T05:49:36.9161660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9j2neqzx/_remote_module_non_scriptable.py
2022-05-18T05:49:36.9271409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3iuzpff_
2022-05-18T05:49:36.9274118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3iuzpff_/_remote_module_non_scriptable.py
2022-05-18T05:49:36.9295884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:36.9409927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:49:36.9437705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9yq0t00
2022-05-18T05:49:36.9440474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9yq0t00/_remote_module_non_scriptable.py
2022-05-18T05:49:36.9577231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:36.9949815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprxxs9pt2
2022-05-18T05:49:36.9951015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprxxs9pt2/_remote_module_non_scriptable.py
2022-05-18T05:49:37.0093490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:49:37.1791717Z skip: Need at least 4 CUDA devices (1.258s)
2022-05-18T05:49:37.1803402Z   test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec)
2022-05-18T05:49:37.1971379Z Test sharded_tensor.ones(...) with the custom ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115815
2022-05-18T05:49:37.2090258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115816
2022-05-18T05:49:37.2210456Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 115817
2022-05-18T05:49:37.2337444Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 115818
2022-05-18T05:49:38.1155902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpccmtff7s
2022-05-18T05:49:38.1156841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpccmtff7s/_remote_module_non_scriptable.py
2022-05-18T05:49:38.1235021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4qzdax4
2022-05-18T05:49:38.1237888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4qzdax4/_remote_module_non_scriptable.py
2022-05-18T05:49:38.1270448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzm3wzop
2022-05-18T05:49:38.1273357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzm3wzop/_remote_module_non_scriptable.py
2022-05-18T05:49:38.1288627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:49:38.1372466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:38.1413498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:49:38.1489775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswaicuzy
2022-05-18T05:49:38.1491372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswaicuzy/_remote_module_non_scriptable.py
2022-05-18T05:49:38.1631821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:38.3376570Z skip: Need at least 4 CUDA devices (1.158s)
2022-05-18T05:49:38.3531830Z   test_chunked_sharding_spec (__main__.TestShardingSpec) ... ok (0.015s)
2022-05-18T05:49:38.3624052Z   test_device_placement (__main__.TestShardingSpec) ... ok (0.009s)
2022-05-18T05:49:38.3720821Z   test_enumerable_sharding_spec (__main__.TestShardingSpec) ... ok (0.010s)
2022-05-18T05:49:38.3744651Z   test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s)
2022-05-18T05:49:38.3758410Z   test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-05-18T05:49:38.3773077Z   test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-05-18T05:49:38.3896085Z   test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.012s)
2022-05-18T05:49:38.3896633Z 
2022-05-18T05:49:38.3897312Z ----------------------------------------------------------------------
2022-05-18T05:49:38.3897931Z Ran 10 tests in 5.315s
2022-05-18T05:49:38.3898265Z 
2022-05-18T05:49:38.3898457Z OK (skipped=2)
2022-05-18T05:49:38.3898747Z 
2022-05-18T05:49:38.3898981Z Generating XML reports...
2022-05-18T05:49:38.3956242Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220518054933.xml
2022-05-18T05:49:38.3968490Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220518054933.xml
2022-05-18T05:49:38.6740654Z Running distributed/_shard/sharded_tensor/ops/test_linear ... [2022-05-18 05:49:38.673535]
2022-05-18T05:49:38.6741463Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_linear.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:49:38.673639]
2022-05-18T05:49:39.5692917Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear
2022-05-18T05:49:39.5708692Z 
2022-05-18T05:49:39.5709418Z Running tests...
2022-05-18T05:49:39.5709889Z ----------------------------------------------------------------------
2022-05-18T05:49:41.2022875Z   test_sharded_linear_colwise (__main__.TestShardedTensorOpsLinear) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:49:41.2429377Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115996
2022-05-18T05:49:41.2556769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115997
2022-05-18T05:49:41.2687424Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 115998
2022-05-18T05:49:41.2822458Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 115999
2022-05-18T05:49:42.1721513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:42.1751882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:49:42.1848071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:49:42.1870460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:42.3865418Z skip: Need at least 4 CUDA devices (2.815s)
2022-05-18T05:49:42.4081076Z   test_sharded_linear_errors (__main__.TestShardedTensorOpsLinear) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116140
2022-05-18T05:49:42.4197439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116141
2022-05-18T05:49:42.4323310Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 116142
2022-05-18T05:49:42.4455252Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 116143
2022-05-18T05:49:43.3967565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:49:43.4435680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:43.4460711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:43.4660198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:49:43.6494235Z skip: Need at least 4 CUDA devices (1.263s)
2022-05-18T05:49:43.6680463Z   test_sharded_linear_rowwise (__main__.TestShardedTensorOpsLinear) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116284
2022-05-18T05:49:43.6796278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116285
2022-05-18T05:49:43.6923173Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 116286
2022-05-18T05:49:43.7055053Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 116287
2022-05-18T05:49:44.5920176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:44.5982290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:44.6029233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:49:44.6323700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:49:44.8091309Z skip: Need at least 4 CUDA devices (1.160s)
2022-05-18T05:49:44.8091629Z 
2022-05-18T05:49:44.8092044Z ----------------------------------------------------------------------
2022-05-18T05:49:44.8092444Z Ran 3 tests in 5.238s
2022-05-18T05:49:44.8092763Z 
2022-05-18T05:49:44.8092957Z OK (skipped=3)
2022-05-18T05:49:44.8093137Z 
2022-05-18T05:49:44.8093253Z Generating XML reports...
2022-05-18T05:49:44.8156357Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20220518054939.xml
2022-05-18T05:49:45.1102552Z Running distributed/_shard/sharded_tensor/ops/test_init ... [2022-05-18 05:49:45.109730]
2022-05-18T05:49:45.1103330Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_init.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:49:45.109835]
2022-05-18T05:49:46.0117642Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init
2022-05-18T05:49:46.0133130Z 
2022-05-18T05:49:46.0133284Z Running tests...
2022-05-18T05:49:46.0134021Z ----------------------------------------------------------------------
2022-05-18T05:49:46.0149991Z   test_init_sharded_tensor_with_kaiming_uniform (__main__.TestShardedTensorNNInit)
2022-05-18T05:49:47.6534949Z Test torch.nn.init.kaiming_uniform_(ShardedTensor, a, mode, nonlinearit) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:49:47.6940198Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116465
2022-05-18T05:49:47.7063031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116466
2022-05-18T05:49:47.7193087Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 116467
2022-05-18T05:49:47.7323467Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 116468
2022-05-18T05:49:48.6237028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:48.6364550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:49:48.6851426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:48.6852349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:49:48.8362830Z skip: Need at least 4 CUDA devices (2.823s)
2022-05-18T05:49:48.8378853Z   test_init_sharded_tensor_with_normal (__main__.TestShardedTensorNNInit)
2022-05-18T05:49:48.8544806Z Test torch.nn.init.normal_(ShardedTensor, mean, std) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116609
2022-05-18T05:49:48.8660118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116610
2022-05-18T05:49:48.8784981Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 116611
2022-05-18T05:49:48.8913706Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 116612
2022-05-18T05:49:49.7671806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:49.8243798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:49.8244335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:49:49.8652209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:49:50.0953302Z skip: Need at least 4 CUDA devices (1.259s)
2022-05-18T05:49:50.0968248Z   test_init_sharded_tensor_with_uniform (__main__.TestShardedTensorNNInit)
2022-05-18T05:49:50.1137773Z Test torch.nn.init.uniform_(ShardedTensor, a, b) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116753
2022-05-18T05:49:50.1251248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116754
2022-05-18T05:49:50.1376899Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 116755
2022-05-18T05:49:50.1511369Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 116756
2022-05-18T05:49:51.0804032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:49:51.0857711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:49:51.1050978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:49:51.1074298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:49:51.3548190Z skip: Need at least 4 CUDA devices (1.259s)
2022-05-18T05:49:51.3548482Z 
2022-05-18T05:49:51.3548879Z ----------------------------------------------------------------------
2022-05-18T05:49:51.3549230Z Ran 3 tests in 5.341s
2022-05-18T05:49:51.3549401Z 
2022-05-18T05:49:51.3549514Z OK (skipped=3)
2022-05-18T05:49:51.3549660Z 
2022-05-18T05:49:51.3550109Z Generating XML reports...
2022-05-18T05:49:51.3611850Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20220518054946.xml
2022-05-18T05:49:51.6404745Z Running distributed/elastic/utils/distributed_test ... [2022-05-18 05:49:51.639907]
2022-05-18T05:49:51.6406161Z Executing ['/opt/conda/bin/python', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:49:51.640023]
2022-05-18T05:49:52.5767818Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test
2022-05-18T05:49:52.5783766Z 
2022-05-18T05:49:52.5784069Z Running tests...
2022-05-18T05:49:52.5784507Z ----------------------------------------------------------------------
2022-05-18T05:49:54.2525117Z   test_create_store_multi (__main__.DistributedUtilTest) ... ok (1.674s)
2022-05-18T05:49:54.2537517Z   test_create_store_no_port_multi (__main__.DistributedUtilTest) ... ok (0.001s)
2022-05-18T05:49:54.2544122Z   test_create_store_single_server (__main__.DistributedUtilTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/66207 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.000s)
2022-05-18T05:49:57.2812346Z   test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... ok (3.027s)
2022-05-18T05:49:57.2821452Z   test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (e9eae2ba3bb0, 0).
2022-05-18T05:49:57.2823247Z ok (0.001s)
2022-05-18T05:49:57.2839749Z   test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... [W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:42271 (errno: 98 - Address already in use).
2022-05-18T05:49:57.2857462Z [W socket.cpp:401] [c10d] The server socket has failed to bind to 0.0.0.0:42271 (errno: 98 - Address already in use).
2022-05-18T05:49:57.2857944Z [E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address.
2022-05-18T05:49:57.2861303Z ok (0.004s)
2022-05-18T05:49:57.2890316Z   test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (e9eae2ba3bb0, 34963).
2022-05-18T05:49:57.2892220Z ok (0.003s)
2022-05-18T05:49:57.2893650Z 
2022-05-18T05:49:57.2894176Z ----------------------------------------------------------------------
2022-05-18T05:49:57.2894739Z Ran 7 tests in 4.711s
2022-05-18T05:49:57.2895061Z 
2022-05-18T05:49:57.2895228Z OK (skipped=1)
2022-05-18T05:49:57.2895370Z 
2022-05-18T05:49:57.2895498Z Generating XML reports...
2022-05-18T05:49:57.2938603Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220518054952.xml
2022-05-18T05:49:57.5633670Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2022-05-18 05:49:57.562793]
2022-05-18T05:49:57.5635103Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:49:57.562908]
2022-05-18T05:49:58.4966043Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward
2022-05-18T05:49:58.4982433Z 
2022-05-18T05:49:58.4982879Z Running tests...
2022-05-18T05:49:58.4983380Z ----------------------------------------------------------------------
2022-05-18T05:50:00.1478091Z   test_multi_forward (__main__.TestMultiForward) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:50:00.1901914Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116980
2022-05-18T05:50:00.2027361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116981
2022-05-18T05:50:01.1369688Z dist init r=0, world=2
2022-05-18T05:50:01.1372950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:50:01.1796703Z dist init r=1, world=2
2022-05-18T05:50:01.1801095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:50:01.1801892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:50:01.1880949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:50:02.6050459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:50:02.6051003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:50:02.9180913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:50:02.9181465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:50:02.9220395Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:50:02.9221343Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:50:02.9222204Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:50:02.9222845Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:50:03.2099920Z ok (4.711s)
2022-05-18T05:50:03.2100155Z 
2022-05-18T05:50:03.2100544Z ----------------------------------------------------------------------
2022-05-18T05:50:03.2100886Z Ran 1 test in 4.712s
2022-05-18T05:50:03.2101053Z 
2022-05-18T05:50:03.2101154Z OK
2022-05-18T05:50:03.2101270Z 
2022-05-18T05:50:03.2101402Z Generating XML reports...
2022-05-18T05:50:03.2145391Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20220518054958.xml
2022-05-18T05:50:03.4955828Z Running distributed/fsdp/test_fsdp_uneven ... [2022-05-18 05:50:03.495064]
2022-05-18T05:50:03.4956559Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:50:03.495181]
2022-05-18T05:50:04.4080883Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven
2022-05-18T05:50:04.4096578Z 
2022-05-18T05:50:04.4097016Z Running tests...
2022-05-18T05:50:04.4097506Z ----------------------------------------------------------------------
2022-05-18T05:50:04.4119265Z   test_one_iteration (__main__.TestUnevenParamShard)
2022-05-18T05:50:06.0486566Z Test FSDP with uneven divide of parameter shards. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:50:06.0895074Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117104
2022-05-18T05:50:06.1019385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117105
2022-05-18T05:50:07.0564583Z dist init r=0, world=2
2022-05-18T05:50:07.0567743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:50:07.0599329Z dist init r=1, world=2
2022-05-18T05:50:07.0603927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:50:07.0604961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:50:07.0671015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:50:08.4727471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:50:08.4727985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:50:09.1090760Z ok (4.699s)
2022-05-18T05:50:09.1090977Z 
2022-05-18T05:50:09.1091596Z ----------------------------------------------------------------------
2022-05-18T05:50:09.1091926Z Ran 1 test in 4.699s
2022-05-18T05:50:09.1092094Z 
2022-05-18T05:50:09.1092194Z OK
2022-05-18T05:50:09.1092328Z 
2022-05-18T05:50:09.1092460Z Generating XML reports...
2022-05-18T05:50:09.1137534Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20220518055004.xml
2022-05-18T05:50:09.4016188Z Running distributed/fsdp/test_fsdp_traversal ... [2022-05-18 05:50:09.401093]
2022-05-18T05:50:09.4017016Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:50:09.401205]
2022-05-18T05:50:10.3156426Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal
2022-05-18T05:50:10.3172769Z 
2022-05-18T05:50:10.3172918Z Running tests...
2022-05-18T05:50:10.3173662Z ----------------------------------------------------------------------
2022-05-18T05:50:11.9345541Z   test_fsdp_modules (__main__.TestTraversal) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:50:11.9757188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117228
2022-05-18T05:50:11.9879576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117229
2022-05-18T05:50:12.9279392Z dist init r=0, world=2
2022-05-18T05:50:12.9282963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:50:12.9638486Z dist init r=1, world=2
2022-05-18T05:50:12.9643008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:50:12.9644052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:50:12.9689714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:50:14.3754850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:50:14.3755380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:50:14.3965366Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:50:14.3966046Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:50:14.3966917Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:912: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:50:14.3967575Z   f"Module is input on CPU, we are moving it to {torch.cuda.current_device()}"
2022-05-18T05:50:14.6945641Z ok (4.377s)
2022-05-18T05:50:14.6945865Z 
2022-05-18T05:50:14.6946243Z ----------------------------------------------------------------------
2022-05-18T05:50:14.6946584Z Ran 1 test in 4.377s
2022-05-18T05:50:14.6946754Z 
2022-05-18T05:50:14.6946829Z OK
2022-05-18T05:50:14.6946965Z 
2022-05-18T05:50:14.6947095Z Generating XML reports...
2022-05-18T05:50:14.6991000Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20220518055010.xml
2022-05-18T05:50:14.9732054Z Running distributed/_shard/sharded_tensor/ops/test_embedding ... [2022-05-18 05:50:14.972702]
2022-05-18T05:50:14.9733084Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:50:14.972805]
2022-05-18T05:50:15.8770629Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding
2022-05-18T05:50:15.8786515Z 
2022-05-18T05:50:15.8786776Z Running tests...
2022-05-18T05:50:15.8787236Z ----------------------------------------------------------------------
2022-05-18T05:50:17.5414141Z   test_sharded_embedding_colwise (__main__.TestShardedEmbedding) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:50:17.5820968Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117348
2022-05-18T05:50:17.5944435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117349
2022-05-18T05:50:17.6074750Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 117350
2022-05-18T05:50:17.6207229Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 117351
2022-05-18T05:50:18.5140545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:50:18.5329058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:50:18.5638477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:50:18.6020599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:50:18.8251341Z skip: Need at least 4 CUDA devices (2.946s)
2022-05-18T05:50:18.8460922Z   test_sharded_embedding_rowwise (__main__.TestShardedEmbedding) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117492
2022-05-18T05:50:18.8582735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117493
2022-05-18T05:50:18.8717859Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 117494
2022-05-18T05:50:18.8855568Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 117495
2022-05-18T05:50:19.8736054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:50:19.8767967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:50:19.8777504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:50:19.9048715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:50:20.0897909Z skip: Need at least 4 CUDA devices (1.265s)
2022-05-18T05:50:20.0898177Z 
2022-05-18T05:50:20.0898545Z ----------------------------------------------------------------------
2022-05-18T05:50:20.0898888Z Ran 2 tests in 4.211s
2022-05-18T05:50:20.0899056Z 
2022-05-18T05:50:20.0899170Z OK (skipped=2)
2022-05-18T05:50:20.0899338Z 
2022-05-18T05:50:20.0899466Z Generating XML reports...
2022-05-18T05:50:20.0959963Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20220518055015.xml
2022-05-18T05:50:20.3720264Z Running distributed/_shard/sharded_tensor/ops/test_chunk ... [2022-05-18 05:50:20.371519]
2022-05-18T05:50:20.3721068Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_chunk.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:50:20.371625]
2022-05-18T05:50:21.2726945Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk
2022-05-18T05:50:21.2742875Z 
2022-05-18T05:50:21.2743134Z Running tests...
2022-05-18T05:50:21.2743554Z ----------------------------------------------------------------------
2022-05-18T05:50:22.9441928Z   test_sharded_chunk (__main__.TestShardedTensorChunkOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:50:22.9853608Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117673
2022-05-18T05:50:22.9978400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117674
2022-05-18T05:50:23.0107893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 117675
2022-05-18T05:50:23.0244096Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 117676
2022-05-18T05:50:23.9711906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:50:24.0111835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:50:24.0128703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:50:24.0731877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:50:24.2286309Z skip: Need at least 4 CUDA devices (2.954s)
2022-05-18T05:50:24.2475306Z   test_sharded_chunk_error (__main__.TestShardedTensorChunkOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117817
2022-05-18T05:50:24.2591236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117818
2022-05-18T05:50:24.2719108Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 117819
2022-05-18T05:50:24.2853408Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 117820
2022-05-18T05:50:25.2288477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:50:25.2947050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:50:25.2983903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:50:25.3221449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:50:25.4892819Z skip: Need at least 4 CUDA devices (1.260s)
2022-05-18T05:50:25.4893265Z 
2022-05-18T05:50:25.4894034Z ----------------------------------------------------------------------
2022-05-18T05:50:25.4894568Z Ran 2 tests in 4.215s
2022-05-18T05:50:25.4894738Z 
2022-05-18T05:50:25.4894849Z OK (skipped=2)
2022-05-18T05:50:25.4895010Z 
2022-05-18T05:50:25.4895139Z Generating XML reports...
2022-05-18T05:50:25.4953224Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20220518055021.xml
2022-05-18T05:50:25.7765633Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2022-05-18 05:50:25.776039]
2022-05-18T05:50:25.7766440Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:50:25.776144]
2022-05-18T05:50:26.6887420Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag
2022-05-18T05:50:26.6904350Z 
2022-05-18T05:50:26.6904615Z Running tests...
2022-05-18T05:50:26.6905072Z ----------------------------------------------------------------------
2022-05-18T05:50:28.3579359Z   test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:50:28.3994004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117998
2022-05-18T05:50:28.4119802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117999
2022-05-18T05:50:28.4249242Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 118000
2022-05-18T05:50:28.4383526Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 118001
2022-05-18T05:50:29.4321586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:50:29.4424418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:50:29.4684413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:50:29.5083201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:50:29.7429922Z skip: Need at least 4 CUDA devices (3.052s)
2022-05-18T05:50:29.7604489Z   test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118142
2022-05-18T05:50:29.7721838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118143
2022-05-18T05:50:29.7853303Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 118144
2022-05-18T05:50:29.7983396Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 118145
2022-05-18T05:50:30.7093582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:50:30.7443587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:50:30.7449962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:50:30.7668152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:50:31.0021252Z skip: Need at least 4 CUDA devices (1.259s)
2022-05-18T05:50:31.0021732Z 
2022-05-18T05:50:31.0022119Z ----------------------------------------------------------------------
2022-05-18T05:50:31.0022460Z Ran 2 tests in 4.312s
2022-05-18T05:50:31.0022625Z 
2022-05-18T05:50:31.0022740Z OK (skipped=2)
2022-05-18T05:50:31.0022881Z 
2022-05-18T05:50:31.0023004Z Generating XML reports...
2022-05-18T05:50:31.0081562Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220518055026.xml
2022-05-18T05:50:31.2944099Z Running distributed/fsdp/test_flatten_params_wrapper ... [2022-05-18 05:50:31.293799]
2022-05-18T05:50:31.2945537Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_flatten_params_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:50:31.293915]
2022-05-18T05:50:32.2124435Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper
2022-05-18T05:50:32.2144719Z 
2022-05-18T05:50:32.2145141Z Running tests...
2022-05-18T05:50:32.2145656Z ----------------------------------------------------------------------
2022-05-18T05:50:33.9054102Z   test_empty_module (__main__.TestFlattenParams) ... ok (1.691s)
2022-05-18T05:50:33.9154722Z   test_flatten_nothing (__main__.TestFlattenParams) ... ok (0.010s)
2022-05-18T05:50:33.9269759Z   test_num_params (__main__.TestFlattenParams) ... ok (0.011s)
2022-05-18T05:50:33.9552385Z   test_output (__main__.TestFlattenParams) ... ok (0.028s)
2022-05-18T05:50:33.9692468Z   test_partial_flattening (__main__.TestFlattenParams) ... ok (0.014s)
2022-05-18T05:50:33.9815663Z   test_sharded_flat_param (__main__.TestFlattenParams) ... ok (0.012s)
2022-05-18T05:50:33.9929648Z   test_shared_params_num_params (__main__.TestFlattenParams) ... ok (0.011s)
2022-05-18T05:50:34.0162482Z   test_shared_params_output (__main__.TestFlattenParams) ... ok (0.023s)
2022-05-18T05:50:34.0617612Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParams) ... ok (0.045s)
2022-05-18T05:50:34.0633520Z   test_empty_module (__main__.TestFlattenParamsCUDA) ... ok (0.002s)
2022-05-18T05:50:34.0750939Z   test_flatten_nothing (__main__.TestFlattenParamsCUDA) ... ok (0.012s)
2022-05-18T05:50:34.0890328Z   test_num_params (__main__.TestFlattenParamsCUDA) ... ok (0.014s)
2022-05-18T05:50:34.3991743Z   test_output (__main__.TestFlattenParamsCUDA) ... ok (0.310s)
2022-05-18T05:50:34.4167917Z   test_partial_flattening (__main__.TestFlattenParamsCUDA) ... ok (0.018s)
2022-05-18T05:50:34.4287185Z   test_sharded_flat_param (__main__.TestFlattenParamsCUDA) ... ok (0.012s)
2022-05-18T05:50:34.4428103Z   test_shared_params_num_params (__main__.TestFlattenParamsCUDA) ... ok (0.014s)
2022-05-18T05:50:34.4689500Z   test_shared_params_output (__main__.TestFlattenParamsCUDA) ... ok (0.026s)
2022-05-18T05:50:34.5249223Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDA) ... ok (0.056s)
2022-05-18T05:50:34.5266320Z   test_empty_module (__main__.TestFlattenParamsCUDAHalf) ... ok (0.002s)
2022-05-18T05:50:34.5727161Z   test_flatten_nothing (__main__.TestFlattenParamsCUDAHalf) ... ok (0.046s)
2022-05-18T05:50:34.5890511Z   test_num_params (__main__.TestFlattenParamsCUDAHalf) ... ok (0.016s)
2022-05-18T05:50:34.6189679Z   test_output (__main__.TestFlattenParamsCUDAHalf) ... ok (0.030s)
2022-05-18T05:50:34.6386156Z   test_partial_flattening (__main__.TestFlattenParamsCUDAHalf) ... ok (0.020s)
2022-05-18T05:50:34.6507791Z   test_sharded_flat_param (__main__.TestFlattenParamsCUDAHalf) ... ok (0.012s)
2022-05-18T05:50:34.6671669Z   test_shared_params_num_params (__main__.TestFlattenParamsCUDAHalf) ... ok (0.016s)
2022-05-18T05:50:34.6963583Z   test_shared_params_output (__main__.TestFlattenParamsCUDAHalf) ... ok (0.029s)
2022-05-18T05:50:34.7577884Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDAHalf) ... ok (0.061s)
2022-05-18T05:50:34.7578394Z 
2022-05-18T05:50:34.7578796Z ----------------------------------------------------------------------
2022-05-18T05:50:34.7579138Z Ran 27 tests in 2.543s
2022-05-18T05:50:34.7579567Z 
2022-05-18T05:50:34.7579665Z OK
2022-05-18T05:50:34.7579782Z 
2022-05-18T05:50:34.7579909Z Generating XML reports...
2022-05-18T05:50:34.7626748Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20220518055032.xml
2022-05-18T05:50:34.7640054Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20220518055032.xml
2022-05-18T05:50:34.7652618Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20220518055032.xml
2022-05-18T05:50:35.0402665Z Running distributed/elastic/utils/logging_test ... [2022-05-18 05:50:35.039717]
2022-05-18T05:50:35.0403464Z Executing ['/opt/conda/bin/python', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:50:35.039826]
2022-05-18T05:50:35.9936123Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test
2022-05-18T05:50:35.9953341Z 
2022-05-18T05:50:35.9953758Z Running tests...
2022-05-18T05:50:35.9954282Z ----------------------------------------------------------------------
2022-05-18T05:50:37.6961983Z   test_derive_module_name (__main__.LoggingTest) ... ok (1.700s)
2022-05-18T05:50:37.6986772Z   test_logger_name (__main__.LoggingTest) ... ok (0.003s)
2022-05-18T05:50:37.6987568Z 
2022-05-18T05:50:37.6988041Z ----------------------------------------------------------------------
2022-05-18T05:50:37.6988432Z Ran 2 tests in 1.703s
2022-05-18T05:50:37.6988613Z 
2022-05-18T05:50:37.6988709Z OK
2022-05-18T05:50:37.6988849Z 
2022-05-18T05:50:37.6988987Z Generating XML reports...
2022-05-18T05:50:37.7023680Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20220518055035.xml
2022-05-18T05:50:37.9593692Z Running distributed/nn/jit/test_instantiator ... [2022-05-18 05:50:37.958865]
2022-05-18T05:50:37.9594448Z Executing ['/opt/conda/bin/python', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:50:37.958971]
2022-05-18T05:50:38.8356563Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator
2022-05-18T05:50:38.8371220Z 
2022-05-18T05:50:38.8371643Z Running tests...
2022-05-18T05:50:38.8372136Z ----------------------------------------------------------------------
2022-05-18T05:50:40.4989922Z   test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (1.661s)
2022-05-18T05:50:40.5011318Z   test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s)
2022-05-18T05:50:40.5177280Z   test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.016s)
2022-05-18T05:50:40.5177641Z 
2022-05-18T05:50:40.5178026Z ----------------------------------------------------------------------
2022-05-18T05:50:40.5178371Z Ran 3 tests in 1.681s
2022-05-18T05:50:40.5178549Z 
2022-05-18T05:50:40.5178624Z OK
2022-05-18T05:50:40.5178759Z 
2022-05-18T05:50:40.5178885Z Generating XML reports...
2022-05-18T05:50:40.5214074Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220518055038.xml
2022-05-18T05:50:40.7697738Z Running distributed/test_nccl ... [2022-05-18 05:50:40.769255]
2022-05-18T05:50:40.7698416Z Executing ['/opt/conda/bin/python', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:50:40.769361]
2022-05-18T05:50:43.3397856Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl
2022-05-18T05:50:43.3420946Z 
2022-05-18T05:50:43.3421319Z Running tests...
2022-05-18T05:50:43.3421831Z ----------------------------------------------------------------------
2022-05-18T05:50:44.4277200Z   test_all_gather_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (1.085s)
2022-05-18T05:50:44.4325644Z   test_all_gather_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.005s)
2022-05-18T05:50:44.4381359Z   test_all_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.005s)
2022-05-18T05:50:44.4433603Z   test_all_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.005s)
2022-05-18T05:50:44.4479666Z   test_broadcast_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.005s)
2022-05-18T05:50:44.4527234Z   test_broadcast_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.005s)
2022-05-18T05:50:44.4556832Z   test_collective_errors_cuda (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-05-18T05:50:44.4595539Z   test_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-05-18T05:50:44.4637538Z   test_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-05-18T05:50:44.4688277Z   test_reduce_scatter_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.005s)
2022-05-18T05:50:44.4738499Z   test_reduce_scatter_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.005s)
2022-05-18T05:50:44.4757955Z   test_unique_id_cuda (__main__.TestNCCLCUDA) ... ok (0.002s)
2022-05-18T05:50:44.4758366Z 
2022-05-18T05:50:44.4758803Z ----------------------------------------------------------------------
2022-05-18T05:50:44.4759143Z Ran 12 tests in 1.134s
2022-05-18T05:50:44.4759309Z 
2022-05-18T05:50:44.4759405Z OK
2022-05-18T05:50:44.4759520Z 
2022-05-18T05:50:44.4759654Z Generating XML reports...
2022-05-18T05:50:44.4807357Z Generated XML report: test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20220518055043.xml
2022-05-18T05:50:44.7847562Z Running distributed/_shard/sharding_plan/test_sharding_plan ... [2022-05-18 05:50:44.784206]
2022-05-18T05:50:44.7848389Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:50:44.784309]
2022-05-18T05:50:45.7879116Z Running distributed/_shard/test_sharder ... [2022-05-18 05:50:45.787395]
2022-05-18T05:50:45.7879872Z Executing ['/opt/conda/bin/python', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:50:45.787502]
2022-05-18T05:50:46.6826413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8pmmece
2022-05-18T05:50:46.6827743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8pmmece/_remote_module_non_scriptable.py
2022-05-18T05:50:46.8173539Z Running distributed/elastic/timer/api_test ... [2022-05-18 05:50:46.816756]
2022-05-18T05:50:46.8174907Z Executing ['/opt/conda/bin/python', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:50:46.816862]
2022-05-18T05:50:47.7428451Z Running distributed/pipeline/sync/skip/test_api ... [2022-05-18 05:50:47.742332]
2022-05-18T05:50:47.7429423Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2022-05-18 05:50:47.742443]
2022-05-18T05:50:49.2629246Z ============================= test session starts ==============================
2022-05-18T05:50:49.2629826Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:50:49.2688921Z cachedir: .pytest_cache
2022-05-18T05:50:49.2689820Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:50:49.2690284Z torch: 1.12.0a0+git3b23752
2022-05-18T05:50:49.2690601Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:50:49.2690990Z plugins: hypothesis-4.53.2
2022-05-18T05:50:49.2847959Z collecting ... 
2022-05-18T05:50:49.2848347Z collected 3 items                                                              
2022-05-18T05:50:49.2853794Z 
2022-05-18T05:50:49.2888744Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%]
2022-05-18T05:50:49.2906782Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED   [ 66%]
2022-05-18T05:50:49.2947659Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED   [100%]
2022-05-18T05:50:49.2949501Z 
2022-05-18T05:50:49.2950222Z ============================== 3 passed in 0.03s ===============================
2022-05-18T05:50:49.4440897Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-05-18 05:50:49.443590]
2022-05-18T05:50:49.4441623Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-05-18 05:50:49.443710]
2022-05-18T05:50:50.6719059Z ============================= test session starts ==============================
2022-05-18T05:50:50.6719660Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:50:50.6740784Z cachedir: .pytest_cache
2022-05-18T05:50:50.6741371Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:50:50.6741835Z torch: 1.12.0a0+git3b23752
2022-05-18T05:50:50.6742169Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:50:50.6742554Z plugins: hypothesis-4.53.2
2022-05-18T05:50:50.6921410Z collecting ... 
2022-05-18T05:50:50.6921824Z collected 6 items                                                              
2022-05-18T05:50:50.6926850Z 
2022-05-18T05:50:50.6964008Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%]
2022-05-18T05:50:50.6986599Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%]
2022-05-18T05:50:50.7006519Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%]
2022-05-18T05:50:50.7027662Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%]
2022-05-18T05:50:50.7049539Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%]
2022-05-18T05:50:50.7108404Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%]
2022-05-18T05:50:50.7110603Z 
2022-05-18T05:50:50.7111244Z ============================== 6 passed in 0.04s ===============================
2022-05-18T05:50:50.8646833Z Running distributed/pipeline/sync/skip/test_portal ... [2022-05-18 05:50:50.864213]
2022-05-18T05:50:50.8647514Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-05-18 05:50:50.864327]
2022-05-18T05:50:52.1121694Z ============================= test session starts ==============================
2022-05-18T05:50:52.1122312Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:50:52.1142573Z cachedir: .pytest_cache
2022-05-18T05:50:52.1143830Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:50:52.1144414Z torch: 1.12.0a0+git3b23752
2022-05-18T05:50:52.1144727Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:50:52.1145110Z plugins: hypothesis-4.53.2
2022-05-18T05:50:52.1553405Z collecting ... 
2022-05-18T05:50:52.1554288Z collected 10 items                                                             
2022-05-18T05:50:52.1558470Z 
2022-05-18T05:50:53.4282653Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device PASSED [ 10%]
2022-05-18T05:50:53.4310852Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED   [ 20%]
2022-05-18T05:50:53.4331104Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%]
2022-05-18T05:50:53.4347896Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED      [ 40%]
2022-05-18T05:50:53.4365229Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%]
2022-05-18T05:50:53.4381926Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%]
2022-05-18T05:50:53.4398736Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%]
2022-05-18T05:50:53.4416402Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%]
2022-05-18T05:50:53.4433144Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%]
2022-05-18T05:50:53.4454860Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%]
2022-05-18T05:50:53.4456136Z 
2022-05-18T05:50:53.4456747Z ============================== 10 passed in 1.33s ==============================
2022-05-18T05:50:53.6622904Z Running distributed/pipeline/sync/skip/test_tracker ... [2022-05-18 05:50:53.661708]
2022-05-18T05:50:53.6623711Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2022-05-18 05:50:53.661828]
2022-05-18T05:50:54.8912367Z ============================= test session starts ==============================
2022-05-18T05:50:54.8912967Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:50:54.8932994Z cachedir: .pytest_cache
2022-05-18T05:50:54.8933925Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:50:54.8934402Z torch: 1.12.0a0+git3b23752
2022-05-18T05:50:54.8934880Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:50:54.8935396Z plugins: hypothesis-4.53.2
2022-05-18T05:50:54.9231821Z collecting ... 
2022-05-18T05:50:54.9232559Z collected 6 items                                                              
2022-05-18T05:50:54.9237079Z 
2022-05-18T05:50:54.9273621Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%]
2022-05-18T05:50:56.1610944Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel PASSED [ 33%]
2022-05-18T05:50:56.1628118Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%]
2022-05-18T05:50:56.1642718Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%]
2022-05-18T05:50:56.1658209Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%]
2022-05-18T05:50:56.1677364Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%]
2022-05-18T05:50:56.1679421Z 
2022-05-18T05:50:56.1680290Z ============================== 6 passed in 1.28s ===============================
2022-05-18T05:50:56.3754437Z Running distributed/pipeline/sync/test_balance ... [2022-05-18 05:50:56.374949]
2022-05-18T05:50:56.3755114Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-05-18 05:50:56.375059]
2022-05-18T05:50:57.6033550Z ============================= test session starts ==============================
2022-05-18T05:50:57.6034146Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:50:57.6054650Z cachedir: .pytest_cache
2022-05-18T05:50:57.6055682Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:50:57.6056152Z torch: 1.12.0a0+git3b23752
2022-05-18T05:50:57.6056467Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:50:57.6057164Z plugins: hypothesis-4.53.2
2022-05-18T05:50:57.6441296Z collecting ... 
2022-05-18T05:50:57.6441813Z collected 18 items                                                             
2022-05-18T05:50:57.6446383Z 
2022-05-18T05:50:57.6481356Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED    [  5%]
2022-05-18T05:50:57.6499059Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 11%]
2022-05-18T05:50:57.6518672Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 16%]
2022-05-18T05:50:57.6536810Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 22%]
2022-05-18T05:50:57.6550145Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 27%]
2022-05-18T05:50:57.6564142Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda] SKIPPED [ 33%]
2022-05-18T05:50:58.6618803Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 38%]
2022-05-18T05:50:59.9302177Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent PASSED [ 44%]
2022-05-18T05:51:00.2269734Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param PASSED [ 50%]
2022-05-18T05:51:00.2426357Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale PASSED [ 55%]
2022-05-18T05:51:00.2458863Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 61%]
2022-05-18T05:51:00.2490350Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda] PASSED [ 66%]
2022-05-18T05:51:01.2526537Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 72%]
2022-05-18T05:51:02.2566404Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] PASSED [ 77%]
2022-05-18T05:51:03.2605193Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED      [ 83%]
2022-05-18T05:51:04.2639459Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 88%]
2022-05-18T05:51:04.2669739Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple PASSED [ 94%]
2022-05-18T05:51:04.2701844Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED  [100%]
2022-05-18T05:51:04.2705429Z 
2022-05-18T05:51:04.2705797Z =========================== short test summary info ============================
2022-05-18T05:51:04.2706841Z SKIPPED [2] distributed/pipeline/sync/test_balance.py:47: Flaky due to time.sleep()
2022-05-18T05:51:04.2707793Z ======================== 16 passed, 2 skipped in 6.67s =========================
2022-05-18T05:51:04.5706007Z Running distributed/pipeline/sync/test_checkpoint ... [2022-05-18 05:51:04.570107]
2022-05-18T05:51:04.5706661Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-05-18 05:51:04.570217]
2022-05-18T05:51:05.8076118Z ============================= test session starts ==============================
2022-05-18T05:51:05.8076713Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:51:05.8098294Z cachedir: .pytest_cache
2022-05-18T05:51:05.8099298Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:51:05.8099767Z torch: 1.12.0a0+git3b23752
2022-05-18T05:51:05.8100122Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:51:05.8100494Z plugins: hypothesis-4.53.2
2022-05-18T05:51:05.8372902Z collecting ... 
2022-05-18T05:51:05.8373328Z collected 9 items                                                              
2022-05-18T05:51:05.8379407Z 
2022-05-18T05:51:05.8446469Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 11%]
2022-05-18T05:51:07.1195618Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda] PASSED [ 22%]
2022-05-18T05:51:07.1216078Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 33%]
2022-05-18T05:51:07.1236254Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 44%]
2022-05-18T05:51:07.1269873Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 55%]
2022-05-18T05:51:07.1307412Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda] PASSED [ 66%]
2022-05-18T05:51:07.1327687Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 77%]
2022-05-18T05:51:07.1344528Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 88%]
2022-05-18T05:51:07.1370242Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%]
2022-05-18T05:51:07.1372726Z 
2022-05-18T05:51:07.1373165Z ============================== 9 passed in 1.33s ===============================
2022-05-18T05:51:07.3557639Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2022-05-18 05:51:07.355125]
2022-05-18T05:51:07.3558663Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2022-05-18 05:51:07.355232]
2022-05-18T05:51:08.5591750Z ============================= test session starts ==============================
2022-05-18T05:51:08.5592865Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:51:08.5615551Z cachedir: .pytest_cache
2022-05-18T05:51:08.5616829Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:51:08.5617703Z torch: 1.12.0a0+git3b23752
2022-05-18T05:51:08.5618365Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:51:08.5619147Z plugins: hypothesis-4.53.2
2022-05-18T05:51:08.5927571Z collecting ... 
2022-05-18T05:51:08.5928434Z collected 11 items                                                             
2022-05-18T05:51:08.5934158Z 
2022-05-18T05:51:08.6737508Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [  9%]
2022-05-18T05:51:08.7346879Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%]
2022-05-18T05:51:08.7819582Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%]
2022-05-18T05:51:08.8254418Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%]
2022-05-18T05:51:08.8588313Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%]
2022-05-18T05:51:08.8910254Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%]
2022-05-18T05:51:08.8934274Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%]
2022-05-18T05:51:08.9345469Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED  [ 72%]
2022-05-18T05:51:09.0961995Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%]
2022-05-18T05:51:09.2039426Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%]
2022-05-18T05:51:09.2286623Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%]
2022-05-18T05:51:09.2287634Z 
2022-05-18T05:51:09.2287975Z ============================== 11 passed in 0.67s ==============================
2022-05-18T05:51:09.3818342Z Running distributed/pipeline/sync/test_inplace ... [2022-05-18 05:51:09.381329]
2022-05-18T05:51:09.3819031Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-05-18 05:51:09.381440]
2022-05-18T05:51:10.6147173Z ============================= test session starts ==============================
2022-05-18T05:51:10.6147719Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:51:10.6168104Z cachedir: .pytest_cache
2022-05-18T05:51:10.6168714Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:51:10.6169162Z torch: 1.12.0a0+git3b23752
2022-05-18T05:51:10.6169494Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:51:10.6170537Z plugins: hypothesis-4.53.2
2022-05-18T05:51:10.6304230Z collecting ... 
2022-05-18T05:51:10.6304623Z collected 3 items                                                              
2022-05-18T05:51:10.6309183Z 
2022-05-18T05:51:10.7181126Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%]
2022-05-18T05:51:10.7341011Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%]
2022-05-18T05:51:10.7502648Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%]
2022-05-18T05:51:10.7506689Z 
2022-05-18T05:51:10.7507301Z =========================== short test summary info ============================
2022-05-18T05:51:10.7508112Z XFAIL distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad
2022-05-18T05:51:10.7508608Z XFAIL distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad
2022-05-18T05:51:10.7509224Z ========================= 1 passed, 2 xfailed in 0.14s =========================
2022-05-18T05:51:10.9004995Z Running distributed/pipeline/sync/test_phony ... [2022-05-18 05:51:10.899999]
2022-05-18T05:51:10.9005634Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-05-18 05:51:10.900119]
2022-05-18T05:51:12.1555459Z ============================= test session starts ==============================
2022-05-18T05:51:12.1556020Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:51:12.1578157Z cachedir: .pytest_cache
2022-05-18T05:51:12.1579289Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:51:12.1579760Z torch: 1.12.0a0+git3b23752
2022-05-18T05:51:12.1580087Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:51:12.1580477Z plugins: hypothesis-4.53.2
2022-05-18T05:51:12.1758418Z collecting ... 
2022-05-18T05:51:12.1759211Z collected 4 items                                                              
2022-05-18T05:51:12.1764496Z 
2022-05-18T05:51:12.1800515Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED          [ 25%]
2022-05-18T05:51:12.1818990Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%]
2022-05-18T05:51:12.1836472Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED        [ 75%]
2022-05-18T05:51:12.1870077Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%]
2022-05-18T05:51:12.1871160Z 
2022-05-18T05:51:12.1871871Z ============================== 4 passed in 0.03s ===============================
2022-05-18T05:51:12.3251078Z Running distributed/pipeline/sync/test_pipeline ... [2022-05-18 05:51:12.324585]
2022-05-18T05:51:12.3251734Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2022-05-18 05:51:12.324697]
2022-05-18T05:51:13.5375049Z ============================= test session starts ==============================
2022-05-18T05:51:13.5375607Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:51:13.5395583Z cachedir: .pytest_cache
2022-05-18T05:51:13.5396190Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:51:13.5396650Z torch: 1.12.0a0+git3b23752
2022-05-18T05:51:13.5396986Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:51:13.5397375Z plugins: hypothesis-4.53.2
2022-05-18T05:51:13.5561553Z collecting ... 
2022-05-18T05:51:13.5562311Z collected 1 item                                                               
2022-05-18T05:51:13.5567227Z 
2022-05-18T05:51:13.5604319Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED     [100%]
2022-05-18T05:51:13.5606206Z 
2022-05-18T05:51:13.5606544Z ============================== 1 passed in 0.02s ===============================
2022-05-18T05:51:13.6954168Z Running distributed/pipeline/sync/test_transparency ... [2022-05-18 05:51:13.694944]
2022-05-18T05:51:13.6954848Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-05-18 05:51:13.695056]
2022-05-18T05:51:14.9581801Z ============================= test session starts ==============================
2022-05-18T05:51:14.9582370Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:51:14.9604129Z cachedir: .pytest_cache
2022-05-18T05:51:14.9605564Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:51:14.9606484Z torch: 1.12.0a0+git3b23752
2022-05-18T05:51:14.9607137Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:51:14.9607920Z plugins: hypothesis-4.53.2
2022-05-18T05:51:14.9732456Z collecting ... 
2022-05-18T05:51:14.9733274Z collected 1 item                                                               
2022-05-18T05:51:14.9739146Z 
2022-05-18T05:51:15.0800612Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%]
2022-05-18T05:51:15.0801207Z 
2022-05-18T05:51:15.0801796Z ============================== 1 passed in 0.12s ===============================
2022-05-18T05:51:15.2291508Z Running distributed/rpc/test_faulty_agent ... [2022-05-18 05:51:15.228560]
2022-05-18T05:51:15.2292314Z Executing ['/opt/conda/bin/python', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:51:15.228668]
2022-05-18T05:51:16.1169929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzds4ty_b
2022-05-18T05:51:16.1171258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzds4ty_b/_remote_module_non_scriptable.py
2022-05-18T05:51:17.4790512Z 
2022-05-18T05:51:17.4791112Z real	88m25.309s
2022-05-18T05:51:17.4791429Z user	135m13.299s
2022-05-18T05:51:17.4791653Z sys	110m36.842s
2022-05-18T05:51:17.4791901Z + assert_git_not_dirty
2022-05-18T05:51:17.4792468Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *rocm* ]]
2022-05-18T05:51:17.4793253Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *xla* ]]
2022-05-18T05:51:17.4794456Z ++ git status --porcelain
2022-05-18T05:51:17.8472544Z + git_status=
2022-05-18T05:51:17.8473015Z + [[ -n '' ]]
2022-05-18T05:51:17.8473456Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *cuda* ]]
2022-05-18T05:51:17.8474007Z + [[ 1 == 1 ]]
2022-05-18T05:51:17.8474992Z + echo 'Testing distributed C++ tests'
2022-05-18T05:51:17.8475513Z Testing distributed C++ tests
2022-05-18T05:51:17.8478804Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cpp.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.7/site-packages/torch/bin
2022-05-18T05:51:17.8490819Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.7/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.7/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.7/site-packages/torch/bin
2022-05-18T05:51:17.8500514Z + TEST_REPORTS_DIR=test/test-reports/cpp-distributed/test_distributed
2022-05-18T05:51:17.8501017Z + mkdir -p test/test-reports/cpp-distributed/test_distributed
2022-05-18T05:51:17.8514103Z + /opt/conda/lib/python3.7/site-packages/torch/bin/FileStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml
2022-05-18T05:51:18.0806863Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-05-18T05:51:18.0807463Z [==========] Running 4 tests from 1 test suite.
2022-05-18T05:51:18.0807873Z [----------] Global test environment set-up.
2022-05-18T05:51:18.0808289Z [----------] 4 tests from FileStoreTest
2022-05-18T05:51:18.0808693Z [ RUN      ] FileStoreTest.testGetAndSet
2022-05-18T05:51:18.0812696Z [       OK ] FileStoreTest.testGetAndSet (0 ms)
2022-05-18T05:51:18.0813219Z [ RUN      ] FileStoreTest.testGetAndSetWithPrefix
2022-05-18T05:51:18.0817386Z [       OK ] FileStoreTest.testGetAndSetWithPrefix (0 ms)
2022-05-18T05:51:18.0817859Z [ RUN      ] FileStoreTest.testStressStore
2022-05-18T05:51:18.0997269Z [       OK ] FileStoreTest.testStressStore (17 ms)
2022-05-18T05:51:18.0997756Z [ RUN      ] FileStoreTest.testStressStoreWithPrefix
2022-05-18T05:51:18.1180144Z [       OK ] FileStoreTest.testStressStoreWithPrefix (18 ms)
2022-05-18T05:51:18.1180648Z [----------] 4 tests from FileStoreTest (37 ms total)
2022-05-18T05:51:18.1180848Z 
2022-05-18T05:51:18.1181082Z [----------] Global test environment tear-down
2022-05-18T05:51:18.1182466Z [==========] 4 tests from 1 test suite ran. (37 ms total)
2022-05-18T05:51:18.1182855Z [  PASSED  ] 4 tests.
2022-05-18T05:51:18.1599492Z + /opt/conda/lib/python3.7/site-packages/torch/bin/HashStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml
2022-05-18T05:51:18.3907956Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-05-18T05:51:18.3908619Z [==========] Running 4 tests from 1 test suite.
2022-05-18T05:51:18.3909036Z [----------] Global test environment set-up.
2022-05-18T05:51:18.3909442Z [----------] 4 tests from HashStoreTest
2022-05-18T05:51:18.3909824Z [ RUN      ] HashStoreTest.testGetAndSet
2022-05-18T05:51:18.4914400Z [       OK ] HashStoreTest.testGetAndSet (100 ms)
2022-05-18T05:51:18.4914879Z [ RUN      ] HashStoreTest.testGetAndSetWithPrefix
2022-05-18T05:51:18.5915891Z [       OK ] HashStoreTest.testGetAndSetWithPrefix (100 ms)
2022-05-18T05:51:18.5916383Z [ RUN      ] HashStoreTest.testStressStore
2022-05-18T05:51:18.5922897Z [       OK ] HashStoreTest.testStressStore (0 ms)
2022-05-18T05:51:18.5923421Z [ RUN      ] HashStoreTest.testStressStoreWithPrefix
2022-05-18T05:51:18.5928655Z [       OK ] HashStoreTest.testStressStoreWithPrefix (0 ms)
2022-05-18T05:51:18.5929447Z [----------] 4 tests from HashStoreTest (202 ms total)
2022-05-18T05:51:18.5930036Z 
2022-05-18T05:51:18.5930285Z [----------] Global test environment tear-down
2022-05-18T05:51:18.5931223Z [==========] 4 tests from 1 test suite ran. (202 ms total)
2022-05-18T05:51:18.5931584Z [  PASSED  ] 4 tests.
2022-05-18T05:51:18.6373124Z + /opt/conda/lib/python3.7/site-packages/torch/bin/TCPStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml
2022-05-18T05:51:18.8683007Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-05-18T05:51:18.8683837Z [==========] Running 11 tests from 1 test suite.
2022-05-18T05:51:18.8684245Z [----------] Global test environment set-up.
2022-05-18T05:51:18.8684642Z [----------] 11 tests from TCPStoreTest
2022-05-18T05:51:18.8685023Z [ RUN      ] TCPStoreTest.testHelper
2022-05-18T05:51:19.8540091Z [       OK ] TCPStoreTest.testHelper (985 ms)
2022-05-18T05:51:19.8540567Z [ RUN      ] TCPStoreTest.testHelperPrefix
2022-05-18T05:51:20.8642776Z [       OK ] TCPStoreTest.testHelperPrefix (1010 ms)
2022-05-18T05:51:20.8643725Z [ RUN      ] TCPStoreTest.testWatchKeyCallback
2022-05-18T05:51:20.8781695Z [       OK ] TCPStoreTest.testWatchKeyCallback (13 ms)
2022-05-18T05:51:20.8782856Z [ RUN      ] TCPStoreTest.testWatchKeyCallbackWithPrefix
2022-05-18T05:51:20.8918509Z [       OK ] TCPStoreTest.testWatchKeyCallbackWithPrefix (13 ms)
2022-05-18T05:51:20.8919380Z [ RUN      ] TCPStoreTest.testKeyEmptyUpdate
2022-05-18T05:51:21.0977676Z [       OK ] TCPStoreTest.testKeyEmptyUpdate (205 ms)
2022-05-18T05:51:21.0978658Z [ RUN      ] TCPStoreTest.testKeyUpdate
2022-05-18T05:51:21.0984233Z [       OK ] TCPStoreTest.testKeyUpdate (0 ms)
2022-05-18T05:51:21.0985191Z [ RUN      ] TCPStoreTest.testKeyCreate
2022-05-18T05:51:21.0989356Z [       OK ] TCPStoreTest.testKeyCreate (0 ms)
2022-05-18T05:51:21.0990270Z [ RUN      ] TCPStoreTest.testKeyAdd
2022-05-18T05:51:21.0994791Z [       OK ] TCPStoreTest.testKeyAdd (0 ms)
2022-05-18T05:51:21.0995721Z [ RUN      ] TCPStoreTest.testKeyDelete
2022-05-18T05:51:21.3057057Z [       OK ] TCPStoreTest.testKeyDelete (206 ms)
2022-05-18T05:51:21.3057988Z [ RUN      ] TCPStoreTest.testCleanShutdown
2022-05-18T05:51:21.3065235Z [       OK ] TCPStoreTest.testCleanShutdown (0 ms)
2022-05-18T05:51:21.3066364Z [ RUN      ] TCPStoreTest.testMultiTenantStores
2022-05-18T05:51:21.3078163Z [       OK ] TCPStoreTest.testMultiTenantStores (1 ms)
2022-05-18T05:51:21.3079178Z [----------] 11 tests from TCPStoreTest (2439 ms total)
2022-05-18T05:51:21.3079454Z 
2022-05-18T05:51:21.3079691Z [----------] Global test environment tear-down
2022-05-18T05:51:21.3082638Z [==========] 11 tests from 1 test suite ran. (2439 ms total)
2022-05-18T05:51:21.3083419Z [  PASSED  ] 11 tests.
2022-05-18T05:51:21.3575583Z ++ command -v mpiexec
2022-05-18T05:51:21.3578566Z + MPIEXEC=/usr/bin/mpiexec
2022-05-18T05:51:21.3579206Z + [[ -n /usr/bin/mpiexec ]]
2022-05-18T05:51:21.3579797Z + [[ -z true ]]
2022-05-18T05:51:21.3580859Z + /opt/conda/lib/python3.7/site-packages/torch/bin/ProcessGroupGlooTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml
2022-05-18T05:51:21.6301538Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-05-18T05:51:21.6302730Z [==========] Running 12 tests from 1 test suite.
2022-05-18T05:51:21.6303151Z [----------] Global test environment set-up.
2022-05-18T05:51:21.6303580Z [----------] 12 tests from ProcessGroupGlooTest
2022-05-18T05:51:21.6304308Z [ RUN      ] ProcessGroupGlooTest.testSIGSTOPException
2022-05-18T05:51:22.6828748Z [       OK ] ProcessGroupGlooTest.testSIGSTOPException (1052 ms)
2022-05-18T05:51:22.6829893Z [ RUN      ] ProcessGroupGlooTest.testSIGKILLException
2022-05-18T05:51:22.7069909Z [       OK ] ProcessGroupGlooTest.testSIGKILLException (24 ms)
2022-05-18T05:51:22.7071164Z [ RUN      ] ProcessGroupGlooTest.testAllReduceCPU
2022-05-18T05:51:22.9766173Z [       OK ] ProcessGroupGlooTest.testAllReduceCPU (269 ms)
2022-05-18T05:51:22.9766730Z [ RUN      ] ProcessGroupGlooTest.testBroadcastCPU
2022-05-18T05:51:23.0194775Z [       OK ] ProcessGroupGlooTest.testBroadcastCPU (42 ms)
2022-05-18T05:51:23.0195331Z [ RUN      ] ProcessGroupGlooTest.testAllToAllCPU
2022-05-18T05:51:23.1736560Z [       OK ] ProcessGroupGlooTest.testAllToAllCPU (154 ms)
2022-05-18T05:51:23.1737125Z [ RUN      ] ProcessGroupGlooTest.testBarrier
2022-05-18T05:51:23.2151162Z [       OK ] ProcessGroupGlooTest.testBarrier (41 ms)
2022-05-18T05:51:23.2151699Z [ RUN      ] ProcessGroupGlooTest.testMonitoredBarrier
2022-05-18T05:51:24.2364335Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 1000 ms
2022-05-18T05:51:24.2567991Z [       OK ] ProcessGroupGlooTest.testMonitoredBarrier (1041 ms)
2022-05-18T05:51:24.2569052Z [ RUN      ] ProcessGroupGlooTest.testSequenceNumInit
2022-05-18T05:51:24.4000570Z [       OK ] ProcessGroupGlooTest.testSequenceNumInit (143 ms)
2022-05-18T05:51:24.4001109Z [ RUN      ] ProcessGroupGlooTest.testSend
2022-05-18T05:51:24.4522923Z [       OK ] ProcessGroupGlooTest.testSend (52 ms)
2022-05-18T05:51:24.4523452Z [ RUN      ] ProcessGroupGlooTest.testRecv
2022-05-18T05:51:24.5043858Z [       OK ] ProcessGroupGlooTest.testRecv (52 ms)
2022-05-18T05:51:24.5044399Z [ RUN      ] ProcessGroupGlooTest.testStoreSetGet
2022-05-18T05:51:24.5456351Z [       OK ] ProcessGroupGlooTest.testStoreSetGet (41 ms)
2022-05-18T05:51:24.5456849Z [ RUN      ] ProcessGroupGlooTest.testWaitDelay
2022-05-18T05:51:24.6877299Z [       OK ] ProcessGroupGlooTest.testWaitDelay (142 ms)
2022-05-18T05:51:24.6877880Z [----------] 12 tests from ProcessGroupGlooTest (3057 ms total)
2022-05-18T05:51:24.6878143Z 
2022-05-18T05:51:24.6878398Z [----------] Global test environment tear-down
2022-05-18T05:51:24.6881610Z [==========] 12 tests from 1 test suite ran. (3057 ms total)
2022-05-18T05:51:24.6882029Z [  PASSED  ] 12 tests.
2022-05-18T05:51:24.7582326Z + /opt/conda/lib/python3.7/site-packages/torch/bin/ProcessGroupNCCLTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml
2022-05-18T05:51:25.0304863Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-05-18T05:51:25.0305548Z [==========] Running 11 tests from 1 test suite.
2022-05-18T05:51:25.0306000Z [----------] Global test environment set-up.
2022-05-18T05:51:25.0306509Z [----------] 11 tests from ProcessGroupNCCLTest
2022-05-18T05:51:25.0306990Z [ RUN      ] ProcessGroupNCCLTest.testAllreduce
2022-05-18T05:51:32.7309104Z [       OK ] ProcessGroupNCCLTest.testAllreduce (7700 ms)
2022-05-18T05:51:32.7309654Z [ RUN      ] ProcessGroupNCCLTest.testBroadcast
2022-05-18T05:51:32.8144554Z [       OK ] ProcessGroupNCCLTest.testBroadcast (83 ms)
2022-05-18T05:51:32.8145053Z [ RUN      ] ProcessGroupNCCLTest.testReduce
2022-05-18T05:51:32.8985347Z [       OK ] ProcessGroupNCCLTest.testReduce (84 ms)
2022-05-18T05:51:32.8985870Z [ RUN      ] ProcessGroupNCCLTest.testAllgather
2022-05-18T05:51:32.9795970Z [       OK ] ProcessGroupNCCLTest.testAllgather (81 ms)
2022-05-18T05:51:32.9796786Z [ RUN      ] ProcessGroupNCCLTest.testAllgatherBase
2022-05-18T05:51:33.0505714Z [       OK ] ProcessGroupNCCLTest.testAllgatherBase (70 ms)
2022-05-18T05:51:33.0506297Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatter
2022-05-18T05:51:33.1280572Z [       OK ] ProcessGroupNCCLTest.testReduceScatter (77 ms)
2022-05-18T05:51:33.1281139Z [ RUN      ] ProcessGroupNCCLTest.testSequenceNumInit
2022-05-18T05:51:33.1809457Z [       OK ] ProcessGroupNCCLTest.testSequenceNumInit (52 ms)
2022-05-18T05:51:33.1810686Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout
2022-05-18T05:51:36.1825883Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout (3001 ms)
2022-05-18T05:51:36.1826672Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException
2022-05-18T05:51:39.1837461Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException (3001 ms)
2022-05-18T05:51:39.1838148Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatterBase
2022-05-18T05:51:39.2601253Z [       OK ] ProcessGroupNCCLTest.testReduceScatterBase (76 ms)
2022-05-18T05:51:39.2601877Z [ RUN      ] ProcessGroupNCCLTest.testBackendName
2022-05-18T05:51:39.2971682Z [       OK ] ProcessGroupNCCLTest.testBackendName (37 ms)
2022-05-18T05:51:39.2972248Z [----------] 11 tests from ProcessGroupNCCLTest (14266 ms total)
2022-05-18T05:51:39.2972481Z 
2022-05-18T05:51:39.2972717Z [----------] Global test environment tear-down
2022-05-18T05:51:39.2975886Z [==========] 11 tests from 1 test suite ran. (14266 ms total)
2022-05-18T05:51:39.2976260Z [  PASSED  ] 11 tests.
2022-05-18T05:51:40.0356600Z + /opt/conda/lib/python3.7/site-packages/torch/bin/ProcessGroupNCCLErrorsTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml
2022-05-18T05:51:40.3240497Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-05-18T05:51:40.3241181Z [==========] Running 3 tests from 1 test suite.
2022-05-18T05:51:40.3241598Z [----------] Global test environment set-up.
2022-05-18T05:51:40.3242055Z [----------] 3 tests from ProcessGroupNCCLErrorsTest
2022-05-18T05:51:40.3242568Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking
2022-05-18T05:51:46.4161659Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking (6091 ms)
2022-05-18T05:51:46.4162765Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking
2022-05-18T05:51:49.4634645Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking (3047 ms)
2022-05-18T05:51:49.4635606Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking
2022-05-18T05:51:49.5165154Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking (52 ms)
2022-05-18T05:51:49.5165964Z [----------] 3 tests from ProcessGroupNCCLErrorsTest (9192 ms total)
2022-05-18T05:51:49.5166403Z 
2022-05-18T05:51:49.5166630Z [----------] Global test environment tear-down
2022-05-18T05:51:49.5167065Z [==========] 3 tests from 1 test suite ran. (9192 ms total)
2022-05-18T05:51:49.5167430Z [  PASSED  ] 3 tests.
2022-05-18T05:51:50.2595088Z + [[ 1 == 1 ]]
2022-05-18T05:51:50.2595490Z + test_rpc
2022-05-18T05:51:50.2596153Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *rocm* ]]
2022-05-18T05:51:50.2596566Z + echo 'Testing RPC C++ tests'
2022-05-18T05:51:50.2596921Z Testing RPC C++ tests
2022-05-18T05:51:50.2598689Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cpp.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.7/site-packages/torch/bin
2022-05-18T05:51:50.2611909Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.7/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.7/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.7/site-packages/torch/bin
2022-05-18T05:51:50.2623083Z + ln -sf '/opt/conda/lib/python3.7/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.7/site-packages/torch/bin
2022-05-18T05:51:50.2632073Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc
2022-05-18T05:51:50.2632712Z + mkdir -p test/test-reports/cpp-rpc/test_rpc
2022-05-18T05:51:50.2645342Z + /opt/conda/lib/python3.7/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml
2022-05-18T05:51:54.3079698Z [==========] Running 8 tests from 3 test suites.
2022-05-18T05:51:54.3080168Z [----------] Global test environment set-up.
2022-05-18T05:51:54.3080683Z [----------] 4 tests from WireSerialize
2022-05-18T05:51:54.3081202Z [ RUN      ] WireSerialize.Base
2022-05-18T05:51:54.3267476Z [       OK ] WireSerialize.Base (18 ms)
2022-05-18T05:51:54.3267918Z [ RUN      ] WireSerialize.RecopySparseTensors
2022-05-18T05:51:54.3370778Z [       OK ] WireSerialize.RecopySparseTensors (10 ms)
2022-05-18T05:51:54.3371268Z [ RUN      ] WireSerialize.CloneSparseTensors
2022-05-18T05:51:54.3461804Z [       OK ] WireSerialize.CloneSparseTensors (9 ms)
2022-05-18T05:51:54.3462230Z [ RUN      ] WireSerialize.Errors
2022-05-18T05:51:54.3488651Z [       OK ] WireSerialize.Errors (2 ms)
2022-05-18T05:51:54.3489098Z [----------] 4 tests from WireSerialize (41 ms total)
2022-05-18T05:51:54.3489330Z 
2022-05-18T05:51:54.3489834Z [----------] 1 test from TestE2ETensorPipe
2022-05-18T05:51:54.3490283Z [ RUN      ] TestE2ETensorPipe.TestTrainingLoop
2022-05-18T05:51:55.1622100Z [W tensorpipe_agent.cpp:728] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T05:51:55.1640969Z [       OK ] TestE2ETensorPipe.TestTrainingLoop (815 ms)
2022-05-18T05:51:55.1641689Z [----------] 1 test from TestE2ETensorPipe (815 ms total)
2022-05-18T05:51:55.1641930Z 
2022-05-18T05:51:55.1642172Z [----------] 3 tests from TensorpipeSerialize
2022-05-18T05:51:55.1642787Z [ RUN      ] TensorpipeSerialize.Base
2022-05-18T05:51:55.1643232Z [       OK ] TensorpipeSerialize.Base (0 ms)
2022-05-18T05:51:55.1643676Z [ RUN      ] TensorpipeSerialize.RecopySparseTensors
2022-05-18T05:51:55.1739951Z [       OK ] TensorpipeSerialize.RecopySparseTensors (9 ms)
2022-05-18T05:51:55.1740756Z [ RUN      ] TensorpipeSerialize.NoDeleterTensors
2022-05-18T05:51:55.1741261Z [       OK ] TensorpipeSerialize.NoDeleterTensors (0 ms)
2022-05-18T05:51:55.1741749Z [----------] 3 tests from TensorpipeSerialize (9 ms total)
2022-05-18T05:51:55.1742042Z 
2022-05-18T05:51:55.1742482Z [----------] Global test environment tear-down
2022-05-18T05:51:55.1744040Z [==========] 8 tests from 3 test suites ran. (866 ms total)
2022-05-18T05:51:55.1744764Z [  PASSED  ] 8 tests.
2022-05-18T05:51:55.1745147Z 
2022-05-18T05:51:55.1745507Z   YOU HAVE 1 DISABLED TEST
2022-05-18T05:51:55.1745838Z 
2022-05-18T05:51:55.7272119Z + cleanup
2022-05-18T05:51:55.7272697Z + retcode=0
2022-05-18T05:51:55.7273138Z + set +x
2022-05-18T05:51:55.7273459Z EXITED_USER_LAND
2022-05-18T05:51:55.7351843Z ##[group]Run pytorch/pytorch/.github/actions/get-workflow-job-id@master
2022-05-18T05:51:55.7352313Z with:
2022-05-18T05:51:55.7352880Z   github-token: ***
2022-05-18T05:51:55.7353128Z env:
2022-05-18T05:51:55.7353326Z   IN_CI: 1
2022-05-18T05:51:55.7353550Z   IS_GHA: 1
2022-05-18T05:51:55.7353801Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:51:55.7354051Z   GPU_FLAG: --gpus all
2022-05-18T05:51:55.7354305Z ##[endgroup]
2022-05-18T05:51:55.7399759Z ##[group]Run nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a
2022-05-18T05:51:55.7400075Z with:
2022-05-18T05:51:55.7400296Z   shell: bash
2022-05-18T05:51:55.7400519Z   timeout_minutes: 10
2022-05-18T05:51:55.7400767Z   max_attempts: 5
2022-05-18T05:51:55.7401016Z   retry_wait_seconds: 30
2022-05-18T05:51:55.7401540Z   command: set -x
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}"

2022-05-18T05:51:55.7402043Z   polling_interval_seconds: 1
2022-05-18T05:51:55.7402300Z   warning_on_retry: true
2022-05-18T05:51:55.7402559Z   continue_on_error: false
2022-05-18T05:51:55.7402801Z env:
2022-05-18T05:51:55.7402995Z   IN_CI: 1
2022-05-18T05:51:55.7403215Z   IS_GHA: 1
2022-05-18T05:51:55.7403460Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:51:55.7403706Z   GPU_FLAG: --gpus all
2022-05-18T05:51:55.7404096Z   GITHUB_TOKEN: ***
2022-05-18T05:51:55.7404341Z ##[endgroup]
2022-05-18T05:51:55.7840432Z 
2022-05-18T05:51:55.7915224Z + python3 -m pip install requests==2.26.0
2022-05-18T05:51:56.8766866Z Defaulting to user installation because normal site-packages is not writeable
2022-05-18T05:51:57.0232544Z Collecting requests==2.26.0
2022-05-18T05:51:57.0448120Z   Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
2022-05-18T05:51:57.1049258Z Collecting idna<4,>=2.5; python_version >= "3"
2022-05-18T05:51:57.1093297Z   Downloading idna-3.3-py3-none-any.whl (61 kB)
2022-05-18T05:51:57.1629361Z Collecting certifi>=2017.4.17
2022-05-18T05:51:57.1707283Z   Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
2022-05-18T05:51:57.2265139Z Collecting charset-normalizer~=2.0.0; python_version >= "3"
2022-05-18T05:51:57.2327361Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2022-05-18T05:51:57.3293761Z Collecting urllib3<1.27,>=1.21.1
2022-05-18T05:51:57.3334370Z   Downloading urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
2022-05-18T05:51:57.4433179Z Installing collected packages: idna, certifi, charset-normalizer, urllib3, requests
2022-05-18T05:51:57.5658808Z   WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-05-18T05:51:57.5659456Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-05-18T05:51:57.6923351Z Successfully installed certifi-2021.10.8 charset-normalizer-2.0.12 idna-3.3 requests-2.26.0 urllib3-1.26.9
2022-05-18T05:51:57.8745842Z ++ python3 .github/scripts/get_workflow_job_id.py 2342799944 i-0e12f07c7a192d642
2022-05-18T05:51:59.1718107Z + GHA_WORKFLOW_JOB_ID=6482805607
2022-05-18T05:51:59.1719019Z + echo '::set-output name=job-id::6482805607'
2022-05-18T05:51:59.7938916Z Command completed after 1 attempt(s).
2022-05-18T05:51:59.7939306Z 
2022-05-18T05:51:59.8100681Z Prepare all required actions
2022-05-18T05:51:59.8101139Z Getting action download info
2022-05-18T05:51:59.9484321Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2)
2022-05-18T05:52:00.0958662Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-05-18T05:52:00.0958960Z with:
2022-05-18T05:52:00.0959299Z   file-suffix: test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482805607
2022-05-18T05:52:00.0959643Z env:
2022-05-18T05:52:00.0959861Z   IN_CI: 1
2022-05-18T05:52:00.0960067Z   IS_GHA: 1
2022-05-18T05:52:00.0960314Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:52:00.0960580Z   GPU_FLAG: --gpus all
2022-05-18T05:52:00.0960812Z ##[endgroup]
2022-05-18T05:52:00.0989075Z ##[group]Run # Remove any previous test jsons if they exist
2022-05-18T05:52:00.0989562Z # Remove any previous test jsons if they exist
2022-05-18T05:52:00.0989873Z rm -f test-jsons-*.zip
2022-05-18T05:52:00.0990226Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-05-18T05:52:00.1003041Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T05:52:00.1003322Z env:
2022-05-18T05:52:00.1003539Z   IN_CI: 1
2022-05-18T05:52:00.1003759Z   IS_GHA: 1
2022-05-18T05:52:00.1003991Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:52:00.1004257Z   GPU_FLAG: --gpus all
2022-05-18T05:52:00.1004624Z   FILE_SUFFIX: test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482805607
2022-05-18T05:52:00.1004963Z ##[endgroup]
2022-05-18T05:52:00.1196985Z   adding: test/allowlist_for_publicAPI.json (deflated 82%)
2022-05-18T05:52:00.1231242Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-05-18T05:52:00.1232385Z   adding: test/.pytorch-slow-tests.json (deflated 71%)
2022-05-18T05:52:00.1236684Z   adding: test/.pytorch-disabled-tests.json (deflated 83%)
2022-05-18T05:52:00.1260269Z ##[group]Run # Remove any previous test reports if they exist
2022-05-18T05:52:00.1260664Z # Remove any previous test reports if they exist
2022-05-18T05:52:00.1260993Z rm -f test-reports-*.zip
2022-05-18T05:52:00.1261314Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml'
2022-05-18T05:52:00.1273413Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T05:52:00.1273712Z env:
2022-05-18T05:52:00.1273931Z   IN_CI: 1
2022-05-18T05:52:00.1274135Z   IS_GHA: 1
2022-05-18T05:52:00.1274383Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:52:00.1274649Z   GPU_FLAG: --gpus all
2022-05-18T05:52:00.1275000Z   FILE_SUFFIX: test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482805607
2022-05-18T05:52:00.1275357Z ##[endgroup]
2022-05-18T05:52:00.1449407Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042306.xml (deflated 42%)
2022-05-18T05:52:00.1450512Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042313.xml (deflated 42%)
2022-05-18T05:52:00.1451319Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042315.xml (deflated 42%)
2022-05-18T05:52:00.1452119Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042319.xml (deflated 42%)
2022-05-18T05:52:00.1452886Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042322.xml (deflated 42%)
2022-05-18T05:52:00.1453668Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042323.xml (deflated 42%)
2022-05-18T05:52:00.1454459Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042325.xml (deflated 40%)
2022-05-18T05:52:00.1455238Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042333.xml (deflated 40%)
2022-05-18T05:52:00.1456148Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042340.xml (deflated 40%)
2022-05-18T05:52:00.1456964Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042347.xml (deflated 39%)
2022-05-18T05:52:00.1457739Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042354.xml (deflated 40%)
2022-05-18T05:52:00.1458521Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042402.xml (deflated 40%)
2022-05-18T05:52:00.1459270Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042409.xml (deflated 40%)
2022-05-18T05:52:00.1460046Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042417.xml (deflated 42%)
2022-05-18T05:52:00.1460932Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042420.xml (deflated 42%)
2022-05-18T05:52:00.1461700Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042424.xml (deflated 42%)
2022-05-18T05:52:00.1462472Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042427.xml (deflated 42%)
2022-05-18T05:52:00.1463223Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042434.xml (deflated 42%)
2022-05-18T05:52:00.1464009Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042436.xml (deflated 46%)
2022-05-18T05:52:00.1464774Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042443.xml (deflated 47%)
2022-05-18T05:52:00.1465552Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042450.xml (deflated 48%)
2022-05-18T05:52:00.1466301Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042458.xml (deflated 46%)
2022-05-18T05:52:00.1467071Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042505.xml (deflated 42%)
2022-05-18T05:52:00.1467834Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042506.xml (deflated 44%)
2022-05-18T05:52:00.1468602Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042508.xml (deflated 44%)
2022-05-18T05:52:00.1469346Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042509.xml (deflated 44%)
2022-05-18T05:52:00.1470114Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042510.xml (deflated 44%)
2022-05-18T05:52:00.1470885Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042511.xml (deflated 44%)
2022-05-18T05:52:00.1471650Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042513.xml (deflated 42%)
2022-05-18T05:52:00.1472414Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042514.xml (deflated 40%)
2022-05-18T05:52:00.1473184Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042522.xml (deflated 40%)
2022-05-18T05:52:00.1473945Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042531.xml (deflated 42%)
2022-05-18T05:52:00.1474713Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042532.xml (deflated 41%)
2022-05-18T05:52:00.1475556Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042533.xml (deflated 40%)
2022-05-18T05:52:00.1476325Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042540.xml (deflated 40%)
2022-05-18T05:52:00.1477090Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042546.xml (deflated 40%)
2022-05-18T05:52:00.1477855Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042553.xml (deflated 40%)
2022-05-18T05:52:00.1478621Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042602.xml (deflated 41%)
2022-05-18T05:52:00.1479442Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042603.xml (deflated 41%)
2022-05-18T05:52:00.1480211Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042604.xml (deflated 41%)
2022-05-18T05:52:00.1480981Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042606.xml (deflated 41%)
2022-05-18T05:52:00.1481745Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042607.xml (deflated 41%)
2022-05-18T05:52:00.1482493Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042608.xml (deflated 41%)
2022-05-18T05:52:00.1483260Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042610.xml (deflated 42%)
2022-05-18T05:52:00.1484032Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042611.xml (deflated 41%)
2022-05-18T05:52:00.1484798Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042612.xml (deflated 41%)
2022-05-18T05:52:00.1485570Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042613.xml (deflated 42%)
2022-05-18T05:52:00.1486341Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042615.xml (deflated 41%)
2022-05-18T05:52:00.1487104Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042616.xml (deflated 41%)
2022-05-18T05:52:00.1487870Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042617.xml (deflated 42%)
2022-05-18T05:52:00.1488615Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042618.xml (deflated 42%)
2022-05-18T05:52:00.1489394Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042620.xml (deflated 42%)
2022-05-18T05:52:00.1490411Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042621.xml (deflated 42%)
2022-05-18T05:52:00.1491175Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042622.xml (deflated 42%)
2022-05-18T05:52:00.1491924Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042624.xml (deflated 42%)
2022-05-18T05:52:00.1492690Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042625.xml (deflated 42%)
2022-05-18T05:52:00.1493451Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042626.xml (deflated 41%)
2022-05-18T05:52:00.1494309Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042627.xml (deflated 42%)
2022-05-18T05:52:00.1495075Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042629.xml (deflated 42%)
2022-05-18T05:52:00.1495846Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042630.xml (deflated 42%)
2022-05-18T05:52:00.1496613Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042631.xml (deflated 41%)
2022-05-18T05:52:00.1497376Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042632.xml (deflated 43%)
2022-05-18T05:52:00.1498137Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042634.xml (deflated 43%)
2022-05-18T05:52:00.1498974Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042635.xml (deflated 42%)
2022-05-18T05:52:00.1499746Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042636.xml (deflated 41%)
2022-05-18T05:52:00.1500517Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042643.xml (deflated 41%)
2022-05-18T05:52:00.1501278Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042645.xml (deflated 41%)
2022-05-18T05:52:00.1502023Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042646.xml (deflated 41%)
2022-05-18T05:52:00.1502788Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042647.xml (deflated 41%)
2022-05-18T05:52:00.1503562Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042654.xml (deflated 41%)
2022-05-18T05:52:00.1504329Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042700.xml (deflated 41%)
2022-05-18T05:52:00.1505072Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042707.xml (deflated 43%)
2022-05-18T05:52:00.1505836Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042708.xml (deflated 42%)
2022-05-18T05:52:00.1506642Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042709.xml (deflated 41%)
2022-05-18T05:52:00.1507403Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042716.xml (deflated 40%)
2022-05-18T05:52:00.1508152Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042723.xml (deflated 42%)
2022-05-18T05:52:00.1508927Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042724.xml (deflated 40%)
2022-05-18T05:52:00.1509694Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042731.xml (deflated 43%)
2022-05-18T05:52:00.1510459Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042733.xml (deflated 42%)
2022-05-18T05:52:00.1511221Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042737.xml (deflated 43%)
2022-05-18T05:52:00.1512016Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042738.xml (deflated 42%)
2022-05-18T05:52:00.1512789Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042740.xml (deflated 41%)
2022-05-18T05:52:00.1513611Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042748.xml (deflated 41%)
2022-05-18T05:52:00.1514385Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042756.xml (deflated 42%)
2022-05-18T05:52:00.1515131Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042758.xml (deflated 41%)
2022-05-18T05:52:00.1515895Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042806.xml (deflated 43%)
2022-05-18T05:52:00.1516655Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042807.xml (deflated 42%)
2022-05-18T05:52:00.1517414Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042812.xml (deflated 43%)
2022-05-18T05:52:00.1518236Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042813.xml (deflated 42%)
2022-05-18T05:52:00.1519002Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042814.xml (deflated 41%)
2022-05-18T05:52:00.1519762Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042822.xml (deflated 41%)
2022-05-18T05:52:00.1520526Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042829.xml (deflated 42%)
2022-05-18T05:52:00.1521275Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042830.xml (deflated 41%)
2022-05-18T05:52:00.1522041Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042837.xml (deflated 42%)
2022-05-18T05:52:00.1522819Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042838.xml (deflated 42%)
2022-05-18T05:52:00.1523583Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042843.xml (deflated 41%)
2022-05-18T05:52:00.1524327Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042850.xml (deflated 42%)
2022-05-18T05:52:00.1525091Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042854.xml (deflated 43%)
2022-05-18T05:52:00.1525853Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042855.xml (deflated 43%)
2022-05-18T05:52:00.1526615Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042857.xml (deflated 41%)
2022-05-18T05:52:00.1527368Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042904.xml (deflated 43%)
2022-05-18T05:52:00.1528138Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042906.xml (deflated 42%)
2022-05-18T05:52:00.1528897Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042910.xml (deflated 43%)
2022-05-18T05:52:00.1529853Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042911.xml (deflated 42%)
2022-05-18T05:52:00.1530612Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042916.xml (deflated 43%)
2022-05-18T05:52:00.1531375Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042917.xml (deflated 43%)
2022-05-18T05:52:00.1532144Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042918.xml (deflated 43%)
2022-05-18T05:52:00.1532995Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042920.xml (deflated 41%)
2022-05-18T05:52:00.1533760Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042921.xml (deflated 41%)
2022-05-18T05:52:00.1534522Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042922.xml (deflated 41%)
2022-05-18T05:52:00.1535293Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042927.xml (deflated 41%)
2022-05-18T05:52:00.1536056Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042933.xml (deflated 42%)
2022-05-18T05:52:00.1536887Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042938.xml (deflated 41%)
2022-05-18T05:52:00.1537659Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042944.xml (deflated 41%)
2022-05-18T05:52:00.1538421Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042948.xml (deflated 40%)
2022-05-18T05:52:00.1539226Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042954.xml (deflated 40%)
2022-05-18T05:52:00.1539987Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043000.xml (deflated 41%)
2022-05-18T05:52:00.1540740Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043005.xml (deflated 42%)
2022-05-18T05:52:00.1541509Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043006.xml (deflated 41%)
2022-05-18T05:52:00.1542282Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043013.xml (deflated 41%)
2022-05-18T05:52:00.1543045Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043014.xml (deflated 42%)
2022-05-18T05:52:00.1543791Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043016.xml (deflated 44%)
2022-05-18T05:52:00.1544556Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043017.xml (deflated 41%)
2022-05-18T05:52:00.1545320Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043024.xml (deflated 40%)
2022-05-18T05:52:00.1546082Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043030.xml (deflated 41%)
2022-05-18T05:52:00.1546834Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043036.xml (deflated 41%)
2022-05-18T05:52:00.1547599Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043043.xml (deflated 40%)
2022-05-18T05:52:00.1548365Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043049.xml (deflated 40%)
2022-05-18T05:52:00.1549124Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043056.xml (deflated 41%)
2022-05-18T05:52:00.1549873Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043059.xml (deflated 41%)
2022-05-18T05:52:00.1550639Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043106.xml (deflated 40%)
2022-05-18T05:52:00.1551463Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043113.xml (deflated 41%)
2022-05-18T05:52:00.1552240Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043120.xml (deflated 40%)
2022-05-18T05:52:00.1552984Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043126.xml (deflated 41%)
2022-05-18T05:52:00.1553745Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043133.xml (deflated 43%)
2022-05-18T05:52:00.1554510Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043134.xml (deflated 41%)
2022-05-18T05:52:00.1555275Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043141.xml (deflated 40%)
2022-05-18T05:52:00.1556108Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043149.xml (deflated 41%)
2022-05-18T05:52:00.1556879Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043156.xml (deflated 41%)
2022-05-18T05:52:00.1557641Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043159.xml (deflated 40%)
2022-05-18T05:52:00.1558407Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043207.xml (deflated 40%)
2022-05-18T05:52:00.1559151Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043215.xml (deflated 40%)
2022-05-18T05:52:00.1559916Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043223.xml (deflated 42%)
2022-05-18T05:52:00.1560691Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043224.xml (deflated 42%)
2022-05-18T05:52:00.1561459Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043226.xml (deflated 42%)
2022-05-18T05:52:00.1562204Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043227.xml (deflated 42%)
2022-05-18T05:52:00.1562970Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043228.xml (deflated 42%)
2022-05-18T05:52:00.1563731Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043229.xml (deflated 42%)
2022-05-18T05:52:00.1564488Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043231.xml (deflated 42%)
2022-05-18T05:52:00.1565239Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043232.xml (deflated 42%)
2022-05-18T05:52:00.1566009Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043233.xml (deflated 42%)
2022-05-18T05:52:00.1566773Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043234.xml (deflated 42%)
2022-05-18T05:52:00.1567534Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043236.xml (deflated 42%)
2022-05-18T05:52:00.1568295Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043237.xml (deflated 42%)
2022-05-18T05:52:00.1569045Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043238.xml (deflated 42%)
2022-05-18T05:52:00.1570023Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043241.xml (deflated 41%)
2022-05-18T05:52:00.1570874Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043248.xml (deflated 40%)
2022-05-18T05:52:00.1571658Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043255.xml (deflated 41%)
2022-05-18T05:52:00.1572403Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043257.xml (deflated 40%)
2022-05-18T05:52:00.1573170Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043304.xml (deflated 40%)
2022-05-18T05:52:00.1573934Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043310.xml (deflated 41%)
2022-05-18T05:52:00.1574700Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043327.xml (deflated 41%)
2022-05-18T05:52:00.1575539Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043335.xml (deflated 41%)
2022-05-18T05:52:00.1576300Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043342.xml (deflated 41%)
2022-05-18T05:52:00.1577066Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043350.xml (deflated 41%)
2022-05-18T05:52:00.1577827Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043357.xml (deflated 41%)
2022-05-18T05:52:00.1578570Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043404.xml (deflated 42%)
2022-05-18T05:52:00.1579340Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043408.xml (deflated 41%)
2022-05-18T05:52:00.1580112Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043414.xml (deflated 40%)
2022-05-18T05:52:00.1580880Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043421.xml (deflated 41%)
2022-05-18T05:52:00.1581627Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043428.xml (deflated 41%)
2022-05-18T05:52:00.1582393Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043434.xml (deflated 42%)
2022-05-18T05:52:00.1583154Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043437.xml (deflated 40%)
2022-05-18T05:52:00.1583923Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043444.xml (deflated 41%)
2022-05-18T05:52:00.1584673Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043451.xml (deflated 40%)
2022-05-18T05:52:00.1585437Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043459.xml (deflated 41%)
2022-05-18T05:52:00.1586202Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043506.xml (deflated 42%)
2022-05-18T05:52:00.1586964Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043509.xml (deflated 41%)
2022-05-18T05:52:00.1587709Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043516.xml (deflated 41%)
2022-05-18T05:52:00.1588531Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043523.xml (deflated 40%)
2022-05-18T05:52:00.1589305Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043528.xml (deflated 41%)
2022-05-18T05:52:00.1590164Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043532.xml (deflated 41%)
2022-05-18T05:52:00.1590944Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043538.xml (deflated 41%)
2022-05-18T05:52:00.1591689Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043545.xml (deflated 40%)
2022-05-18T05:52:00.1592456Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043549.xml (deflated 42%)
2022-05-18T05:52:00.1593223Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043551.xml (deflated 42%)
2022-05-18T05:52:00.1594057Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043552.xml (deflated 40%)
2022-05-18T05:52:00.1594809Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043600.xml (deflated 41%)
2022-05-18T05:52:00.1595577Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043602.xml (deflated 41%)
2022-05-18T05:52:00.1596345Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043603.xml (deflated 41%)
2022-05-18T05:52:00.1597107Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043610.xml (deflated 40%)
2022-05-18T05:52:00.1597855Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043617.xml (deflated 41%)
2022-05-18T05:52:00.1598631Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043622.xml (deflated 40%)
2022-05-18T05:52:00.1599403Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043629.xml (deflated 41%)
2022-05-18T05:52:00.1600171Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043634.xml (deflated 40%)
2022-05-18T05:52:00.1600931Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043638.xml (deflated 40%)
2022-05-18T05:52:00.1601684Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043643.xml (deflated 40%)
2022-05-18T05:52:00.1602447Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043649.xml (deflated 42%)
2022-05-18T05:52:00.1603212Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043651.xml (deflated 43%)
2022-05-18T05:52:00.1603985Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043652.xml (deflated 42%)
2022-05-18T05:52:00.1604732Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043653.xml (deflated 43%)
2022-05-18T05:52:00.1605498Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043655.xml (deflated 40%)
2022-05-18T05:52:00.1606260Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043656.xml (deflated 40%)
2022-05-18T05:52:00.1607025Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043657.xml (deflated 42%)
2022-05-18T05:52:00.1607776Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043658.xml (deflated 41%)
2022-05-18T05:52:00.1608599Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043700.xml (deflated 42%)
2022-05-18T05:52:00.1609375Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043701.xml (deflated 41%)
2022-05-18T05:52:00.1610326Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043702.xml (deflated 41%)
2022-05-18T05:52:00.1611077Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043703.xml (deflated 40%)
2022-05-18T05:52:00.1611878Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043710.xml (deflated 41%)
2022-05-18T05:52:00.1612646Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043716.xml (deflated 41%)
2022-05-18T05:52:00.1613517Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043723.xml (deflated 41%)
2022-05-18T05:52:00.1614267Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043729.xml (deflated 40%)
2022-05-18T05:52:00.1615032Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043736.xml (deflated 42%)
2022-05-18T05:52:00.1615798Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043737.xml (deflated 42%)
2022-05-18T05:52:00.1616565Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043738.xml (deflated 42%)
2022-05-18T05:52:00.1617312Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043739.xml (deflated 40%)
2022-05-18T05:52:00.1618087Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043744.xml (deflated 41%)
2022-05-18T05:52:00.1618854Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043748.xml (deflated 42%)
2022-05-18T05:52:00.1619615Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043750.xml (deflated 43%)
2022-05-18T05:52:00.1620367Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043751.xml (deflated 41%)
2022-05-18T05:52:00.1621131Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043758.xml (deflated 40%)
2022-05-18T05:52:00.1621900Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043806.xml (deflated 41%)
2022-05-18T05:52:00.1622670Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043813.xml (deflated 40%)
2022-05-18T05:52:00.1623418Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043820.xml (deflated 41%)
2022-05-18T05:52:00.1624187Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043823.xml (deflated 41%)
2022-05-18T05:52:00.1624954Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043826.xml (deflated 40%)
2022-05-18T05:52:00.1625717Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043830.xml (deflated 40%)
2022-05-18T05:52:00.1626460Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043835.xml (deflated 41%)
2022-05-18T05:52:00.1627228Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043836.xml (deflated 41%)
2022-05-18T05:52:00.1628067Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043837.xml (deflated 42%)
2022-05-18T05:52:00.1628849Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043839.xml (deflated 41%)
2022-05-18T05:52:00.1629593Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043840.xml (deflated 42%)
2022-05-18T05:52:00.1630364Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043841.xml (deflated 42%)
2022-05-18T05:52:00.1631130Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043843.xml (deflated 42%)
2022-05-18T05:52:00.1631889Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043844.xml (deflated 42%)
2022-05-18T05:52:00.1632704Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043845.xml (deflated 42%)
2022-05-18T05:52:00.1633470Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043846.xml (deflated 42%)
2022-05-18T05:52:00.1634234Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043848.xml (deflated 41%)
2022-05-18T05:52:00.1634996Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043854.xml (deflated 42%)
2022-05-18T05:52:00.1635738Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043856.xml (deflated 42%)
2022-05-18T05:52:00.1636499Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043857.xml (deflated 40%)
2022-05-18T05:52:00.1637271Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043903.xml (deflated 41%)
2022-05-18T05:52:00.1638034Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043910.xml (deflated 41%)
2022-05-18T05:52:00.1638795Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043911.xml (deflated 42%)
2022-05-18T05:52:00.1639549Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043913.xml (deflated 41%)
2022-05-18T05:52:00.1640311Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043914.xml (deflated 41%)
2022-05-18T05:52:00.1641071Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043915.xml (deflated 41%)
2022-05-18T05:52:00.1641836Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043924.xml (deflated 41%)
2022-05-18T05:52:00.1642582Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043932.xml (deflated 41%)
2022-05-18T05:52:00.1643358Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043933.xml (deflated 42%)
2022-05-18T05:52:00.1644128Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043934.xml (deflated 41%)
2022-05-18T05:52:00.1644899Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043936.xml (deflated 44%)
2022-05-18T05:52:00.1645651Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043937.xml (deflated 43%)
2022-05-18T05:52:00.1646422Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043938.xml (deflated 42%)
2022-05-18T05:52:00.1647249Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043940.xml (deflated 43%)
2022-05-18T05:52:00.1648026Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043941.xml (deflated 43%)
2022-05-18T05:52:00.1648779Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043942.xml (deflated 40%)
2022-05-18T05:52:00.1649719Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043948.xml (deflated 41%)
2022-05-18T05:52:00.1650509Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043955.xml (deflated 41%)
2022-05-18T05:52:00.1651373Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044001.xml (deflated 44%)
2022-05-18T05:52:00.1652137Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044002.xml (deflated 44%)
2022-05-18T05:52:00.1652904Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044004.xml (deflated 43%)
2022-05-18T05:52:00.1653669Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044005.xml (deflated 43%)
2022-05-18T05:52:00.1654432Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044006.xml (deflated 43%)
2022-05-18T05:52:00.1655185Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044008.xml (deflated 43%)
2022-05-18T05:52:00.1655950Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044009.xml (deflated 41%)
2022-05-18T05:52:00.1656721Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044016.xml (deflated 41%)
2022-05-18T05:52:00.1657491Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044017.xml (deflated 40%)
2022-05-18T05:52:00.1658236Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044024.xml (deflated 41%)
2022-05-18T05:52:00.1659005Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044032.xml (deflated 41%)
2022-05-18T05:52:00.1659771Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044043.xml (deflated 40%)
2022-05-18T05:52:00.1660529Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044056.xml (deflated 42%)
2022-05-18T05:52:00.1661281Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044103.xml (deflated 42%)
2022-05-18T05:52:00.1662047Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044104.xml (deflated 42%)
2022-05-18T05:52:00.1662807Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044109.xml (deflated 42%)
2022-05-18T05:52:00.1663569Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044112.xml (deflated 42%)
2022-05-18T05:52:00.1664415Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044113.xml (deflated 42%)
2022-05-18T05:52:00.1665178Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044114.xml (deflated 41%)
2022-05-18T05:52:00.1665997Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044123.xml (deflated 40%)
2022-05-18T05:52:00.1666789Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044130.xml (deflated 41%)
2022-05-18T05:52:00.1667555Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044137.xml (deflated 39%)
2022-05-18T05:52:00.1668318Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044144.xml (deflated 40%)
2022-05-18T05:52:00.1669064Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044151.xml (deflated 40%)
2022-05-18T05:52:00.1669831Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044159.xml (deflated 40%)
2022-05-18T05:52:00.1670660Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044206.xml (deflated 42%)
2022-05-18T05:52:00.1671429Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044209.xml (deflated 41%)
2022-05-18T05:52:00.1672195Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044213.xml (deflated 42%)
2022-05-18T05:52:00.1672940Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044217.xml (deflated 42%)
2022-05-18T05:52:00.1673707Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044224.xml (deflated 42%)
2022-05-18T05:52:00.1674473Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044225.xml (deflated 46%)
2022-05-18T05:52:00.1675239Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044232.xml (deflated 47%)
2022-05-18T05:52:00.1675985Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044240.xml (deflated 48%)
2022-05-18T05:52:00.1676750Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044247.xml (deflated 46%)
2022-05-18T05:52:00.1677511Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044254.xml (deflated 42%)
2022-05-18T05:52:00.1678274Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044256.xml (deflated 44%)
2022-05-18T05:52:00.1679020Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044257.xml (deflated 44%)
2022-05-18T05:52:00.1679787Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044258.xml (deflated 44%)
2022-05-18T05:52:00.1680555Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044300.xml (deflated 44%)
2022-05-18T05:52:00.1681317Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044301.xml (deflated 44%)
2022-05-18T05:52:00.1682062Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044302.xml (deflated 42%)
2022-05-18T05:52:00.1682827Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044303.xml (deflated 41%)
2022-05-18T05:52:00.1683593Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044312.xml (deflated 40%)
2022-05-18T05:52:00.1684361Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044320.xml (deflated 42%)
2022-05-18T05:52:00.1685166Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044321.xml (deflated 41%)
2022-05-18T05:52:00.1685945Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044323.xml (deflated 40%)
2022-05-18T05:52:00.1686704Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044329.xml (deflated 40%)
2022-05-18T05:52:00.1687466Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044336.xml (deflated 40%)
2022-05-18T05:52:00.1688213Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044343.xml (deflated 40%)
2022-05-18T05:52:00.1688976Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044351.xml (deflated 41%)
2022-05-18T05:52:00.1690013Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044352.xml (deflated 41%)
2022-05-18T05:52:00.1690780Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044353.xml (deflated 41%)
2022-05-18T05:52:00.1691528Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044355.xml (deflated 41%)
2022-05-18T05:52:00.1692292Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044356.xml (deflated 41%)
2022-05-18T05:52:00.1693052Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044357.xml (deflated 41%)
2022-05-18T05:52:00.1693812Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044359.xml (deflated 41%)
2022-05-18T05:52:00.1694563Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044400.xml (deflated 41%)
2022-05-18T05:52:00.1695328Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044401.xml (deflated 41%)
2022-05-18T05:52:00.1696088Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044402.xml (deflated 42%)
2022-05-18T05:52:00.1696851Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044404.xml (deflated 42%)
2022-05-18T05:52:00.1697595Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044405.xml (deflated 42%)
2022-05-18T05:52:00.1698360Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044406.xml (deflated 42%)
2022-05-18T05:52:00.1699127Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044407.xml (deflated 42%)
2022-05-18T05:52:00.1699901Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044409.xml (deflated 41%)
2022-05-18T05:52:00.1700645Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044410.xml (deflated 42%)
2022-05-18T05:52:00.1701409Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044411.xml (deflated 42%)
2022-05-18T05:52:00.1702173Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044413.xml (deflated 42%)
2022-05-18T05:52:00.1702935Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044414.xml (deflated 42%)
2022-05-18T05:52:00.1703683Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044415.xml (deflated 42%)
2022-05-18T05:52:00.1704524Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044416.xml (deflated 42%)
2022-05-18T05:52:00.1705301Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044418.xml (deflated 42%)
2022-05-18T05:52:00.1706064Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044419.xml (deflated 42%)
2022-05-18T05:52:00.1706806Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044420.xml (deflated 42%)
2022-05-18T05:52:00.1707575Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044421.xml (deflated 43%)
2022-05-18T05:52:00.1708422Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044423.xml (deflated 43%)
2022-05-18T05:52:00.1709193Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044424.xml (deflated 42%)
2022-05-18T05:52:00.1709941Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044425.xml (deflated 41%)
2022-05-18T05:52:00.1710702Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044432.xml (deflated 42%)
2022-05-18T05:52:00.1711470Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044434.xml (deflated 42%)
2022-05-18T05:52:00.1712273Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044435.xml (deflated 41%)
2022-05-18T05:52:00.1713036Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044436.xml (deflated 41%)
2022-05-18T05:52:00.1713791Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044443.xml (deflated 41%)
2022-05-18T05:52:00.1714556Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044449.xml (deflated 41%)
2022-05-18T05:52:00.1715317Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044456.xml (deflated 43%)
2022-05-18T05:52:00.1716081Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044457.xml (deflated 42%)
2022-05-18T05:52:00.1716837Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044458.xml (deflated 41%)
2022-05-18T05:52:00.1717597Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044505.xml (deflated 41%)
2022-05-18T05:52:00.1718362Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044512.xml (deflated 43%)
2022-05-18T05:52:00.1719132Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044514.xml (deflated 40%)
2022-05-18T05:52:00.1719876Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044521.xml (deflated 43%)
2022-05-18T05:52:00.1720636Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044522.xml (deflated 42%)
2022-05-18T05:52:00.1721400Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044526.xml (deflated 43%)
2022-05-18T05:52:00.1722162Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044528.xml (deflated 42%)
2022-05-18T05:52:00.1722970Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044529.xml (deflated 41%)
2022-05-18T05:52:00.1723751Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044537.xml (deflated 41%)
2022-05-18T05:52:00.1724512Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044546.xml (deflated 42%)
2022-05-18T05:52:00.1725273Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044547.xml (deflated 41%)
2022-05-18T05:52:00.1726019Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044555.xml (deflated 43%)
2022-05-18T05:52:00.1726775Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044557.xml (deflated 42%)
2022-05-18T05:52:00.1727603Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044601.xml (deflated 42%)
2022-05-18T05:52:00.1728370Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044603.xml (deflated 42%)
2022-05-18T05:52:00.1729116Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044604.xml (deflated 41%)
2022-05-18T05:52:00.1730041Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044611.xml (deflated 41%)
2022-05-18T05:52:00.1730806Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044618.xml (deflated 42%)
2022-05-18T05:52:00.1731567Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044619.xml (deflated 40%)
2022-05-18T05:52:00.1732318Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044626.xml (deflated 42%)
2022-05-18T05:52:00.1733090Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044627.xml (deflated 42%)
2022-05-18T05:52:00.1733856Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044632.xml (deflated 41%)
2022-05-18T05:52:00.1734626Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044639.xml (deflated 42%)
2022-05-18T05:52:00.1735371Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044644.xml (deflated 43%)
2022-05-18T05:52:00.1736132Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044645.xml (deflated 43%)
2022-05-18T05:52:00.1736902Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044646.xml (deflated 41%)
2022-05-18T05:52:00.1737665Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044654.xml (deflated 42%)
2022-05-18T05:52:00.1738409Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044656.xml (deflated 42%)
2022-05-18T05:52:00.1739177Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044700.xml (deflated 43%)
2022-05-18T05:52:00.1739940Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044701.xml (deflated 42%)
2022-05-18T05:52:00.1740702Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044706.xml (deflated 42%)
2022-05-18T05:52:00.1741447Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044707.xml (deflated 43%)
2022-05-18T05:52:00.1742302Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044708.xml (deflated 43%)
2022-05-18T05:52:00.1743082Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044710.xml (deflated 41%)
2022-05-18T05:52:00.1743850Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044711.xml (deflated 41%)
2022-05-18T05:52:00.1744596Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044712.xml (deflated 41%)
2022-05-18T05:52:00.1745358Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044717.xml (deflated 41%)
2022-05-18T05:52:00.1746125Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044723.xml (deflated 42%)
2022-05-18T05:52:00.1746975Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044727.xml (deflated 41%)
2022-05-18T05:52:00.1747722Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044733.xml (deflated 41%)
2022-05-18T05:52:00.1748539Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044738.xml (deflated 40%)
2022-05-18T05:52:00.1749304Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044744.xml (deflated 40%)
2022-05-18T05:52:00.1750071Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044750.xml (deflated 41%)
2022-05-18T05:52:00.1750814Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044755.xml (deflated 42%)
2022-05-18T05:52:00.1751587Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044756.xml (deflated 41%)
2022-05-18T05:52:00.1752350Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044803.xml (deflated 41%)
2022-05-18T05:52:00.1753115Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044804.xml (deflated 42%)
2022-05-18T05:52:00.1753876Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044805.xml (deflated 44%)
2022-05-18T05:52:00.1754623Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044807.xml (deflated 41%)
2022-05-18T05:52:00.1755385Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044814.xml (deflated 41%)
2022-05-18T05:52:00.1756151Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044820.xml (deflated 41%)
2022-05-18T05:52:00.1756919Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044826.xml (deflated 40%)
2022-05-18T05:52:00.1757671Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044832.xml (deflated 40%)
2022-05-18T05:52:00.1758431Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044839.xml (deflated 41%)
2022-05-18T05:52:00.1759192Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044845.xml (deflated 42%)
2022-05-18T05:52:00.1759954Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044848.xml (deflated 41%)
2022-05-18T05:52:00.1760709Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044855.xml (deflated 41%)
2022-05-18T05:52:00.1761547Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044902.xml (deflated 40%)
2022-05-18T05:52:00.1762323Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044910.xml (deflated 41%)
2022-05-18T05:52:00.1763086Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044916.xml (deflated 41%)
2022-05-18T05:52:00.1763828Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044922.xml (deflated 43%)
2022-05-18T05:52:00.1764594Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044924.xml (deflated 41%)
2022-05-18T05:52:00.1765419Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044931.xml (deflated 41%)
2022-05-18T05:52:00.1766185Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044938.xml (deflated 41%)
2022-05-18T05:52:00.1766935Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044946.xml (deflated 41%)
2022-05-18T05:52:00.1767700Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044949.xml (deflated 40%)
2022-05-18T05:52:00.1768461Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044957.xml (deflated 40%)
2022-05-18T05:52:00.1769221Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045004.xml (deflated 40%)
2022-05-18T05:52:00.1770147Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045012.xml (deflated 42%)
2022-05-18T05:52:00.1770924Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045014.xml (deflated 42%)
2022-05-18T05:52:00.1771686Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045015.xml (deflated 42%)
2022-05-18T05:52:00.1772457Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045016.xml (deflated 42%)
2022-05-18T05:52:00.1773200Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045018.xml (deflated 42%)
2022-05-18T05:52:00.1773964Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045019.xml (deflated 42%)
2022-05-18T05:52:00.1774729Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045020.xml (deflated 42%)
2022-05-18T05:52:00.1775499Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045021.xml (deflated 42%)
2022-05-18T05:52:00.1776244Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045023.xml (deflated 42%)
2022-05-18T05:52:00.1777009Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045024.xml (deflated 42%)
2022-05-18T05:52:00.1777766Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045025.xml (deflated 42%)
2022-05-18T05:52:00.1778521Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045026.xml (deflated 42%)
2022-05-18T05:52:00.1779272Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045028.xml (deflated 42%)
2022-05-18T05:52:00.1780112Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045031.xml (deflated 41%)
2022-05-18T05:52:00.1780893Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045037.xml (deflated 40%)
2022-05-18T05:52:00.1781756Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045045.xml (deflated 41%)
2022-05-18T05:52:00.1782521Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045046.xml (deflated 41%)
2022-05-18T05:52:00.1783282Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045053.xml (deflated 41%)
2022-05-18T05:52:00.1784024Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045100.xml (deflated 41%)
2022-05-18T05:52:00.1784882Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045116.xml (deflated 41%)
2022-05-18T05:52:00.1785657Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045124.xml (deflated 41%)
2022-05-18T05:52:00.1786419Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045132.xml (deflated 41%)
2022-05-18T05:52:00.1787180Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045139.xml (deflated 41%)
2022-05-18T05:52:00.1787924Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045146.xml (deflated 41%)
2022-05-18T05:52:00.1788685Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045154.xml (deflated 42%)
2022-05-18T05:52:00.1789448Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045157.xml (deflated 41%)
2022-05-18T05:52:00.1790214Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045204.xml (deflated 41%)
2022-05-18T05:52:00.1790969Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045211.xml (deflated 41%)
2022-05-18T05:52:00.1791730Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045217.xml (deflated 41%)
2022-05-18T05:52:00.1792490Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045224.xml (deflated 42%)
2022-05-18T05:52:00.1793247Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045227.xml (deflated 40%)
2022-05-18T05:52:00.1794004Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045234.xml (deflated 41%)
2022-05-18T05:52:00.1794767Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045241.xml (deflated 41%)
2022-05-18T05:52:00.1795534Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045248.xml (deflated 41%)
2022-05-18T05:52:00.1796296Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045256.xml (deflated 42%)
2022-05-18T05:52:00.1797042Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045259.xml (deflated 40%)
2022-05-18T05:52:00.1797805Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045305.xml (deflated 41%)
2022-05-18T05:52:00.1798564Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045312.xml (deflated 41%)
2022-05-18T05:52:00.1799397Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045317.xml (deflated 41%)
2022-05-18T05:52:00.1800161Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045322.xml (deflated 41%)
2022-05-18T05:52:00.1800928Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045328.xml (deflated 41%)
2022-05-18T05:52:00.1801683Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045334.xml (deflated 40%)
2022-05-18T05:52:00.1802441Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045339.xml (deflated 42%)
2022-05-18T05:52:00.1803190Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045340.xml (deflated 42%)
2022-05-18T05:52:00.1804015Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045342.xml (deflated 40%)
2022-05-18T05:52:00.1804779Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045350.xml (deflated 41%)
2022-05-18T05:52:00.1805538Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045351.xml (deflated 41%)
2022-05-18T05:52:00.1806282Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045353.xml (deflated 40%)
2022-05-18T05:52:00.1807043Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045400.xml (deflated 41%)
2022-05-18T05:52:00.1807801Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045407.xml (deflated 41%)
2022-05-18T05:52:00.1808565Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045411.xml (deflated 41%)
2022-05-18T05:52:00.1809318Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045418.xml (deflated 41%)
2022-05-18T05:52:00.1810265Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045423.xml (deflated 40%)
2022-05-18T05:52:00.1811027Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045428.xml (deflated 40%)
2022-05-18T05:52:00.1811787Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045432.xml (deflated 40%)
2022-05-18T05:52:00.1812564Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045439.xml (deflated 43%)
2022-05-18T05:52:00.1813335Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045440.xml (deflated 43%)
2022-05-18T05:52:00.1814099Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045441.xml (deflated 42%)
2022-05-18T05:52:00.1814866Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045442.xml (deflated 43%)
2022-05-18T05:52:00.1815611Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045444.xml (deflated 40%)
2022-05-18T05:52:00.1816370Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045445.xml (deflated 40%)
2022-05-18T05:52:00.1817184Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045446.xml (deflated 42%)
2022-05-18T05:52:00.1817950Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045447.xml (deflated 42%)
2022-05-18T05:52:00.1818769Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045449.xml (deflated 42%)
2022-05-18T05:52:00.1819551Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045450.xml (deflated 42%)
2022-05-18T05:52:00.1820309Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045451.xml (deflated 42%)
2022-05-18T05:52:00.1821077Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045453.xml (deflated 40%)
2022-05-18T05:52:00.1821818Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045459.xml (deflated 41%)
2022-05-18T05:52:00.1822678Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045505.xml (deflated 41%)
2022-05-18T05:52:00.1823452Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045512.xml (deflated 41%)
2022-05-18T05:52:00.1824218Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045518.xml (deflated 41%)
2022-05-18T05:52:00.1824982Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045525.xml (deflated 42%)
2022-05-18T05:52:00.1825728Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045527.xml (deflated 42%)
2022-05-18T05:52:00.1826495Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045528.xml (deflated 42%)
2022-05-18T05:52:00.1827257Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045529.xml (deflated 40%)
2022-05-18T05:52:00.1828028Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045534.xml (deflated 41%)
2022-05-18T05:52:00.1828784Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045538.xml (deflated 42%)
2022-05-18T05:52:00.1829546Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045540.xml (deflated 43%)
2022-05-18T05:52:00.1830312Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045541.xml (deflated 41%)
2022-05-18T05:52:00.1831081Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045548.xml (deflated 41%)
2022-05-18T05:52:00.1831832Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045556.xml (deflated 40%)
2022-05-18T05:52:00.1832608Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045603.xml (deflated 41%)
2022-05-18T05:52:00.1833380Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045610.xml (deflated 41%)
2022-05-18T05:52:00.1834143Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045613.xml (deflated 41%)
2022-05-18T05:52:00.1834888Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045616.xml (deflated 40%)
2022-05-18T05:52:00.1835651Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045621.xml (deflated 40%)
2022-05-18T05:52:00.1836413Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045625.xml (deflated 41%)
2022-05-18T05:52:00.1837177Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045627.xml (deflated 41%)
2022-05-18T05:52:00.1837982Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045628.xml (deflated 42%)
2022-05-18T05:52:00.1838757Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045629.xml (deflated 41%)
2022-05-18T05:52:00.1839520Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045631.xml (deflated 41%)
2022-05-18T05:52:00.1840280Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045632.xml (deflated 42%)
2022-05-18T05:52:00.1841027Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045633.xml (deflated 42%)
2022-05-18T05:52:00.1841854Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045634.xml (deflated 42%)
2022-05-18T05:52:00.1842626Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045636.xml (deflated 42%)
2022-05-18T05:52:00.1843390Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045637.xml (deflated 42%)
2022-05-18T05:52:00.1844136Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045638.xml (deflated 41%)
2022-05-18T05:52:00.1844904Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045645.xml (deflated 42%)
2022-05-18T05:52:00.1845670Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045646.xml (deflated 42%)
2022-05-18T05:52:00.1846437Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045647.xml (deflated 40%)
2022-05-18T05:52:00.1847188Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045654.xml (deflated 40%)
2022-05-18T05:52:00.1847952Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045700.xml (deflated 42%)
2022-05-18T05:52:00.1848716Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045702.xml (deflated 42%)
2022-05-18T05:52:00.1849475Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045703.xml (deflated 42%)
2022-05-18T05:52:00.1850390Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045704.xml (deflated 42%)
2022-05-18T05:52:00.1851159Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045706.xml (deflated 41%)
2022-05-18T05:52:00.1851920Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045714.xml (deflated 41%)
2022-05-18T05:52:00.1852687Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045722.xml (deflated 42%)
2022-05-18T05:52:00.1853430Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045723.xml (deflated 42%)
2022-05-18T05:52:00.1854194Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045725.xml (deflated 41%)
2022-05-18T05:52:00.1854955Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045726.xml (deflated 44%)
2022-05-18T05:52:00.1855716Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045727.xml (deflated 43%)
2022-05-18T05:52:00.1856542Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045729.xml (deflated 42%)
2022-05-18T05:52:00.1857329Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045730.xml (deflated 43%)
2022-05-18T05:52:00.1858097Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045731.xml (deflated 43%)
2022-05-18T05:52:00.1858859Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045732.xml (deflated 41%)
2022-05-18T05:52:00.1859607Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045738.xml (deflated 40%)
2022-05-18T05:52:00.1860374Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045745.xml (deflated 41%)
2022-05-18T05:52:00.1861225Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045751.xml (deflated 44%)
2022-05-18T05:52:00.1861991Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045753.xml (deflated 43%)
2022-05-18T05:52:00.1862752Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045754.xml (deflated 43%)
2022-05-18T05:52:00.1863500Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045755.xml (deflated 43%)
2022-05-18T05:52:00.1864267Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045757.xml (deflated 43%)
2022-05-18T05:52:00.1865028Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045758.xml (deflated 43%)
2022-05-18T05:52:00.1865792Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045759.xml (deflated 41%)
2022-05-18T05:52:00.1866539Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045807.xml (deflated 41%)
2022-05-18T05:52:00.1867308Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045808.xml (deflated 41%)
2022-05-18T05:52:00.1868063Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045815.xml (deflated 41%)
2022-05-18T05:52:00.1868822Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045822.xml (deflated 41%)
2022-05-18T05:52:00.1869567Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045834.xml (deflated 40%)
2022-05-18T05:52:00.1870332Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045846.xml (deflated 42%)
2022-05-18T05:52:00.1871097Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045853.xml (deflated 42%)
2022-05-18T05:52:00.1871866Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045855.xml (deflated 42%)
2022-05-18T05:52:00.1872608Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045859.xml (deflated 42%)
2022-05-18T05:52:00.1873366Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045902.xml (deflated 41%)
2022-05-18T05:52:00.1874133Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045907.xml (deflated 41%)
2022-05-18T05:52:00.1874897Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045911.xml (deflated 40%)
2022-05-18T05:52:00.1875693Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045918.xml (deflated 40%)
2022-05-18T05:52:00.1876470Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045924.xml (deflated 40%)
2022-05-18T05:52:00.1877234Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045930.xml (deflated 39%)
2022-05-18T05:52:00.1877998Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045936.xml (deflated 40%)
2022-05-18T05:52:00.1878746Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045943.xml (deflated 40%)
2022-05-18T05:52:00.1879618Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045949.xml (deflated 40%)
2022-05-18T05:52:00.1880381Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045955.xml (deflated 42%)
2022-05-18T05:52:00.1881149Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045958.xml (deflated 42%)
2022-05-18T05:52:00.1881894Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050003.xml (deflated 42%)
2022-05-18T05:52:00.1882655Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050006.xml (deflated 42%)
2022-05-18T05:52:00.1883421Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050012.xml (deflated 42%)
2022-05-18T05:52:00.1884181Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050014.xml (deflated 46%)
2022-05-18T05:52:00.1884938Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050018.xml (deflated 47%)
2022-05-18T05:52:00.1885708Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050023.xml (deflated 48%)
2022-05-18T05:52:00.1886474Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050027.xml (deflated 46%)
2022-05-18T05:52:00.1887239Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050032.xml (deflated 41%)
2022-05-18T05:52:00.1887983Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050036.xml (deflated 41%)
2022-05-18T05:52:00.1888749Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050041.xml (deflated 41%)
2022-05-18T05:52:00.1889678Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050046.xml (deflated 42%)
2022-05-18T05:52:00.1890468Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050050.xml (deflated 41%)
2022-05-18T05:52:00.1891213Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050055.xml (deflated 41%)
2022-05-18T05:52:00.1891974Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050059.xml (deflated 41%)
2022-05-18T05:52:00.1892740Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050104.xml (deflated 42%)
2022-05-18T05:52:00.1893496Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050105.xml (deflated 42%)
2022-05-18T05:52:00.1894247Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050107.xml (deflated 41%)
2022-05-18T05:52:00.1895083Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050111.xml (deflated 42%)
2022-05-18T05:52:00.1895861Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050116.xml (deflated 43%)
2022-05-18T05:52:00.1896628Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050117.xml (deflated 43%)
2022-05-18T05:52:00.1897371Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050118.xml (deflated 40%)
2022-05-18T05:52:00.1898137Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050123.xml (deflated 40%)
2022-05-18T05:52:00.1898982Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050128.xml (deflated 41%)
2022-05-18T05:52:00.1899753Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050132.xml (deflated 41%)
2022-05-18T05:52:00.1900502Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050137.xml (deflated 41%)
2022-05-18T05:52:00.1901262Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050141.xml (deflated 41%)
2022-05-18T05:52:00.1902028Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050146.xml (deflated 42%)
2022-05-18T05:52:00.1902793Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050150.xml (deflated 43%)
2022-05-18T05:52:00.1903544Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050155.xml (deflated 42%)
2022-05-18T05:52:00.1904314Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050200.xml (deflated 42%)
2022-05-18T05:52:00.1905078Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050204.xml (deflated 41%)
2022-05-18T05:52:00.1905837Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050209.xml (deflated 41%)
2022-05-18T05:52:00.1906602Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050213.xml (deflated 41%)
2022-05-18T05:52:00.1907353Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050218.xml (deflated 41%)
2022-05-18T05:52:00.1908114Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050222.xml (deflated 41%)
2022-05-18T05:52:00.1908880Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050227.xml (deflated 41%)
2022-05-18T05:52:00.1909644Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050231.xml (deflated 41%)
2022-05-18T05:52:00.1910393Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050236.xml (deflated 41%)
2022-05-18T05:52:00.1911151Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050241.xml (deflated 41%)
2022-05-18T05:52:00.1911952Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050245.xml (deflated 41%)
2022-05-18T05:52:00.1912717Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050250.xml (deflated 42%)
2022-05-18T05:52:00.1913522Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050254.xml (deflated 42%)
2022-05-18T05:52:00.1914303Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050259.xml (deflated 42%)
2022-05-18T05:52:00.1915067Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050303.xml (deflated 42%)
2022-05-18T05:52:00.1915829Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050308.xml (deflated 41%)
2022-05-18T05:52:00.1916572Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050313.xml (deflated 41%)
2022-05-18T05:52:00.1917337Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050317.xml (deflated 41%)
2022-05-18T05:52:00.1918168Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050323.xml (deflated 40%)
2022-05-18T05:52:00.1918930Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050330.xml (deflated 41%)
2022-05-18T05:52:00.1919681Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050334.xml (deflated 41%)
2022-05-18T05:52:00.1920450Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050340.xml (deflated 41%)
2022-05-18T05:52:00.1921210Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050345.xml (deflated 40%)
2022-05-18T05:52:00.1921975Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050350.xml (deflated 41%)
2022-05-18T05:52:00.1922724Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050354.xml (deflated 41%)
2022-05-18T05:52:00.1923490Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050403.xml (deflated 41%)
2022-05-18T05:52:00.1924257Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050411.xml (deflated 40%)
2022-05-18T05:52:00.1925014Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050419.xml (deflated 43%)
2022-05-18T05:52:00.1925759Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050421.xml (deflated 42%)
2022-05-18T05:52:00.1926523Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050422.xml (deflated 42%)
2022-05-18T05:52:00.1927291Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050423.xml (deflated 42%)
2022-05-18T05:52:00.1928051Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050424.xml (deflated 43%)
2022-05-18T05:52:00.1928800Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050426.xml (deflated 42%)
2022-05-18T05:52:00.1929715Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050427.xml (deflated 43%)
2022-05-18T05:52:00.1930490Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050428.xml (deflated 42%)
2022-05-18T05:52:00.1931250Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050430.xml (deflated 43%)
2022-05-18T05:52:00.1932001Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050431.xml (deflated 42%)
2022-05-18T05:52:00.1932838Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050432.xml (deflated 43%)
2022-05-18T05:52:00.1933619Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050433.xml (deflated 42%)
2022-05-18T05:52:00.1934387Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050435.xml (deflated 43%)
2022-05-18T05:52:00.1935132Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050436.xml (deflated 43%)
2022-05-18T05:52:00.1935892Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050437.xml (deflated 42%)
2022-05-18T05:52:00.1936738Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050438.xml (deflated 43%)
2022-05-18T05:52:00.1937503Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050440.xml (deflated 43%)
2022-05-18T05:52:00.1938248Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050441.xml (deflated 42%)
2022-05-18T05:52:00.1939019Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050442.xml (deflated 43%)
2022-05-18T05:52:00.1939779Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050444.xml (deflated 42%)
2022-05-18T05:52:00.1940543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050445.xml (deflated 42%)
2022-05-18T05:52:00.1941302Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050446.xml (deflated 43%)
2022-05-18T05:52:00.1942057Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050447.xml (deflated 42%)
2022-05-18T05:52:00.1942819Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050449.xml (deflated 43%)
2022-05-18T05:52:00.1943585Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050450.xml (deflated 41%)
2022-05-18T05:52:00.1944350Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050457.xml (deflated 42%)
2022-05-18T05:52:00.1945093Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050502.xml (deflated 43%)
2022-05-18T05:52:00.1945855Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050503.xml (deflated 41%)
2022-05-18T05:52:00.1946627Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050508.xml (deflated 41%)
2022-05-18T05:52:00.1947389Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050515.xml (deflated 41%)
2022-05-18T05:52:00.1948130Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050521.xml (deflated 42%)
2022-05-18T05:52:00.1948892Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050525.xml (deflated 42%)
2022-05-18T05:52:00.1949650Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050530.xml (deflated 42%)
2022-05-18T05:52:00.1950411Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050534.xml (deflated 40%)
2022-05-18T05:52:00.1951165Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050540.xml (deflated 42%)
2022-05-18T05:52:00.1951975Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050541.xml (deflated 42%)
2022-05-18T05:52:00.1952747Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050546.xml (deflated 40%)
2022-05-18T05:52:00.1953517Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050551.xml (deflated 40%)
2022-05-18T05:52:00.1954258Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050555.xml (deflated 42%)
2022-05-18T05:52:00.1955021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050556.xml (deflated 42%)
2022-05-18T05:52:00.1955863Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050558.xml (deflated 41%)
2022-05-18T05:52:00.1956628Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050559.xml (deflated 41%)
2022-05-18T05:52:00.1957367Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050600.xml (deflated 42%)
2022-05-18T05:52:00.1958132Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050602.xml (deflated 41%)
2022-05-18T05:52:00.1958893Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050603.xml (deflated 42%)
2022-05-18T05:52:00.1959654Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050604.xml (deflated 42%)
2022-05-18T05:52:00.1960401Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050605.xml (deflated 41%)
2022-05-18T05:52:00.1961165Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050610.xml (deflated 41%)
2022-05-18T05:52:00.1962098Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050616.xml (deflated 40%)
2022-05-18T05:52:00.1962858Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050621.xml (deflated 42%)
2022-05-18T05:52:00.1963599Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050626.xml (deflated 40%)
2022-05-18T05:52:00.1964363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050632.xml (deflated 41%)
2022-05-18T05:52:00.1965128Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050636.xml (deflated 41%)
2022-05-18T05:52:00.1965893Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050642.xml (deflated 41%)
2022-05-18T05:52:00.1966634Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050649.xml (deflated 41%)
2022-05-18T05:52:00.1967403Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050655.xml (deflated 41%)
2022-05-18T05:52:00.1968165Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050701.xml (deflated 40%)
2022-05-18T05:52:00.1968925Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050708.xml (deflated 42%)
2022-05-18T05:52:00.1969851Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050711.xml (deflated 41%)
2022-05-18T05:52:00.1970700Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050717.xml (deflated 41%)
2022-05-18T05:52:00.1971484Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050723.xml (deflated 41%)
2022-05-18T05:52:00.1972248Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050729.xml (deflated 40%)
2022-05-18T05:52:00.1973009Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050735.xml (deflated 41%)
2022-05-18T05:52:00.1973751Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050742.xml (deflated 41%)
2022-05-18T05:52:00.1974516Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050746.xml (deflated 41%)
2022-05-18T05:52:00.1975369Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050753.xml (deflated 41%)
2022-05-18T05:52:00.1976129Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050759.xml (deflated 41%)
2022-05-18T05:52:00.1976879Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050805.xml (deflated 42%)
2022-05-18T05:52:00.1977646Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050808.xml (deflated 41%)
2022-05-18T05:52:00.1978409Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050815.xml (deflated 41%)
2022-05-18T05:52:00.1979169Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050822.xml (deflated 40%)
2022-05-18T05:52:00.1979923Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050829.xml (deflated 41%)
2022-05-18T05:52:00.1980691Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050836.xml (deflated 41%)
2022-05-18T05:52:00.1981460Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050842.xml (deflated 41%)
2022-05-18T05:52:00.1982225Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050849.xml (deflated 41%)
2022-05-18T05:52:00.1982968Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050855.xml (deflated 41%)
2022-05-18T05:52:00.1983728Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050902.xml (deflated 41%)
2022-05-18T05:52:00.1984499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050908.xml (deflated 41%)
2022-05-18T05:52:00.1985262Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050915.xml (deflated 41%)
2022-05-18T05:52:00.1986009Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050921.xml (deflated 41%)
2022-05-18T05:52:00.1986775Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050928.xml (deflated 41%)
2022-05-18T05:52:00.1987539Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050935.xml (deflated 41%)
2022-05-18T05:52:00.1988297Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050941.xml (deflated 41%)
2022-05-18T05:52:00.1989050Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050948.xml (deflated 42%)
2022-05-18T05:52:00.1989861Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050951.xml (deflated 41%)
2022-05-18T05:52:00.1990640Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050957.xml (deflated 41%)
2022-05-18T05:52:00.1991398Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051004.xml (deflated 40%)
2022-05-18T05:52:00.1992141Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051008.xml (deflated 40%)
2022-05-18T05:52:00.1992907Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051015.xml (deflated 41%)
2022-05-18T05:52:00.1993729Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051021.xml (deflated 41%)
2022-05-18T05:52:00.1994494Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051037.xml (deflated 41%)
2022-05-18T05:52:00.1995239Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051044.xml (deflated 41%)
2022-05-18T05:52:00.1996002Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051050.xml (deflated 41%)
2022-05-18T05:52:00.1996765Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051056.xml (deflated 41%)
2022-05-18T05:52:00.1997526Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051103.xml (deflated 41%)
2022-05-18T05:52:00.1998274Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051109.xml (deflated 42%)
2022-05-18T05:52:00.1999049Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051112.xml (deflated 42%)
2022-05-18T05:52:00.1999812Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051119.xml (deflated 41%)
2022-05-18T05:52:00.2000578Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051125.xml (deflated 41%)
2022-05-18T05:52:00.2001320Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051131.xml (deflated 41%)
2022-05-18T05:52:00.2002087Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051137.xml (deflated 42%)
2022-05-18T05:52:00.2002850Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051140.xml (deflated 40%)
2022-05-18T05:52:00.2003620Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051148.xml (deflated 41%)
2022-05-18T05:52:00.2004362Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051154.xml (deflated 41%)
2022-05-18T05:52:00.2005126Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051200.xml (deflated 41%)
2022-05-18T05:52:00.2005880Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051206.xml (deflated 42%)
2022-05-18T05:52:00.2006642Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051209.xml (deflated 40%)
2022-05-18T05:52:00.2007399Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051215.xml (deflated 41%)
2022-05-18T05:52:00.2008146Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051222.xml (deflated 41%)
2022-05-18T05:52:00.2008961Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051226.xml (deflated 41%)
2022-05-18T05:52:00.2009901Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051231.xml (deflated 41%)
2022-05-18T05:52:00.2010676Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051237.xml (deflated 41%)
2022-05-18T05:52:00.2011420Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051244.xml (deflated 40%)
2022-05-18T05:52:00.2012211Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051248.xml (deflated 41%)
2022-05-18T05:52:00.2013064Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051253.xml (deflated 41%)
2022-05-18T05:52:00.2013835Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051257.xml (deflated 42%)
2022-05-18T05:52:00.2014576Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051259.xml (deflated 41%)
2022-05-18T05:52:00.2015341Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051303.xml (deflated 42%)
2022-05-18T05:52:00.2016103Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051308.xml (deflated 40%)
2022-05-18T05:52:00.2016861Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051312.xml (deflated 41%)
2022-05-18T05:52:00.2017615Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051317.xml (deflated 41%)
2022-05-18T05:52:00.2018374Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051322.xml (deflated 40%)
2022-05-18T05:52:00.2019138Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051326.xml (deflated 41%)
2022-05-18T05:52:00.2019901Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051331.xml (deflated 40%)
2022-05-18T05:52:00.2020645Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051335.xml (deflated 41%)
2022-05-18T05:52:00.2021410Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051340.xml (deflated 40%)
2022-05-18T05:52:00.2022172Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051346.xml (deflated 41%)
2022-05-18T05:52:00.2022938Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051351.xml (deflated 41%)
2022-05-18T05:52:00.2023684Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051356.xml (deflated 40%)
2022-05-18T05:52:00.2024446Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051400.xml (deflated 41%)
2022-05-18T05:52:00.2025207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051405.xml (deflated 40%)
2022-05-18T05:52:00.2025969Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051406.xml (deflated 40%)
2022-05-18T05:52:00.2026712Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051408.xml (deflated 43%)
2022-05-18T05:52:00.2027547Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051412.xml (deflated 41%)
2022-05-18T05:52:00.2028327Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051419.xml (deflated 40%)
2022-05-18T05:52:00.2029084Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051424.xml (deflated 40%)
2022-05-18T05:52:00.2029827Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051428.xml (deflated 42%)
2022-05-18T05:52:00.2030583Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051433.xml (deflated 42%)
2022-05-18T05:52:00.2031340Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051434.xml (deflated 42%)
2022-05-18T05:52:00.2032167Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051436.xml (deflated 42%)
2022-05-18T05:52:00.2032919Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051437.xml (deflated 42%)
2022-05-18T05:52:00.2033682Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051438.xml (deflated 43%)
2022-05-18T05:52:00.2034444Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051440.xml (deflated 42%)
2022-05-18T05:52:00.2035207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051441.xml (deflated 42%)
2022-05-18T05:52:00.2035954Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051442.xml (deflated 42%)
2022-05-18T05:52:00.2036719Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051443.xml (deflated 40%)
2022-05-18T05:52:00.2037482Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051448.xml (deflated 41%)
2022-05-18T05:52:00.2038245Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051453.xml (deflated 42%)
2022-05-18T05:52:00.2038986Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051454.xml (deflated 43%)
2022-05-18T05:52:00.2039749Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051455.xml (deflated 41%)
2022-05-18T05:52:00.2040510Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051502.xml (deflated 41%)
2022-05-18T05:52:00.2041274Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051508.xml (deflated 41%)
2022-05-18T05:52:00.2042021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051515.xml (deflated 41%)
2022-05-18T05:52:00.2042793Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051522.xml (deflated 41%)
2022-05-18T05:52:00.2043558Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051525.xml (deflated 41%)
2022-05-18T05:52:00.2044317Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051528.xml (deflated 41%)
2022-05-18T05:52:00.2045060Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051533.xml (deflated 41%)
2022-05-18T05:52:00.2045831Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051537.xml (deflated 40%)
2022-05-18T05:52:00.2046644Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051542.xml (deflated 41%)
2022-05-18T05:52:00.2047417Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051547.xml (deflated 40%)
2022-05-18T05:52:00.2048165Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051551.xml (deflated 41%)
2022-05-18T05:52:00.2048986Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051556.xml (deflated 42%)
2022-05-18T05:52:00.2049943Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051600.xml (deflated 42%)
2022-05-18T05:52:00.2050717Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051605.xml (deflated 43%)
2022-05-18T05:52:00.2051580Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051609.xml (deflated 43%)
2022-05-18T05:52:00.2052328Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051614.xml (deflated 41%)
2022-05-18T05:52:00.2053092Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051619.xml (deflated 41%)
2022-05-18T05:52:00.2053853Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051624.xml (deflated 43%)
2022-05-18T05:52:00.2054613Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051625.xml (deflated 41%)
2022-05-18T05:52:00.2055359Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051629.xml (deflated 40%)
2022-05-18T05:52:00.2056412Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051634.xml (deflated 42%)
2022-05-18T05:52:00.2057179Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051635.xml (deflated 42%)
2022-05-18T05:52:00.2057946Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051637.xml (deflated 41%)
2022-05-18T05:52:00.2058693Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051641.xml (deflated 41%)
2022-05-18T05:52:00.2059456Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051646.xml (deflated 40%)
2022-05-18T05:52:00.2060219Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051650.xml (deflated 41%)
2022-05-18T05:52:00.2060988Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051655.xml (deflated 41%)
2022-05-18T05:52:00.2061741Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051656.xml (deflated 41%)
2022-05-18T05:52:00.2062509Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051658.xml (deflated 40%)
2022-05-18T05:52:00.2063276Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051702.xml (deflated 42%)
2022-05-18T05:52:00.2064043Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051707.xml (deflated 41%)
2022-05-18T05:52:00.2064795Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051711.xml (deflated 41%)
2022-05-18T05:52:00.2065563Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051716.xml (deflated 40%)
2022-05-18T05:52:00.2066402Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051721.xml (deflated 40%)
2022-05-18T05:52:00.2067182Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051725.xml (deflated 41%)
2022-05-18T05:52:00.2067932Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051730.xml (deflated 41%)
2022-05-18T05:52:00.2068697Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051734.xml (deflated 41%)
2022-05-18T05:52:00.2069462Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051736.xml (deflated 41%)
2022-05-18T05:52:00.2070295Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051737.xml (deflated 41%)
2022-05-18T05:52:00.2071045Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051738.xml (deflated 40%)
2022-05-18T05:52:00.2071808Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051743.xml (deflated 40%)
2022-05-18T05:52:00.2072572Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051748.xml (deflated 41%)
2022-05-18T05:52:00.2073333Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051752.xml (deflated 41%)
2022-05-18T05:52:00.2074083Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051757.xml (deflated 41%)
2022-05-18T05:52:00.2074853Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051801.xml (deflated 41%)
2022-05-18T05:52:00.2075619Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051807.xml (deflated 41%)
2022-05-18T05:52:00.2076383Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051814.xml (deflated 41%)
2022-05-18T05:52:00.2077134Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051818.xml (deflated 41%)
2022-05-18T05:52:00.2077899Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051824.xml (deflated 41%)
2022-05-18T05:52:00.2078670Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051830.xml (deflated 41%)
2022-05-18T05:52:00.2079435Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051841.xml (deflated 40%)
2022-05-18T05:52:00.2080190Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051854.xml (deflated 42%)
2022-05-18T05:52:00.2080956Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051901.xml (deflated 42%)
2022-05-18T05:52:00.2081723Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051902.xml (deflated 42%)
2022-05-18T05:52:00.2082489Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051907.xml (deflated 42%)
2022-05-18T05:52:00.2083231Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051910.xml (deflated 41%)
2022-05-18T05:52:00.2083999Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051914.xml (deflated 41%)
2022-05-18T05:52:00.2084817Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051919.xml (deflated 40%)
2022-05-18T05:52:00.2085593Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051926.xml (deflated 40%)
2022-05-18T05:52:00.2086342Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051932.xml (deflated 40%)
2022-05-18T05:52:00.2087106Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051938.xml (deflated 40%)
2022-05-18T05:52:00.2111073Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051944.xml (deflated 40%)
2022-05-18T05:52:00.2111978Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051951.xml (deflated 40%)
2022-05-18T05:52:00.2113028Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051957.xml (deflated 40%)
2022-05-18T05:52:00.2113849Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052003.xml (deflated 42%)
2022-05-18T05:52:00.2114683Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052006.xml (deflated 42%)
2022-05-18T05:52:00.2115516Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052011.xml (deflated 42%)
2022-05-18T05:52:00.2116354Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052014.xml (deflated 42%)
2022-05-18T05:52:00.2117174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052020.xml (deflated 42%)
2022-05-18T05:52:00.2118019Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052022.xml (deflated 46%)
2022-05-18T05:52:00.2118855Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052026.xml (deflated 47%)
2022-05-18T05:52:00.2119684Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052031.xml (deflated 48%)
2022-05-18T05:52:00.2120497Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052035.xml (deflated 46%)
2022-05-18T05:52:00.2121332Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052040.xml (deflated 41%)
2022-05-18T05:52:00.2122167Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052044.xml (deflated 41%)
2022-05-18T05:52:00.2123005Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052049.xml (deflated 41%)
2022-05-18T05:52:00.2123824Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052054.xml (deflated 42%)
2022-05-18T05:52:00.2124654Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052058.xml (deflated 41%)
2022-05-18T05:52:00.2125481Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052103.xml (deflated 41%)
2022-05-18T05:52:00.2126308Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052107.xml (deflated 41%)
2022-05-18T05:52:00.2127114Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052112.xml (deflated 42%)
2022-05-18T05:52:00.2127947Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052113.xml (deflated 42%)
2022-05-18T05:52:00.2128857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052115.xml (deflated 41%)
2022-05-18T05:52:00.2129927Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052119.xml (deflated 43%)
2022-05-18T05:52:00.2130763Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052124.xml (deflated 43%)
2022-05-18T05:52:00.2131595Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052125.xml (deflated 43%)
2022-05-18T05:52:00.2132423Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052126.xml (deflated 40%)
2022-05-18T05:52:00.2133250Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052131.xml (deflated 40%)
2022-05-18T05:52:00.2134174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052136.xml (deflated 41%)
2022-05-18T05:52:00.2134993Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052140.xml (deflated 41%)
2022-05-18T05:52:00.2135813Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052145.xml (deflated 41%)
2022-05-18T05:52:00.2136623Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052150.xml (deflated 41%)
2022-05-18T05:52:00.2137431Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052154.xml (deflated 42%)
2022-05-18T05:52:00.2138235Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052159.xml (deflated 42%)
2022-05-18T05:52:00.2139072Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052203.xml (deflated 42%)
2022-05-18T05:52:00.2139902Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052208.xml (deflated 42%)
2022-05-18T05:52:00.2140717Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052213.xml (deflated 41%)
2022-05-18T05:52:00.2141531Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052217.xml (deflated 41%)
2022-05-18T05:52:00.2142363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052222.xml (deflated 41%)
2022-05-18T05:52:00.2143191Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052226.xml (deflated 41%)
2022-05-18T05:52:00.2144020Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052231.xml (deflated 41%)
2022-05-18T05:52:00.2144836Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052236.xml (deflated 41%)
2022-05-18T05:52:00.2145664Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052240.xml (deflated 41%)
2022-05-18T05:52:00.2146493Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052245.xml (deflated 41%)
2022-05-18T05:52:00.2147318Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052249.xml (deflated 41%)
2022-05-18T05:52:00.2148134Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052254.xml (deflated 41%)
2022-05-18T05:52:00.2148968Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052259.xml (deflated 42%)
2022-05-18T05:52:00.2149867Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052303.xml (deflated 42%)
2022-05-18T05:52:00.2150717Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052308.xml (deflated 42%)
2022-05-18T05:52:00.2151529Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052312.xml (deflated 42%)
2022-05-18T05:52:00.2152360Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052317.xml (deflated 41%)
2022-05-18T05:52:00.2153187Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052322.xml (deflated 41%)
2022-05-18T05:52:00.2154090Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052326.xml (deflated 41%)
2022-05-18T05:52:00.2154907Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052333.xml (deflated 40%)
2022-05-18T05:52:00.2155725Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052339.xml (deflated 41%)
2022-05-18T05:52:00.2156535Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052344.xml (deflated 41%)
2022-05-18T05:52:00.2157342Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052350.xml (deflated 41%)
2022-05-18T05:52:00.2158152Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052354.xml (deflated 41%)
2022-05-18T05:52:00.2158966Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052359.xml (deflated 41%)
2022-05-18T05:52:00.2159776Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052404.xml (deflated 41%)
2022-05-18T05:52:00.2160583Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052412.xml (deflated 41%)
2022-05-18T05:52:00.2161387Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052421.xml (deflated 41%)
2022-05-18T05:52:00.2162200Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052429.xml (deflated 43%)
2022-05-18T05:52:00.2163008Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052430.xml (deflated 43%)
2022-05-18T05:52:00.2163818Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052431.xml (deflated 42%)
2022-05-18T05:52:00.2164627Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052433.xml (deflated 42%)
2022-05-18T05:52:00.2165436Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052434.xml (deflated 43%)
2022-05-18T05:52:00.2166246Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052435.xml (deflated 42%)
2022-05-18T05:52:00.2167073Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052437.xml (deflated 43%)
2022-05-18T05:52:00.2167887Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052438.xml (deflated 42%)
2022-05-18T05:52:00.2168716Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052439.xml (deflated 43%)
2022-05-18T05:52:00.2169813Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052440.xml (deflated 43%)
2022-05-18T05:52:00.2170677Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052442.xml (deflated 43%)
2022-05-18T05:52:00.2171489Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052443.xml (deflated 42%)
2022-05-18T05:52:00.2172321Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052444.xml (deflated 43%)
2022-05-18T05:52:00.2173156Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052445.xml (deflated 43%)
2022-05-18T05:52:00.2173978Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052447.xml (deflated 42%)
2022-05-18T05:52:00.2174889Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052448.xml (deflated 43%)
2022-05-18T05:52:00.2175721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052449.xml (deflated 42%)
2022-05-18T05:52:00.2176550Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052451.xml (deflated 42%)
2022-05-18T05:52:00.2177379Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052452.xml (deflated 43%)
2022-05-18T05:52:00.2178196Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052453.xml (deflated 42%)
2022-05-18T05:52:00.2179024Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052454.xml (deflated 42%)
2022-05-18T05:52:00.2179858Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052456.xml (deflated 42%)
2022-05-18T05:52:00.2180689Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052457.xml (deflated 42%)
2022-05-18T05:52:00.2181504Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052458.xml (deflated 43%)
2022-05-18T05:52:00.2182339Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052459.xml (deflated 41%)
2022-05-18T05:52:00.2183169Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052507.xml (deflated 42%)
2022-05-18T05:52:00.2184000Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052511.xml (deflated 43%)
2022-05-18T05:52:00.2184816Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052512.xml (deflated 41%)
2022-05-18T05:52:00.2185641Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052518.xml (deflated 41%)
2022-05-18T05:52:00.2186473Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052524.xml (deflated 40%)
2022-05-18T05:52:00.2187304Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052530.xml (deflated 42%)
2022-05-18T05:52:00.2188131Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052534.xml (deflated 43%)
2022-05-18T05:52:00.2188946Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052539.xml (deflated 42%)
2022-05-18T05:52:00.2189779Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052543.xml (deflated 40%)
2022-05-18T05:52:00.2190663Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052549.xml (deflated 41%)
2022-05-18T05:52:00.2191504Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052555.xml (deflated 42%)
2022-05-18T05:52:00.2192319Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052559.xml (deflated 41%)
2022-05-18T05:52:00.2193148Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052604.xml (deflated 40%)
2022-05-18T05:52:00.2193978Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052608.xml (deflated 42%)
2022-05-18T05:52:00.2194805Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052610.xml (deflated 42%)
2022-05-18T05:52:00.2195715Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052611.xml (deflated 41%)
2022-05-18T05:52:00.2196543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052612.xml (deflated 41%)
2022-05-18T05:52:00.2197367Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052614.xml (deflated 41%)
2022-05-18T05:52:00.2198189Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052615.xml (deflated 41%)
2022-05-18T05:52:00.2199001Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052616.xml (deflated 41%)
2022-05-18T05:52:00.2199820Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052617.xml (deflated 41%)
2022-05-18T05:52:00.2200644Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052619.xml (deflated 41%)
2022-05-18T05:52:00.2201474Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052623.xml (deflated 40%)
2022-05-18T05:52:00.2202288Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052630.xml (deflated 40%)
2022-05-18T05:52:00.2203107Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052634.xml (deflated 42%)
2022-05-18T05:52:00.2203927Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052639.xml (deflated 40%)
2022-05-18T05:52:00.2204741Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052645.xml (deflated 41%)
2022-05-18T05:52:00.2205561Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052649.xml (deflated 41%)
2022-05-18T05:52:00.2206390Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052656.xml (deflated 41%)
2022-05-18T05:52:00.2207222Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052702.xml (deflated 41%)
2022-05-18T05:52:00.2208046Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052708.xml (deflated 41%)
2022-05-18T05:52:00.2208858Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052714.xml (deflated 40%)
2022-05-18T05:52:00.2209961Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052721.xml (deflated 42%)
2022-05-18T05:52:00.2210815Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052724.xml (deflated 41%)
2022-05-18T05:52:00.2211736Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052730.xml (deflated 41%)
2022-05-18T05:52:00.2212607Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052736.xml (deflated 40%)
2022-05-18T05:52:00.2213436Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052742.xml (deflated 41%)
2022-05-18T05:52:00.2214263Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052748.xml (deflated 40%)
2022-05-18T05:52:00.2215093Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052755.xml (deflated 41%)
2022-05-18T05:52:00.2215990Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052759.xml (deflated 41%)
2022-05-18T05:52:00.2216820Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052806.xml (deflated 41%)
2022-05-18T05:52:00.2217652Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052812.xml (deflated 41%)
2022-05-18T05:52:00.2218480Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052818.xml (deflated 41%)
2022-05-18T05:52:00.2219290Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052821.xml (deflated 41%)
2022-05-18T05:52:00.2220127Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052828.xml (deflated 40%)
2022-05-18T05:52:00.2220963Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052835.xml (deflated 40%)
2022-05-18T05:52:00.2221799Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052843.xml (deflated 41%)
2022-05-18T05:52:00.2222612Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052849.xml (deflated 41%)
2022-05-18T05:52:00.2223437Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052856.xml (deflated 41%)
2022-05-18T05:52:00.2224268Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052902.xml (deflated 41%)
2022-05-18T05:52:00.2225092Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052909.xml (deflated 41%)
2022-05-18T05:52:00.2225899Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052915.xml (deflated 41%)
2022-05-18T05:52:00.2226736Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052922.xml (deflated 41%)
2022-05-18T05:52:00.2227566Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052928.xml (deflated 41%)
2022-05-18T05:52:00.2228394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052935.xml (deflated 41%)
2022-05-18T05:52:00.2229201Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052942.xml (deflated 41%)
2022-05-18T05:52:00.2230029Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052948.xml (deflated 41%)
2022-05-18T05:52:00.2230857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518052955.xml (deflated 41%)
2022-05-18T05:52:00.2231744Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053001.xml (deflated 41%)
2022-05-18T05:52:00.2232573Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053004.xml (deflated 41%)
2022-05-18T05:52:00.2233408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053011.xml (deflated 40%)
2022-05-18T05:52:00.2234238Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053017.xml (deflated 40%)
2022-05-18T05:52:00.2235068Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053022.xml (deflated 41%)
2022-05-18T05:52:00.2235899Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053028.xml (deflated 41%)
2022-05-18T05:52:00.2236777Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053034.xml (deflated 41%)
2022-05-18T05:52:00.2237609Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053050.xml (deflated 41%)
2022-05-18T05:52:00.2238436Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053057.xml (deflated 41%)
2022-05-18T05:52:00.2239266Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053104.xml (deflated 41%)
2022-05-18T05:52:00.2240078Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053110.xml (deflated 41%)
2022-05-18T05:52:00.2240906Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053116.xml (deflated 41%)
2022-05-18T05:52:00.2241737Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053123.xml (deflated 42%)
2022-05-18T05:52:00.2242571Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053126.xml (deflated 42%)
2022-05-18T05:52:00.2243388Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053133.xml (deflated 40%)
2022-05-18T05:52:00.2244215Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053139.xml (deflated 41%)
2022-05-18T05:52:00.2245047Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053145.xml (deflated 41%)
2022-05-18T05:52:00.2245514Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053151.xml (deflated 42%)
2022-05-18T05:52:00.2245993Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053154.xml (deflated 41%)
2022-05-18T05:52:00.2246468Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053201.xml (deflated 41%)
2022-05-18T05:52:00.2246925Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053207.xml (deflated 41%)
2022-05-18T05:52:00.2247394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053213.xml (deflated 41%)
2022-05-18T05:52:00.2247866Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053220.xml (deflated 42%)
2022-05-18T05:52:00.2248337Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053223.xml (deflated 41%)
2022-05-18T05:52:00.2248815Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053229.xml (deflated 41%)
2022-05-18T05:52:00.2249329Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053236.xml (deflated 41%)
2022-05-18T05:52:00.2250001Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053240.xml (deflated 41%)
2022-05-18T05:52:00.2250483Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053245.xml (deflated 41%)
2022-05-18T05:52:00.2250956Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053251.xml (deflated 40%)
2022-05-18T05:52:00.2251427Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053258.xml (deflated 40%)
2022-05-18T05:52:00.2251878Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053302.xml (deflated 41%)
2022-05-18T05:52:00.2252469Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053307.xml (deflated 41%)
2022-05-18T05:52:00.2252942Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053311.xml (deflated 43%)
2022-05-18T05:52:00.2253407Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053312.xml (deflated 41%)
2022-05-18T05:52:00.2253878Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053317.xml (deflated 42%)
2022-05-18T05:52:00.2254347Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053322.xml (deflated 41%)
2022-05-18T05:52:00.2254818Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053326.xml (deflated 40%)
2022-05-18T05:52:00.2255297Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053331.xml (deflated 41%)
2022-05-18T05:52:00.2255768Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053336.xml (deflated 40%)
2022-05-18T05:52:00.2256241Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053340.xml (deflated 41%)
2022-05-18T05:52:00.2256698Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053345.xml (deflated 40%)
2022-05-18T05:52:00.2257169Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053350.xml (deflated 41%)
2022-05-18T05:52:00.2257637Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053354.xml (deflated 40%)
2022-05-18T05:52:00.2258114Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053401.xml (deflated 41%)
2022-05-18T05:52:00.2258585Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053405.xml (deflated 41%)
2022-05-18T05:52:00.2259057Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053410.xml (deflated 41%)
2022-05-18T05:52:00.2259526Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053415.xml (deflated 41%)
2022-05-18T05:52:00.2259997Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053419.xml (deflated 40%)
2022-05-18T05:52:00.2260468Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053421.xml (deflated 40%)
2022-05-18T05:52:00.2260944Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053422.xml (deflated 42%)
2022-05-18T05:52:00.2261471Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053427.xml (deflated 41%)
2022-05-18T05:52:00.2261938Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053433.xml (deflated 41%)
2022-05-18T05:52:00.2262403Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053438.xml (deflated 40%)
2022-05-18T05:52:00.2262944Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053443.xml (deflated 42%)
2022-05-18T05:52:00.2263407Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053447.xml (deflated 42%)
2022-05-18T05:52:00.2263928Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053448.xml (deflated 42%)
2022-05-18T05:52:00.2264392Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053450.xml (deflated 42%)
2022-05-18T05:52:00.2264846Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053451.xml (deflated 42%)
2022-05-18T05:52:00.2265305Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053452.xml (deflated 43%)
2022-05-18T05:52:00.2265774Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053453.xml (deflated 42%)
2022-05-18T05:52:00.2266245Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053455.xml (deflated 42%)
2022-05-18T05:52:00.2266721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053456.xml (deflated 42%)
2022-05-18T05:52:00.2267193Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053457.xml (deflated 41%)
2022-05-18T05:52:00.2267664Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053502.xml (deflated 41%)
2022-05-18T05:52:00.2268131Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053507.xml (deflated 42%)
2022-05-18T05:52:00.2268603Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053508.xml (deflated 43%)
2022-05-18T05:52:00.2269074Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053509.xml (deflated 41%)
2022-05-18T05:52:00.2269529Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053516.xml (deflated 41%)
2022-05-18T05:52:00.2270007Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053522.xml (deflated 41%)
2022-05-18T05:52:00.2270475Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053529.xml (deflated 41%)
2022-05-18T05:52:00.2270944Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053536.xml (deflated 41%)
2022-05-18T05:52:00.2271415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053539.xml (deflated 41%)
2022-05-18T05:52:00.2271884Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053542.xml (deflated 41%)
2022-05-18T05:52:00.2272358Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053547.xml (deflated 40%)
2022-05-18T05:52:00.2272873Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053551.xml (deflated 41%)
2022-05-18T05:52:00.2273351Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053556.xml (deflated 40%)
2022-05-18T05:52:00.2273826Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053601.xml (deflated 41%)
2022-05-18T05:52:00.2274281Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053605.xml (deflated 41%)
2022-05-18T05:52:00.2274752Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053610.xml (deflated 43%)
2022-05-18T05:52:00.2275223Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053614.xml (deflated 43%)
2022-05-18T05:52:00.2275750Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053619.xml (deflated 42%)
2022-05-18T05:52:00.2276221Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053624.xml (deflated 43%)
2022-05-18T05:52:00.2276690Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053628.xml (deflated 41%)
2022-05-18T05:52:00.2277158Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053633.xml (deflated 41%)
2022-05-18T05:52:00.2277630Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053637.xml (deflated 43%)
2022-05-18T05:52:00.2278102Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053638.xml (deflated 41%)
2022-05-18T05:52:00.2278572Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053643.xml (deflated 40%)
2022-05-18T05:52:00.2279040Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053648.xml (deflated 42%)
2022-05-18T05:52:00.2279492Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053649.xml (deflated 42%)
2022-05-18T05:52:00.2279960Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053650.xml (deflated 41%)
2022-05-18T05:52:00.2280429Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053655.xml (deflated 41%)
2022-05-18T05:52:00.2280900Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053659.xml (deflated 41%)
2022-05-18T05:52:00.2281375Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053704.xml (deflated 41%)
2022-05-18T05:52:00.2281848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053708.xml (deflated 41%)
2022-05-18T05:52:00.2282320Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053710.xml (deflated 41%)
2022-05-18T05:52:00.2282791Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053711.xml (deflated 40%)
2022-05-18T05:52:00.2283260Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053716.xml (deflated 42%)
2022-05-18T05:52:00.2283730Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053720.xml (deflated 41%)
2022-05-18T05:52:00.2284189Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053725.xml (deflated 41%)
2022-05-18T05:52:00.2284702Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053729.xml (deflated 40%)
2022-05-18T05:52:00.2285179Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053734.xml (deflated 40%)
2022-05-18T05:52:00.2285649Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053738.xml (deflated 41%)
2022-05-18T05:52:00.2286111Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053743.xml (deflated 40%)
2022-05-18T05:52:00.2286581Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053748.xml (deflated 41%)
2022-05-18T05:52:00.2287054Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053749.xml (deflated 41%)
2022-05-18T05:52:00.2287580Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053750.xml (deflated 41%)
2022-05-18T05:52:00.2288048Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053751.xml (deflated 41%)
2022-05-18T05:52:00.2288518Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053756.xml (deflated 41%)
2022-05-18T05:52:00.2288990Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053801.xml (deflated 41%)
2022-05-18T05:52:00.2289448Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053805.xml (deflated 41%)
2022-05-18T05:52:00.2290111Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053810.xml (deflated 41%)
2022-05-18T05:52:00.2290595Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053814.xml (deflated 41%)
2022-05-18T05:52:00.2291065Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053820.xml (deflated 41%)
2022-05-18T05:52:00.2291535Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053827.xml (deflated 41%)
2022-05-18T05:52:00.2292004Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053831.xml (deflated 41%)
2022-05-18T05:52:00.2292473Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053837.xml (deflated 41%)
2022-05-18T05:52:00.2292942Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053843.xml (deflated 40%)
2022-05-18T05:52:00.2293411Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518053854.xml (deflated 40%)
2022-05-18T05:52:00.2294003Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220518053906.xml (deflated 90%)
2022-05-18T05:52:00.2294578Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220518053906.xml (deflated 73%)
2022-05-18T05:52:00.2295056Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20220518054110.xml (deflated 90%)
2022-05-18T05:52:00.2295467Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518054229.xml (deflated 40%)
2022-05-18T05:52:00.2295873Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518054232.xml (deflated 40%)
2022-05-18T05:52:00.2296293Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518054234.xml (deflated 40%)
2022-05-18T05:52:00.2296778Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518054237.xml (deflated 39%)
2022-05-18T05:52:00.2297234Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518054240.xml (deflated 40%)
2022-05-18T05:52:00.2297675Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518054243.xml (deflated 40%)
2022-05-18T05:52:00.2298107Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518054246.xml (deflated 39%)
2022-05-18T05:52:00.2298540Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518054248.xml (deflated 40%)
2022-05-18T05:52:00.2298945Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220518054251.xml (deflated 40%)
2022-05-18T05:52:00.2299454Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220518054254.xml (deflated 39%)
2022-05-18T05:52:00.2299884Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518054257.xml (deflated 40%)
2022-05-18T05:52:00.2300318Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518054300.xml (deflated 40%)
2022-05-18T05:52:00.2300745Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518054302.xml (deflated 39%)
2022-05-18T05:52:00.2301163Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518054305.xml (deflated 39%)
2022-05-18T05:52:00.2301586Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518054308.xml (deflated 39%)
2022-05-18T05:52:00.2302020Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518054311.xml (deflated 40%)
2022-05-18T05:52:00.2302439Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220518054324.xml (deflated 39%)
2022-05-18T05:52:00.2302852Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054326.xml (deflated 39%)
2022-05-18T05:52:00.2303245Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054329.xml (deflated 39%)
2022-05-18T05:52:00.2303655Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054332.xml (deflated 38%)
2022-05-18T05:52:00.2304063Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054335.xml (deflated 38%)
2022-05-18T05:52:00.2304473Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054338.xml (deflated 38%)
2022-05-18T05:52:00.2304883Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054341.xml (deflated 39%)
2022-05-18T05:52:00.2305297Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054343.xml (deflated 38%)
2022-05-18T05:52:00.2305710Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518054348.xml (deflated 39%)
2022-05-18T05:52:00.2306204Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054352.xml (deflated 41%)
2022-05-18T05:52:00.2306695Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054356.xml (deflated 41%)
2022-05-18T05:52:00.2307187Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054401.xml (deflated 41%)
2022-05-18T05:52:00.2307661Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054405.xml (deflated 41%)
2022-05-18T05:52:00.2308194Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054409.xml (deflated 41%)
2022-05-18T05:52:00.2308698Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054413.xml (deflated 41%)
2022-05-18T05:52:00.2309185Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054417.xml (deflated 41%)
2022-05-18T05:52:00.2309673Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054422.xml (deflated 41%)
2022-05-18T05:52:00.2310161Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518054426.xml (deflated 40%)
2022-05-18T05:52:00.2310642Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518054430.xml (deflated 40%)
2022-05-18T05:52:00.2311192Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518054434.xml (deflated 40%)
2022-05-18T05:52:00.2311681Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518054439.xml (deflated 39%)
2022-05-18T05:52:00.2312210Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518054445.xml (deflated 40%)
2022-05-18T05:52:00.2312699Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518054452.xml (deflated 40%)
2022-05-18T05:52:00.2313161Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220518054458.xml (deflated 80%)
2022-05-18T05:52:00.2313628Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220518054458.xml (deflated 86%)
2022-05-18T05:52:00.2314069Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20220518054544.xml (deflated 93%)
2022-05-18T05:52:00.2314556Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20220518054621.xml (deflated 84%)
2022-05-18T05:52:00.2315049Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20220518054656.xml (deflated 64%)
2022-05-18T05:52:00.2315613Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20220518054656.xml (deflated 83%)
2022-05-18T05:52:00.2316076Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20220518054724.xml (deflated 82%)
2022-05-18T05:52:00.2316593Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220518054751.xml (deflated 43%)
2022-05-18T05:52:00.2317116Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220518054751.xml (deflated 43%)
2022-05-18T05:52:00.2317622Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220518054805.xml (deflated 55%)
2022-05-18T05:52:00.2318146Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220518054805.xml (deflated 80%)
2022-05-18T05:52:00.2318635Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220518054805.xml (deflated 79%)
2022-05-18T05:52:00.2319103Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220518054805.xml (deflated 63%)
2022-05-18T05:52:00.2319646Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20220518054823.xml (deflated 86%)
2022-05-18T05:52:00.2320131Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20220518054840.xml (deflated 55%)
2022-05-18T05:52:00.2320636Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20220518054855.xml (deflated 64%)
2022-05-18T05:52:00.2321126Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220518054907.xml (deflated 54%)
2022-05-18T05:52:00.2321549Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20220518054916.xml (deflated 57%)
2022-05-18T05:52:00.2322032Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20220518054925.xml (deflated 72%)
2022-05-18T05:52:00.2322604Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220518054933.xml (deflated 65%)
2022-05-18T05:52:00.2323101Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220518054933.xml (deflated 78%)
2022-05-18T05:52:00.2323627Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20220518054939.xml (deflated 68%)
2022-05-18T05:52:00.2324120Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20220518054946.xml (deflated 68%)
2022-05-18T05:52:00.2324622Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220518054952.xml (deflated 71%)
2022-05-18T05:52:00.2325093Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20220518054958.xml (deflated 42%)
2022-05-18T05:52:00.2325565Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20220518055004.xml (deflated 41%)
2022-05-18T05:52:00.2326016Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20220518055010.xml (deflated 42%)
2022-05-18T05:52:00.2326521Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20220518055015.xml (deflated 60%)
2022-05-18T05:52:00.2327040Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20220518055021.xml (deflated 60%)
2022-05-18T05:52:00.2327572Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220518055026.xml (deflated 60%)
2022-05-18T05:52:00.2328054Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20220518055032.xml (deflated 81%)
2022-05-18T05:52:00.2328557Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20220518055032.xml (deflated 81%)
2022-05-18T05:52:00.2329067Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20220518055032.xml (deflated 81%)
2022-05-18T05:52:00.2329682Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20220518055035.xml (deflated 54%)
2022-05-18T05:52:00.2330168Z   adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220518055038.xml (deflated 63%)
2022-05-18T05:52:00.2330576Z   adding: test/test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20220518055043.xml (deflated 83%)
2022-05-18T05:52:00.2330921Z   adding: test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml (deflated 71%)
2022-05-18T05:52:00.2331272Z   adding: test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml (deflated 72%)
2022-05-18T05:52:00.2331682Z   adding: test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml (deflated 80%)
2022-05-18T05:52:00.2332067Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml (deflated 80%)
2022-05-18T05:52:00.2332428Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml (deflated 80%)
2022-05-18T05:52:00.2332816Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml (deflated 67%)
2022-05-18T05:52:00.2333097Z   adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%)
2022-05-18T05:52:00.2372936Z ##[group]Run seemethere/upload-artifact-s3@v4
2022-05-18T05:52:00.2373017Z with:
2022-05-18T05:52:00.2373131Z   retention-days: 14
2022-05-18T05:52:00.2373252Z   if-no-files-found: warn
2022-05-18T05:52:00.2373472Z   path: test-jsons-*.zip
2022-05-18T05:52:00.2373568Z   name: artifact
2022-05-18T05:52:00.2373677Z   s3-bucket: gha-artifacts
2022-05-18T05:52:00.2373782Z   region: us-east-1
2022-05-18T05:52:00.2373855Z env:
2022-05-18T05:52:00.2373954Z   IN_CI: 1
2022-05-18T05:52:00.2374048Z   IS_GHA: 1
2022-05-18T05:52:00.2374168Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:52:00.2374277Z   GPU_FLAG: --gpus all
2022-05-18T05:52:00.2374378Z ##[endgroup]
2022-05-18T05:52:00.6569551Z With the provided path, there will be 1 file uploaded
2022-05-18T05:52:00.6569968Z Uploading to s3 prefix: pytorch/pytorch/2342799944/1/artifact
2022-05-18T05:52:00.6580583Z Starting upload of test-jsons-test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482805607.zip
2022-05-18T05:52:00.8676365Z Finished upload of test-jsons-test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482805607.zip
2022-05-18T05:52:00.8844022Z ##[group]Run seemethere/upload-artifact-s3@v4
2022-05-18T05:52:00.8844320Z with:
2022-05-18T05:52:00.8844559Z   retention-days: 14
2022-05-18T05:52:00.8844820Z   if-no-files-found: error
2022-05-18T05:52:00.8845100Z   path: test-reports-*.zip
2022-05-18T05:52:00.8845351Z   name: artifact
2022-05-18T05:52:00.8845601Z   s3-bucket: gha-artifacts
2022-05-18T05:52:00.8845852Z   region: us-east-1
2022-05-18T05:52:00.8846081Z env:
2022-05-18T05:52:00.8846293Z   IN_CI: 1
2022-05-18T05:52:00.8846496Z   IS_GHA: 1
2022-05-18T05:52:00.8846741Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:52:00.8847001Z   GPU_FLAG: --gpus all
2022-05-18T05:52:00.8847227Z ##[endgroup]
2022-05-18T05:52:01.3107621Z With the provided path, there will be 1 file uploaded
2022-05-18T05:52:01.3108036Z Uploading to s3 prefix: pytorch/pytorch/2342799944/1/artifact
2022-05-18T05:52:01.3118653Z Starting upload of test-reports-test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482805607.zip
2022-05-18T05:52:01.4915686Z Finished upload of test-reports-test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482805607.zip
2022-05-18T05:52:01.5077628Z ##[group]Run set -x
2022-05-18T05:52:01.5077922Z set -x
2022-05-18T05:52:01.5078219Z python3 -m pip install -r requirements.txt
2022-05-18T05:52:01.5078559Z python3 -m pip install boto3==1.19.12
2022-05-18T05:52:01.5078953Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-05-18T05:52:01.5092872Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T05:52:01.5093170Z env:
2022-05-18T05:52:01.5093391Z   IN_CI: 1
2022-05-18T05:52:01.5093596Z   IS_GHA: 1
2022-05-18T05:52:01.5093846Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:52:01.5094116Z   GPU_FLAG: --gpus all
2022-05-18T05:52:01.5094365Z   AWS_DEFAULT_REGION: us-east-1
2022-05-18T05:52:01.5094625Z   BRANCH: master
2022-05-18T05:52:01.5094940Z   JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test
2022-05-18T05:52:01.5095246Z   TEST_CONFIG: distributed
2022-05-18T05:52:01.5095499Z   SHARD_NUMBER: 1
2022-05-18T05:52:01.5095851Z   BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7
2022-05-18T05:52:01.5096150Z   PR_NUMBER: 
2022-05-18T05:52:01.5096430Z   SHA1: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T05:52:01.5096698Z   TAG: 
2022-05-18T05:52:01.5096910Z   WORKFLOW_ID: 2342799944
2022-05-18T05:52:01.5097341Z   GITHUB_TOKEN: ***
2022-05-18T05:52:01.5097605Z   GHA_WORKFLOW_JOB_ID: 6482805607
2022-05-18T05:52:01.5097863Z ##[endgroup]
2022-05-18T05:52:01.5128015Z + python3 -m pip install -r requirements.txt
2022-05-18T05:52:01.8079802Z Defaulting to user installation because normal site-packages is not writeable
2022-05-18T05:52:01.8381392Z Ignoring dataclasses: markers 'python_version < "3.7"' don't match your environment
2022-05-18T05:52:01.8869792Z Collecting astunparse
2022-05-18T05:52:01.9019982Z   Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
2022-05-18T05:52:01.9333174Z Collecting expecttest
2022-05-18T05:52:01.9371346Z   Downloading expecttest-0.1.3-py3-none-any.whl (6.5 kB)
2022-05-18T05:52:01.9746774Z Collecting future
2022-05-18T05:52:01.9789978Z   Downloading future-0.18.2.tar.gz (829 kB)
2022-05-18T05:52:03.2162099Z Collecting numpy
2022-05-18T05:52:03.2217497Z   Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
2022-05-18T05:52:03.8673357Z Collecting psutil
2022-05-18T05:52:03.8781574Z   Downloading psutil-5.9.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)
2022-05-18T05:52:04.0068734Z Collecting pyyaml
2022-05-18T05:52:04.0115550Z   Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
2022-05-18T05:52:04.0338309Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (2.26.0)
2022-05-18T05:52:04.0507677Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (49.1.3)
2022-05-18T05:52:04.1095753Z Collecting six
2022-05-18T05:52:04.1136180Z   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
2022-05-18T05:52:04.1449374Z Collecting types-dataclasses
2022-05-18T05:52:04.1488103Z   Downloading types_dataclasses-0.6.5-py3-none-any.whl (2.8 kB)
2022-05-18T05:52:04.1876205Z Collecting typing_extensions
2022-05-18T05:52:04.1917959Z   Downloading typing_extensions-4.2.0-py3-none-any.whl (24 kB)
2022-05-18T05:52:04.2734270Z Collecting wheel<1.0,>=0.23.0
2022-05-18T05:52:04.2773970Z   Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
2022-05-18T05:52:04.2900887Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (3.3)
2022-05-18T05:52:04.2915750Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (1.26.9)
2022-05-18T05:52:04.3124909Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (2021.10.8)
2022-05-18T05:52:04.3135384Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (2.0.12)
2022-05-18T05:52:04.3162497Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
2022-05-18T05:52:04.3524545Z Installing collected packages: six, wheel, astunparse, expecttest, future, numpy, psutil, pyyaml, types-dataclasses, typing-extensions
2022-05-18T05:52:04.3983338Z   WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-05-18T05:52:04.3983960Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-05-18T05:52:04.4308006Z     Running setup.py install for future: started
2022-05-18T05:52:05.0931935Z     Running setup.py install for future: finished with status 'done'
2022-05-18T05:52:07.1153775Z   WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-05-18T05:52:07.1154494Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-05-18T05:52:07.3977109Z Successfully installed astunparse-1.6.3 expecttest-0.1.3 future-0.18.2 numpy-1.21.6 psutil-5.9.0 pyyaml-6.0 six-1.16.0 types-dataclasses-0.6.5 typing-extensions-4.2.0 wheel-0.37.1
2022-05-18T05:52:07.4664337Z + python3 -m pip install boto3==1.19.12
2022-05-18T05:52:07.7633973Z Defaulting to user installation because normal site-packages is not writeable
2022-05-18T05:52:08.5752695Z Collecting boto3==1.19.12
2022-05-18T05:52:08.5937999Z   Downloading boto3-1.19.12-py3-none-any.whl (131 kB)
2022-05-18T05:52:08.6448663Z Collecting jmespath<1.0.0,>=0.7.1
2022-05-18T05:52:08.6487557Z   Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
2022-05-18T05:52:08.6928412Z Collecting s3transfer<0.6.0,>=0.5.0
2022-05-18T05:52:08.6968633Z   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
2022-05-18T05:52:09.6646311Z Collecting botocore<1.23.0,>=1.22.12
2022-05-18T05:52:09.6697406Z   Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB)
2022-05-18T05:52:09.9018136Z Collecting python-dateutil<3.0.0,>=2.1
2022-05-18T05:52:09.9061211Z   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
2022-05-18T05:52:09.9235813Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.9)
2022-05-18T05:52:09.9445057Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-05-18T05:52:10.0287661Z Installing collected packages: jmespath, python-dateutil, botocore, s3transfer, boto3
2022-05-18T05:52:10.9265823Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2
2022-05-18T05:52:10.9778159Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-05-18T05:52:15.5019923Z [scribe] Scribe access token not provided, sending report via boto3...
2022-05-18T05:52:15.5020380Z 
2022-05-18T05:52:15.5020926Z ----- Historic stats comparison result ------
2022-05-18T05:52:15.5021145Z 
2022-05-18T05:52:15.5021596Z     job: linux-xenial-cuda11.3-py3.7-gcc7-test
2022-05-18T05:52:15.5021969Z     commit: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T05:52:15.5022173Z 
2022-05-18T05:52:15.5022398Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-05-18T05:52:15.5022628Z 
2022-05-18T05:52:15.5022729Z     : (master)
2022-05-18T05:52:15.5022947Z     |
2022-05-18T05:52:15.5023207Z     * 3b2375291a (HEAD)              total time  1347.12s
2022-05-18T05:52:15.5023773Z     * 6e3391a7c3 (base)   6 reports, total time  3270.69s ±  964.40s
2022-05-18T05:52:15.5024559Z     * 48581d74ad          6 reports, total time  3345.34s ± 1040.01s
2022-05-18T05:52:15.5025022Z     * c35bd8d423          5 reports, total time  3369.49s ±  631.66s
2022-05-18T05:52:15.5025421Z     * f6beda89c6          6 reports, total time  3022.28s ±  967.61s
2022-05-18T05:52:15.5025840Z     * ee080918df          6 reports, total time  3423.34s ± 1100.01s
2022-05-18T05:52:15.5026139Z     * bbaefdf6b5          0 reports
2022-05-18T05:52:15.5026382Z     * 7c52f204e0          0 reports
2022-05-18T05:52:15.5026640Z     * e0451d8022          0 reports
2022-05-18T05:52:15.5027019Z     * 4e2f5507d0          6 reports, total time  3460.20s ± 1126.45s
2022-05-18T05:52:15.5027430Z     * b64845eb18          6 reports, total time  3428.22s ± 1095.49s
2022-05-18T05:52:15.5027680Z     |
2022-05-18T05:52:15.5027885Z     :
2022-05-18T05:52:15.5028016Z 
2022-05-18T05:52:15.5028180Z Removed  (across  692 suites)     0 tests, totaling     0.00s
2022-05-18T05:52:15.5028509Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-05-18T05:52:15.5028863Z Added    (across   58 suites)   627 tests, totaling +3705.97s
2022-05-18T05:52:15.5579617Z Prepare all required actions
2022-05-18T05:52:15.5601754Z ##[group]Run ./.github/actions/teardown-linux
2022-05-18T05:52:15.5602034Z with:
2022-05-18T05:52:15.5602226Z env:
2022-05-18T05:52:15.5602439Z   IN_CI: 1
2022-05-18T05:52:15.5602657Z   IS_GHA: 1
2022-05-18T05:52:15.5602887Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:52:15.5603149Z   GPU_FLAG: --gpus all
2022-05-18T05:52:15.5603392Z ##[endgroup]
2022-05-18T05:52:15.5619950Z ##[group]Run .github/scripts/wait_for_ssh_to_drain.sh
2022-05-18T05:52:15.5620300Z .github/scripts/wait_for_ssh_to_drain.sh
2022-05-18T05:52:15.5633791Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T05:52:15.5634091Z env:
2022-05-18T05:52:15.5634310Z   IN_CI: 1
2022-05-18T05:52:15.5634514Z   IS_GHA: 1
2022-05-18T05:52:15.5634761Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:52:15.5635028Z   GPU_FLAG: --gpus all
2022-05-18T05:52:15.5635256Z ##[endgroup]
2022-05-18T05:52:15.5679742Z Holding runner for 2 hours until all ssh sessions have logged out
2022-05-18T05:52:15.5762378Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-05-18T05:52:15.5762800Z # ignore expansion of "docker ps -q" since it could be empty
2022-05-18T05:52:15.5763136Z # shellcheck disable=SC2046
2022-05-18T05:52:15.5763428Z docker stop $(docker ps -q) || true
2022-05-18T05:52:15.5763733Z # Prune all of the docker images
2022-05-18T05:52:15.5764026Z docker system prune -af
2022-05-18T05:52:15.5776759Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T05:52:15.5777056Z env:
2022-05-18T05:52:15.5777271Z   IN_CI: 1
2022-05-18T05:52:15.5777472Z   IS_GHA: 1
2022-05-18T05:52:15.5777716Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:52:15.5777977Z   GPU_FLAG: --gpus all
2022-05-18T05:52:15.5778205Z ##[endgroup]
2022-05-18T05:52:16.3856846Z e9eae2ba3bb0
2022-05-18T05:52:16.8947812Z Deleted Containers:
2022-05-18T05:52:16.8948267Z e9eae2ba3bb0fe475a85ca8ae7d95bec67fc2d17586595b7ebbece063df7a74f
2022-05-18T05:52:16.8948547Z 
2022-05-18T05:52:21.3705038Z Deleted Images:
2022-05-18T05:52:21.3705928Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T05:52:21.3706927Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7@sha256:66b56fbc2d0d8bf75af01c4976aba15f28c9802507dc01f27e71a55f8ffc13e0
2022-05-18T05:52:21.3707595Z deleted: sha256:236fb78bf4f994bc51deea39a9f1233f16926c9987665659eb61a9b813bac802
2022-05-18T05:52:21.3708029Z deleted: sha256:b9698d2c371e7954d7e15ccafd1fef13432b7e600eae532a3030f303baec803d
2022-05-18T05:52:21.3708480Z deleted: sha256:aa93f12269ac15489a70d1942bf2ab9744530e9058a09fd7d710c3eea67feb3f
2022-05-18T05:52:21.3708912Z deleted: sha256:b9bc18d2b222e195933fb238ac84e3e03734a45e81528f1753ddff97b9c72a43
2022-05-18T05:52:21.3709352Z deleted: sha256:80ca15a8657ced4067141453e7480c432fcdca438f0497640923838dea73ce43
2022-05-18T05:52:21.3709760Z deleted: sha256:f8866d582b5e8f7e1500d74db64487ea732384350859643a9b2abebb4573e705
2022-05-18T05:52:21.3710180Z deleted: sha256:32eb71a8af1989f422d1ef4e96915a0c579851537919e8b5dcdd98a59eab5b10
2022-05-18T05:52:21.3710632Z deleted: sha256:b93ecf6274d2fb0d04d3f59b9c1fbf6bfdf45a37acc7c109a522f192740766dd
2022-05-18T05:52:21.3711072Z deleted: sha256:8a6e85c33391306e1c7f415daeecf5998a12666c9afcfbeff15218240d0b53f3
2022-05-18T05:52:21.3711553Z deleted: sha256:991307397ef5ab4552d4a5f9293bd89db5bf7682892c0fdcea9bdfec12cd577e
2022-05-18T05:52:21.3711992Z deleted: sha256:1fa50cb91821cfcda4c803fdac009cc768715050e77c6b1266a3413a72f1d649
2022-05-18T05:52:21.3712445Z deleted: sha256:1a97642bd09c887f28cafcf79d1b815c3aa17a2bafacfc0bd7e934c709804b24
2022-05-18T05:52:21.3712893Z deleted: sha256:f3e369c2c977f35de33f7cbe9dbebf1bde8f63488c24b1a916883c49e512b3dd
2022-05-18T05:52:21.3713340Z deleted: sha256:2af47fb560e6db2934a930d03dc373cd29eaf5567afc06954d03b685020af41b
2022-05-18T05:52:21.3713982Z deleted: sha256:f906967fe7d2f8e2cd2fc3856d68bb8b1ae478cb816465468433f3eb48776dab
2022-05-18T05:52:21.3714419Z deleted: sha256:62b1fc0017c5b9541319bb613a6635ef68c0bee21ea46b358cfd629f18902d18
2022-05-18T05:52:21.3714849Z deleted: sha256:d58a6e6766e5fc8073d498afe668a52137146de79e2be473de919412cfb0c5c5
2022-05-18T05:52:21.3715268Z deleted: sha256:61347a34ee15c4d9cca6d181c32640066ba3456f9b9eb00843c55da3108fdc53
2022-05-18T05:52:21.3715690Z deleted: sha256:b0368f26a63aa7f28f7099b397d015f17d1c8785aec8dc11f315b7911912588d
2022-05-18T05:52:21.3716109Z deleted: sha256:6c5d0b2c8ac319699b8d86fc3f073fd7edb990189995fa9b35abece668fbb2d2
2022-05-18T05:52:21.3716553Z deleted: sha256:7eef87c7b27adca5d7956bdc256c2b9c386896dd2f9142cb2c65601315b885a4
2022-05-18T05:52:21.3716992Z deleted: sha256:bad3c554c61dc113ef9996c78f018493bef1c51b614c489258964beb48a212d1
2022-05-18T05:52:21.3717406Z deleted: sha256:4f80a663751e4147a2bcbb9fdaa829c498eb6c18f8e8218502d5a2f868b75d73
2022-05-18T05:52:21.3717878Z deleted: sha256:98ed9491a363c4fd7ebeaa38caf6c4216ad5032d9415ce7abdc39b6149a5ce4c
2022-05-18T05:52:21.3718439Z deleted: sha256:4c71586cb548b8efa3884f847e3dfffb6514648d8131654de3ae8c3165a935c0
2022-05-18T05:52:21.3718857Z deleted: sha256:3365026f6ab431e63e4c2b8b3b4ea77640968a1031c71e229be682a7e6865992
2022-05-18T05:52:21.3719279Z deleted: sha256:6f9d8a5abe7610cc746fabddd5fe5de01ac404017fc5a4f3009a83caec28f771
2022-05-18T05:52:21.3719715Z deleted: sha256:943055dd5d358663a978746c9e41e2c2053d132f045219c20a164176f6228da4
2022-05-18T05:52:21.3720130Z deleted: sha256:675297863e1d26af6fd4cd4550d3aadf811d9e31d4e2f53939ac0de9d13a6447
2022-05-18T05:52:21.3720562Z deleted: sha256:535e5f66a4c0f3149aa7327c8fd55b2a4aab80016c4639571c5268cbbe006ef6
2022-05-18T05:52:21.3720980Z deleted: sha256:4ba20979a69fdebf50a0fcea34ae16be95d9556b6b2e9656387d450204a32739
2022-05-18T05:52:21.3721412Z deleted: sha256:d538786f7a446ecb0b2d077dae03561ad192e00337811bb1a23286d2ff720889
2022-05-18T05:52:21.3721833Z deleted: sha256:3691b15bb27dde54c31442622d1c70d6c31b8fc91576c15a4d031a7390b7c9d7
2022-05-18T05:52:21.3722255Z deleted: sha256:51f052fa0c9fef786ecdeca36b244e1d756793d5ac3c9eab25183237ccad7c44
2022-05-18T05:52:21.3722704Z deleted: sha256:29d62475fed7ed1c35cef64d114cf77e111f7f06ac5f861d3e15ab1423ba403c
2022-05-18T05:52:21.3723139Z deleted: sha256:e64ff51bd9cf74d567736df27f09a71215054d7a6fd387fb8a1525863bea2965
2022-05-18T05:52:21.3723568Z deleted: sha256:8ce2ebc2b9122185e0bf1b7079903ad8839f5dd8b74b422833ed4f32a21786ec
2022-05-18T05:52:21.3723979Z deleted: sha256:c1f0e89774fc81730998b64c9e2cd56d0e5fd033b100a0f74a1b9bddca647997
2022-05-18T05:52:21.3724399Z deleted: sha256:045405eee158f603e8d2840c5696f23bff982b34dea8ce059497806acacc6891
2022-05-18T05:52:21.3724811Z deleted: sha256:ca4847070736f4c4ca7e5075c715e4845d0a30a41aab34f473e0753094e5ebf0
2022-05-18T05:52:21.3725225Z deleted: sha256:89767b0be2e7dda030b2a7bbd1df9bd63bbd13e0737b8b5ee3b0643897b36459
2022-05-18T05:52:21.3725662Z deleted: sha256:739ba1ab17ff0d1e81e1ac36c75472c62975342fa8fe8993206dd31f5780e105
2022-05-18T05:52:21.3726106Z deleted: sha256:da279ce0cf78b3443ee69f3308ecdcfa27525db93b9243e7f7c01ee80da21bd5
2022-05-18T05:52:21.3726545Z deleted: sha256:88f543990c97cd012b9e17b81fd42bff0dff5a06c70187fbc19b1860a6604b96
2022-05-18T05:52:21.3726962Z deleted: sha256:d4c156eabe2ffb174bb8b81474a3551cc41b23647c1448f33a07162a05bcb6d1
2022-05-18T05:52:21.3727388Z deleted: sha256:3823e0dd401f484d0f5471862f350ef88ffd14ecb5f1bd7329f4b6902192a905
2022-05-18T05:52:21.3727811Z deleted: sha256:1d87640a243e42970325889dd0d6ca21c6fc3c50efac95d88624ad1463d2f9a0
2022-05-18T05:52:21.3728207Z deleted: sha256:7423922c27fd43adc890485039d039307be042bd004ce39a462d7f8ee969125b
2022-05-18T05:52:21.3728601Z deleted: sha256:18b3010e02831349f67561e26c28fbace9501706ea0780b77339475581c2e40e
2022-05-18T05:52:21.3728997Z deleted: sha256:0214f4b057d78b44fd12702828152f67c0ce115f9346acc63acdf997cab7e7c8
2022-05-18T05:52:21.3729404Z deleted: sha256:1b9d0485372c5562fa614d5b35766f6c442539bcee9825a6e90d1158c3299a61
2022-05-18T05:52:21.3730238Z deleted: sha256:3c0f34be6eb98057c607b9080237cce0be0b86f52d51ba620dc018a3d421baea
2022-05-18T05:52:21.3730693Z deleted: sha256:be96a3f634de79f523f07c7e4e0216c28af45eb5776e7a6238a2392f71e01069
2022-05-18T05:52:21.3730938Z 
2022-05-18T05:52:21.3860805Z Total reclaimed space: 15.91GB
2022-05-18T05:52:21.3930562Z Post job cleanup.
2022-05-18T05:52:21.3966497Z Post job cleanup.
2022-05-18T05:52:21.5289147Z [command]/usr/bin/git version
2022-05-18T05:52:21.5338455Z git version 2.32.0
2022-05-18T05:52:21.5402822Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/b35ed4a8-49ce-4104-927b-38ee3137ed2a' before making global git config changes
2022-05-18T05:52:21.5403584Z Adding repository directory to the temporary git global config as a safe directory
2022-05-18T05:52:21.5413262Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T05:52:21.5459768Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-05-18T05:52:21.5499598Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-05-18T05:52:21.5828140Z Entering 'android/libs/fbjni'
2022-05-18T05:52:21.5869165Z Entering 'third_party/FP16'
2022-05-18T05:52:21.5911474Z Entering 'third_party/FXdiv'
2022-05-18T05:52:21.5950958Z Entering 'third_party/NNPACK'
2022-05-18T05:52:21.5991482Z Entering 'third_party/QNNPACK'
2022-05-18T05:52:21.6032799Z Entering 'third_party/XNNPACK'
2022-05-18T05:52:21.6084413Z Entering 'third_party/benchmark'
2022-05-18T05:52:21.6126053Z Entering 'third_party/cpuinfo'
2022-05-18T05:52:21.6167441Z Entering 'third_party/cub'
2022-05-18T05:52:21.6207881Z Entering 'third_party/cudnn_frontend'
2022-05-18T05:52:21.6256030Z Entering 'third_party/eigen'
2022-05-18T05:52:21.6299938Z Entering 'third_party/fbgemm'
2022-05-18T05:52:21.6340581Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T05:52:21.6382051Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T05:52:21.6422919Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T05:52:21.6466597Z Entering 'third_party/flatbuffers'
2022-05-18T05:52:21.6511331Z Entering 'third_party/fmt'
2022-05-18T05:52:21.6553071Z Entering 'third_party/foxi'
2022-05-18T05:52:21.6594459Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T05:52:21.6635765Z Entering 'third_party/gloo'
2022-05-18T05:52:21.6676986Z Entering 'third_party/googletest'
2022-05-18T05:52:21.6720440Z Entering 'third_party/ideep'
2022-05-18T05:52:21.6760204Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T05:52:21.6803542Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T05:52:21.6851809Z Entering 'third_party/ios-cmake'
2022-05-18T05:52:21.6894539Z Entering 'third_party/kineto'
2022-05-18T05:52:21.6935770Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T05:52:21.6976483Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T05:52:21.7019819Z Entering 'third_party/nccl/nccl'
2022-05-18T05:52:21.7060791Z Entering 'third_party/neon2sse'
2022-05-18T05:52:21.7100923Z Entering 'third_party/onnx'
2022-05-18T05:52:21.7154875Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T05:52:21.7196233Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T05:52:21.7238862Z Entering 'third_party/onnx-tensorrt'
2022-05-18T05:52:21.7278873Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T05:52:21.7325437Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T05:52:21.7367947Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T05:52:21.7409370Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T05:52:21.7456075Z Entering 'third_party/pocketfft'
2022-05-18T05:52:21.7497019Z Entering 'third_party/protobuf'
2022-05-18T05:52:21.7543419Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T05:52:21.7583971Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T05:52:21.7627728Z Entering 'third_party/psimd'
2022-05-18T05:52:21.7668560Z Entering 'third_party/pthreadpool'
2022-05-18T05:52:21.7711168Z Entering 'third_party/pybind11'
2022-05-18T05:52:21.7752362Z Entering 'third_party/python-enum'
2022-05-18T05:52:21.7793801Z Entering 'third_party/python-peachpy'
2022-05-18T05:52:21.7835238Z Entering 'third_party/python-six'
2022-05-18T05:52:21.7876285Z Entering 'third_party/sleef'
2022-05-18T05:52:21.7918222Z Entering 'third_party/tbb'
2022-05-18T05:52:21.7961093Z Entering 'third_party/tensorpipe'
2022-05-18T05:52:21.8002319Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T05:52:21.8044498Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T05:52:21.8085416Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T05:52:21.8126492Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T05:52:21.8166785Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T05:52:21.8211007Z Entering 'third_party/zstd'
2022-05-18T05:52:21.8271780Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-05-18T05:52:21.8301356Z http.https://github.com/.extraheader
2022-05-18T05:52:21.8312830Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-05-18T05:52:21.8353242Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-05-18T05:52:21.8672544Z Entering 'android/libs/fbjni'
2022-05-18T05:52:21.8697662Z http.https://github.com/.extraheader
2022-05-18T05:52:21.8729514Z Entering 'third_party/FP16'
2022-05-18T05:52:21.8755369Z http.https://github.com/.extraheader
2022-05-18T05:52:21.8787230Z Entering 'third_party/FXdiv'
2022-05-18T05:52:21.8811569Z http.https://github.com/.extraheader
2022-05-18T05:52:21.8843580Z Entering 'third_party/NNPACK'
2022-05-18T05:52:21.8868088Z http.https://github.com/.extraheader
2022-05-18T05:52:21.8899863Z Entering 'third_party/QNNPACK'
2022-05-18T05:52:21.8923746Z http.https://github.com/.extraheader
2022-05-18T05:52:21.8955711Z Entering 'third_party/XNNPACK'
2022-05-18T05:52:21.8979877Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9023922Z Entering 'third_party/benchmark'
2022-05-18T05:52:21.9047869Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9079254Z Entering 'third_party/cpuinfo'
2022-05-18T05:52:21.9104025Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9136860Z Entering 'third_party/cub'
2022-05-18T05:52:21.9161409Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9193373Z Entering 'third_party/cudnn_frontend'
2022-05-18T05:52:21.9218775Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9257424Z Entering 'third_party/eigen'
2022-05-18T05:52:21.9281205Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9315756Z Entering 'third_party/fbgemm'
2022-05-18T05:52:21.9340413Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9372490Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T05:52:21.9396427Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9429874Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T05:52:21.9453886Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9485339Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T05:52:21.9509537Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9543064Z Entering 'third_party/flatbuffers'
2022-05-18T05:52:21.9566834Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9600443Z Entering 'third_party/fmt'
2022-05-18T05:52:21.9625171Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9656814Z Entering 'third_party/foxi'
2022-05-18T05:52:21.9680506Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9713284Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T05:52:21.9737721Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9769337Z Entering 'third_party/gloo'
2022-05-18T05:52:21.9794395Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9826075Z Entering 'third_party/googletest'
2022-05-18T05:52:21.9849790Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9881603Z Entering 'third_party/ideep'
2022-05-18T05:52:21.9906435Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9937074Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T05:52:21.9960343Z http.https://github.com/.extraheader
2022-05-18T05:52:21.9993876Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T05:52:22.0018246Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0057039Z Entering 'third_party/ios-cmake'
2022-05-18T05:52:22.0080982Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0112076Z Entering 'third_party/kineto'
2022-05-18T05:52:22.0137485Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0169182Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T05:52:22.0192980Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0225071Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T05:52:22.0248548Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0281459Z Entering 'third_party/nccl/nccl'
2022-05-18T05:52:22.0306481Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0338436Z Entering 'third_party/neon2sse'
2022-05-18T05:52:22.0362159Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0393060Z Entering 'third_party/onnx'
2022-05-18T05:52:22.0417961Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0461876Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T05:52:22.0485808Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0518679Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T05:52:22.0544216Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0579199Z Entering 'third_party/onnx-tensorrt'
2022-05-18T05:52:22.0603141Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0633727Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T05:52:22.0659271Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0695817Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T05:52:22.0719449Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0751742Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T05:52:22.0776650Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0808184Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T05:52:22.0832211Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0868090Z Entering 'third_party/pocketfft'
2022-05-18T05:52:22.0893854Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0924377Z Entering 'third_party/protobuf'
2022-05-18T05:52:22.0949873Z http.https://github.com/.extraheader
2022-05-18T05:52:22.0984952Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T05:52:22.1008139Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1040749Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T05:52:22.1065727Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1100568Z Entering 'third_party/psimd'
2022-05-18T05:52:22.1125889Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1157960Z Entering 'third_party/pthreadpool'
2022-05-18T05:52:22.1183219Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1215073Z Entering 'third_party/pybind11'
2022-05-18T05:52:22.1238641Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1270665Z Entering 'third_party/python-enum'
2022-05-18T05:52:22.1296506Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1327271Z Entering 'third_party/python-peachpy'
2022-05-18T05:52:22.1351809Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1384049Z Entering 'third_party/python-six'
2022-05-18T05:52:22.1407501Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1438542Z Entering 'third_party/sleef'
2022-05-18T05:52:22.1463385Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1495300Z Entering 'third_party/tbb'
2022-05-18T05:52:22.1519036Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1553382Z Entering 'third_party/tensorpipe'
2022-05-18T05:52:22.1577998Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1608896Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T05:52:22.1632919Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1664608Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T05:52:22.1688000Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1719753Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T05:52:22.1744471Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1776190Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T05:52:22.1799649Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1831009Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T05:52:22.1854890Z http.https://github.com/.extraheader
2022-05-18T05:52:22.1889372Z Entering 'third_party/zstd'
2022-05-18T05:52:22.1913945Z http.https://github.com/.extraheader
2022-05-18T05:52:22.2222011Z Cleaning up orphan processes