2022-05-18T03:57:55.3084958Z Requested labels: linux.8xlarge.nvidia.gpu
2022-05-18T03:57:55.3085069Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master
2022-05-18T03:57:55.3085094Z Waiting for a runner to pick up this job...
2022-05-18T03:57:58.3848113Z Job is about to start running on the runner: i-08f720eed7f2b488f (repository)
2022-05-18T03:58:04.4664452Z Current runner version: '2.291.1'
2022-05-18T03:58:04.4672092Z Runner name: 'i-08f720eed7f2b488f'
2022-05-18T03:58:04.4672913Z Runner group name: 'Default'
2022-05-18T03:58:04.4673606Z Machine name: 'ip-10-0-4-36'
2022-05-18T03:58:04.4676301Z ##[group]GITHUB_TOKEN Permissions
2022-05-18T03:58:04.4677155Z Actions: write
2022-05-18T03:58:04.4677606Z Checks: write
2022-05-18T03:58:04.4677959Z Contents: write
2022-05-18T03:58:04.4678834Z Deployments: write
2022-05-18T03:58:04.4679251Z Discussions: write
2022-05-18T03:58:04.4679587Z Issues: write
2022-05-18T03:58:04.4679964Z Metadata: read
2022-05-18T03:58:04.4680402Z Packages: write
2022-05-18T03:58:04.4680749Z Pages: write
2022-05-18T03:58:04.4681155Z PullRequests: write
2022-05-18T03:58:04.4681604Z RepositoryProjects: write
2022-05-18T03:58:04.4682085Z SecurityEvents: write
2022-05-18T03:58:04.4682470Z Statuses: write
2022-05-18T03:58:04.4682876Z ##[endgroup]
2022-05-18T03:58:04.4687380Z Secret source: Actions
2022-05-18T03:58:04.4688129Z Prepare workflow directory
2022-05-18T03:58:04.7645639Z Prepare all required actions
2022-05-18T03:58:04.7874671Z Getting action download info
2022-05-18T03:58:04.9781326Z Download action repository 'pytorch/pytorch@master' (SHA:7b8cf1f7366bff95e9954037a58a8bb0edaaebd3)
2022-05-18T03:58:08.0328000Z Download action repository 'nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a' (SHA:71062288b76e2b6214ebde0e673ce0de1755740a)
2022-05-18T03:58:08.1442520Z Download action repository 'seemethere/upload-artifact-s3@v4' (SHA:c1c31f57581a11fe6d4d052da6276adb2df71f1e)
2022-05-18T03:58:08.4249243Z Getting action download info
2022-05-18T03:58:08.5950093Z Download action repository 'malfet/checkout@silent-checkout' (SHA:f63e9e15406be6060f159846cd2e098f759c5246)
2022-05-18T03:58:08.7905230Z Getting action download info
2022-05-18T03:58:09.0479031Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-05-18T03:58:09.0479432Z with:
2022-05-18T03:58:09.0479690Z   submodules: recursive
2022-05-18T03:58:09.0479935Z   fetch-depth: 0
2022-05-18T03:58:09.0480196Z env:
2022-05-18T03:58:09.0480418Z   IN_CI: 1
2022-05-18T03:58:09.0480628Z   IS_GHA: 1
2022-05-18T03:58:09.0480879Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:58:09.0481164Z ##[endgroup]
2022-05-18T03:58:09.0772492Z ##[group]Run echo "${GITHUB_WORKSPACE}"
2022-05-18T03:58:09.0772864Z echo "${GITHUB_WORKSPACE}"
2022-05-18T03:58:09.0773179Z if [ -z "${NO_SUDO}" ]; then
2022-05-18T03:58:09.0773485Z   sudo rm -rf "${GITHUB_WORKSPACE}"
2022-05-18T03:58:09.0773746Z else
2022-05-18T03:58:09.0774013Z   rm -rf "${GITHUB_WORKSPACE}"
2022-05-18T03:58:09.0774279Z fi
2022-05-18T03:58:09.0774523Z mkdir "${GITHUB_WORKSPACE}"
2022-05-18T03:58:09.0792666Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:58:09.0792995Z env:
2022-05-18T03:58:09.0793205Z   IN_CI: 1
2022-05-18T03:58:09.0793435Z   IS_GHA: 1
2022-05-18T03:58:09.0793685Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:58:09.0793929Z   NO_SUDO: 
2022-05-18T03:58:09.0794167Z ##[endgroup]
2022-05-18T03:58:09.1015390Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T03:58:11.5718020Z ##[group]Run malfet/checkout@silent-checkout
2022-05-18T03:58:11.5718676Z with:
2022-05-18T03:58:11.5718952Z   ref: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T03:58:11.5719241Z   fetch-depth: 0
2022-05-18T03:58:11.5719490Z   submodules: recursive
2022-05-18T03:58:11.5719754Z   quiet-checkout: true
2022-05-18T03:58:11.5720035Z   repository: pytorch/pytorch
2022-05-18T03:58:11.5720476Z   token: ***
2022-05-18T03:58:11.5720723Z   ssh-strict: true
2022-05-18T03:58:11.5720998Z   persist-credentials: true
2022-05-18T03:58:11.5721250Z   clean: true
2022-05-18T03:58:11.5721480Z   lfs: false
2022-05-18T03:58:11.5721753Z   set-safe-directory: true
2022-05-18T03:58:11.5721993Z env:
2022-05-18T03:58:11.5722208Z   IN_CI: 1
2022-05-18T03:58:11.5722432Z   IS_GHA: 1
2022-05-18T03:58:11.5722663Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:58:11.5722927Z ##[endgroup]
2022-05-18T03:58:11.7255116Z Syncing repository: pytorch/pytorch
2022-05-18T03:58:11.7256959Z ##[group]Getting Git version info
2022-05-18T03:58:11.7257488Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-05-18T03:58:11.7258094Z [command]/usr/bin/git version
2022-05-18T03:58:11.7258366Z git version 2.32.0
2022-05-18T03:58:11.7270163Z ##[endgroup]
2022-05-18T03:58:11.7292330Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/3055aad0-3f03-41d6-a922-af5f70d2596c' before making global git config changes
2022-05-18T03:58:11.7292886Z Adding repository directory to the temporary git global config as a safe directory
2022-05-18T03:58:11.7301286Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T03:58:11.7344793Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-05-18T03:58:11.7350536Z ##[group]Initializing the repository
2022-05-18T03:58:11.7356881Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T03:58:11.7389759Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-05-18T03:58:11.7390214Z hint: is subject to change. To configure the initial branch name to use in all
2022-05-18T03:58:11.7390635Z hint: of your new repositories, which will suppress this warning, call:
2022-05-18T03:58:11.7391100Z hint: 
2022-05-18T03:58:11.7391485Z hint: 	git config --global init.defaultBranch <name>
2022-05-18T03:58:11.7391774Z hint: 
2022-05-18T03:58:11.7392142Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-05-18T03:58:11.7392635Z hint: 'development'. The just-created branch can be renamed via this command:
2022-05-18T03:58:11.7392954Z hint: 
2022-05-18T03:58:11.7393397Z hint: 	git branch -m <name>
2022-05-18T03:58:11.7393905Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-05-18T03:58:11.7404997Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-05-18T03:58:11.7441636Z ##[endgroup]
2022-05-18T03:58:11.7442120Z ##[group]Disabling automatic garbage collection
2022-05-18T03:58:11.7447869Z [command]/usr/bin/git config --local gc.auto 0
2022-05-18T03:58:11.7480257Z ##[endgroup]
2022-05-18T03:58:11.7480706Z ##[group]Setting up auth
2022-05-18T03:58:11.7490957Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-05-18T03:58:11.7529174Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-05-18T03:58:11.7838802Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-05-18T03:58:11.7871255Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-05-18T03:58:11.8181937Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-05-18T03:58:11.8228475Z ##[endgroup]
2022-05-18T03:58:11.8228961Z ##[group]Fetching the repository
2022-05-18T03:58:11.8237393Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-05-18T03:58:52.6672086Z [command]/usr/bin/git rev-parse --verify --quiet 3b2375291aab7b48442f2e6fb1ef66cebc761e24^{object}
2022-05-18T03:58:52.6701975Z 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T03:58:52.6709583Z ##[endgroup]
2022-05-18T03:58:52.6710092Z ##[group]Determining the checkout info
2022-05-18T03:58:52.6710569Z ##[endgroup]
2022-05-18T03:58:52.6711025Z ##[group]Checking out the ref
2022-05-18T03:58:52.6716227Z [command]/usr/bin/git checkout --quiet --force 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T03:58:54.2494869Z ##[endgroup]
2022-05-18T03:58:54.2495621Z ##[group]Setting up auth for fetching submodules
2022-05-18T03:58:54.2502815Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-05-18T03:58:54.2561788Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-05-18T03:58:54.2596460Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-05-18T03:58:54.2630682Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-05-18T03:58:54.2662541Z ##[endgroup]
2022-05-18T03:58:54.2662978Z ##[group]Fetching submodules
2022-05-18T03:58:54.2669384Z [command]/usr/bin/git submodule sync --recursive
2022-05-18T03:58:54.3003481Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-05-18T03:58:54.3319798Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-05-18T03:58:54.3323025Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-05-18T03:58:54.3326034Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-05-18T03:58:54.3329215Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-05-18T03:58:54.3332537Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-05-18T03:58:54.3336019Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-05-18T03:58:54.3339817Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-05-18T03:58:54.3343473Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-05-18T03:58:54.3347266Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-05-18T03:58:54.3351458Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-05-18T03:58:54.3355424Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-05-18T03:58:54.3360102Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-05-18T03:58:54.3364431Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-05-18T03:58:54.3369062Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-05-18T03:58:54.3373508Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-05-18T03:58:54.3378369Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-05-18T03:58:54.3383181Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-05-18T03:58:54.3388195Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-05-18T03:58:54.3393269Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-05-18T03:58:54.3398813Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-05-18T03:58:54.3404542Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-05-18T03:58:54.3410018Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-05-18T03:58:54.3415741Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-05-18T03:58:54.3421551Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-05-18T03:58:54.3427503Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-05-18T03:58:54.3433559Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-05-18T03:58:54.3440280Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-05-18T03:58:54.3446569Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-05-18T03:58:54.3453055Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-05-18T03:58:54.3459602Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-05-18T03:58:54.3466342Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-05-18T03:58:54.3473205Z Submodule 'third_party/python-peachpy' (https://github.com/Maratyszcza/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-05-18T03:58:54.3480572Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-05-18T03:58:54.3487628Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-05-18T03:58:54.3494964Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-05-18T03:58:54.3502331Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-05-18T03:58:54.3510028Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-05-18T03:58:54.3572252Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-05-18T03:58:54.6084535Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-05-18T03:58:54.8995527Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-05-18T03:58:55.0789957Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-05-18T03:58:55.3260255Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-05-18T03:58:55.5640541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-05-18T03:58:59.0434959Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-05-18T03:58:59.4691526Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-05-18T03:58:59.9315150Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-05-18T03:59:01.0957456Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-05-18T03:59:02.2951602Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-05-18T03:59:06.9143426Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-05-18T03:59:07.3809730Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-05-18T03:59:08.2797581Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-05-18T03:59:09.1893378Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-05-18T03:59:09.3756047Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-05-18T03:59:09.7892344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-05-18T03:59:10.0612018Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-05-18T03:59:10.8543305Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-05-18T03:59:11.1916066Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-05-18T03:59:11.3803810Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-05-18T03:59:12.9134882Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-05-18T03:59:13.2804115Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-05-18T03:59:13.7048493Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-05-18T03:59:14.8138692Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-05-18T03:59:15.1544896Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-05-18T03:59:15.3606911Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-05-18T03:59:19.6625472Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-05-18T03:59:19.8502244Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-05-18T03:59:20.0530694Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-05-18T03:59:20.7612228Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-05-18T03:59:20.9608379Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-05-18T03:59:21.2375782Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-05-18T03:59:21.4959983Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-05-18T03:59:22.0025912Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-05-18T03:59:23.7506022Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-05-18T03:59:24.1585514Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-05-18T03:59:25.7155000Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-05-18T03:59:25.7550880Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-05-18T03:59:25.7916139Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-05-18T03:59:25.8464877Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-05-18T03:59:25.9005480Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-05-18T03:59:26.7030130Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-05-18T03:59:26.7562256Z Submodule path 'third_party/benchmark': checked out 'e991355c02b93fe17713efe04cbc2e278e00fdbd'
2022-05-18T03:59:26.9054702Z Submodule path 'third_party/cpuinfo': checked out '5916273f79a21551890fd3d56fc5375a78d1598d'
2022-05-18T03:59:26.9760088Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-05-18T03:59:27.3581257Z Submodule path 'third_party/cudnn_frontend': checked out '43709ab96c47e26eebcdac72f93f946d44ceffa8'
2022-05-18T03:59:27.6798483Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-05-18T03:59:27.7641805Z Submodule path 'third_party/fbgemm': checked out '2e9be65810107a9595da717f95d21924b73be833'
2022-05-18T03:59:27.7694229Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-05-18T03:59:27.7697443Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T03:59:27.7700777Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-05-18T03:59:27.7747070Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-05-18T03:59:28.6567177Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-05-18T03:59:29.0922600Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-05-18T03:59:29.9416529Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out '8b35b4cffb62ecb58a903bf91cb7537d7a672211'
2022-05-18T03:59:30.0923743Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-05-18T03:59:30.1898948Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-05-18T03:59:30.3219829Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-05-18T03:59:30.3894023Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2022-05-18T03:59:30.4258113Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-05-18T03:59:30.4993542Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-05-18T03:59:30.5549706Z Submodule path 'third_party/gloo': checked out 'c22a5cfba94edf8ea4f53a174d38aa0c629d070f'
2022-05-18T03:59:30.6379854Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-05-18T03:59:30.6771990Z Submodule path 'third_party/ideep': checked out '02b17c5748c9349dcc586c359af800c684d9b1ab'
2022-05-18T03:59:30.6822267Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-05-18T03:59:30.6866596Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-05-18T03:59:35.6188770Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46'
2022-05-18T03:59:35.6254809Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T03:59:35.6303952Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-05-18T03:59:40.6408846Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815'
2022-05-18T03:59:40.6822400Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-05-18T03:59:40.8208930Z Submodule path 'third_party/kineto': checked out 'b2b48c00c6e5bd8e807e2231adb229db6a1d1c22'
2022-05-18T03:59:40.8261337Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T03:59:40.8264369Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T03:59:40.8310466Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-05-18T03:59:41.7457996Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-05-18T03:59:42.6142272Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-05-18T03:59:42.7043917Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-05-18T03:59:42.7551738Z Submodule path 'third_party/nccl/nccl': checked out '7e515921295adaab72adf56ea71a0fafb0ecb5f3'
2022-05-18T03:59:42.7972544Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-05-18T03:59:43.1207284Z Submodule path 'third_party/onnx': checked out '96046b8ccfb8e6fa82f6b2b34b3d56add2e8849c'
2022-05-18T03:59:43.1274671Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-05-18T03:59:43.1277935Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-05-18T03:59:43.1337881Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-05-18T03:59:43.5099900Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-05-18T03:59:44.1797387Z Submodule path 'third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-05-18T03:59:44.2435873Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '59a2ac2745d8a57ac94c6accced73620d59fb844'
2022-05-18T03:59:44.2880770Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-05-18T03:59:44.2931552Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T03:59:44.2975331Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-05-18T03:59:45.5913911Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-05-18T03:59:45.5981322Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T03:59:45.5985395Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T03:59:45.6038018Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-05-18T03:59:45.9693083Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-05-18T03:59:46.6413631Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-05-18T03:59:46.7439603Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-05-18T03:59:46.7499066Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T03:59:46.7544717Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-05-18T03:59:46.9903659Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-05-18T03:59:47.0293363Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-05-18T03:59:47.3732203Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-05-18T03:59:47.3782188Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-05-18T03:59:47.3785212Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-05-18T03:59:47.3836387Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-05-18T03:59:47.7257190Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-05-18T03:59:48.5248803Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-05-18T03:59:48.6329001Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-05-18T03:59:48.6703602Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-05-18T03:59:48.7089803Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-05-18T03:59:48.7711120Z Submodule path 'third_party/pybind11': checked out '8de7772cc72daca8e947b79b83fea46214931604'
2022-05-18T03:59:48.8070470Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-05-18T03:59:48.8680207Z Submodule path 'third_party/python-peachpy': checked out '07d8fde8ac45d7705129475c0f94ed8925b93473'
2022-05-18T03:59:48.9049922Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-05-18T03:59:48.9845056Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-05-18T03:59:49.1490583Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-05-18T03:59:49.2080481Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-05-18T03:59:49.2131112Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-05-18T03:59:49.2134310Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-05-18T03:59:49.2137277Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-05-18T03:59:49.2140479Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T03:59:49.2186837Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-05-18T03:59:50.0068456Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-05-18T03:59:50.2419016Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-05-18T03:59:51.2387678Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-05-18T03:59:51.9759188Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-05-18T03:59:52.0210394Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-05-18T03:59:52.1288488Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-05-18T03:59:52.1912579Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-05-18T03:59:52.1971804Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T03:59:52.2015755Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-05-18T03:59:52.4379285Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-05-18T03:59:52.6218757Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-05-18T03:59:52.6310814Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-05-18T03:59:52.6637103Z Entering 'android/libs/fbjni'
2022-05-18T03:59:52.6681850Z Entering 'third_party/FP16'
2022-05-18T03:59:52.6724687Z Entering 'third_party/FXdiv'
2022-05-18T03:59:52.6766215Z Entering 'third_party/NNPACK'
2022-05-18T03:59:52.6808684Z Entering 'third_party/QNNPACK'
2022-05-18T03:59:52.6851460Z Entering 'third_party/XNNPACK'
2022-05-18T03:59:52.6906235Z Entering 'third_party/benchmark'
2022-05-18T03:59:52.6948422Z Entering 'third_party/cpuinfo'
2022-05-18T03:59:52.6991115Z Entering 'third_party/cub'
2022-05-18T03:59:52.7033673Z Entering 'third_party/cudnn_frontend'
2022-05-18T03:59:52.7081826Z Entering 'third_party/eigen'
2022-05-18T03:59:52.7126492Z Entering 'third_party/fbgemm'
2022-05-18T03:59:52.7168914Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T03:59:52.7210708Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T03:59:52.7253786Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T03:59:52.7296370Z Entering 'third_party/flatbuffers'
2022-05-18T03:59:52.7341312Z Entering 'third_party/fmt'
2022-05-18T03:59:52.7383193Z Entering 'third_party/foxi'
2022-05-18T03:59:52.7425691Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T03:59:52.7467944Z Entering 'third_party/gloo'
2022-05-18T03:59:52.7509559Z Entering 'third_party/googletest'
2022-05-18T03:59:52.7552648Z Entering 'third_party/ideep'
2022-05-18T03:59:52.7593692Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T03:59:52.7638031Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T03:59:52.7687529Z Entering 'third_party/ios-cmake'
2022-05-18T03:59:52.7729092Z Entering 'third_party/kineto'
2022-05-18T03:59:52.7773622Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T03:59:52.7815404Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T03:59:52.7858487Z Entering 'third_party/nccl/nccl'
2022-05-18T03:59:52.7900114Z Entering 'third_party/neon2sse'
2022-05-18T03:59:52.7941650Z Entering 'third_party/onnx'
2022-05-18T03:59:52.7996343Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T03:59:52.8038215Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T03:59:52.8083820Z Entering 'third_party/onnx-tensorrt'
2022-05-18T03:59:52.8125258Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T03:59:52.8172678Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T03:59:52.8214710Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T03:59:52.8256035Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T03:59:52.8302530Z Entering 'third_party/pocketfft'
2022-05-18T03:59:52.8343767Z Entering 'third_party/protobuf'
2022-05-18T03:59:52.8389066Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T03:59:52.8430297Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T03:59:52.8473511Z Entering 'third_party/psimd'
2022-05-18T03:59:52.8515078Z Entering 'third_party/pthreadpool'
2022-05-18T03:59:52.8557094Z Entering 'third_party/pybind11'
2022-05-18T03:59:52.8598920Z Entering 'third_party/python-enum'
2022-05-18T03:59:52.8639937Z Entering 'third_party/python-peachpy'
2022-05-18T03:59:52.8681231Z Entering 'third_party/python-six'
2022-05-18T03:59:52.8722497Z Entering 'third_party/sleef'
2022-05-18T03:59:52.8763685Z Entering 'third_party/tbb'
2022-05-18T03:59:52.8807683Z Entering 'third_party/tensorpipe'
2022-05-18T03:59:52.8850015Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T03:59:52.8891171Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T03:59:52.8932861Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T03:59:52.8975392Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T03:59:52.9016320Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T03:59:52.9060862Z Entering 'third_party/zstd'
2022-05-18T03:59:52.9114525Z ##[endgroup]
2022-05-18T03:59:52.9117713Z ##[group]Persisting credentials for submodules
2022-05-18T03:59:52.9124365Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-05-18T03:59:52.9444246Z Entering 'android/libs/fbjni'
2022-05-18T03:59:52.9485235Z Entering 'third_party/FP16'
2022-05-18T03:59:52.9526245Z Entering 'third_party/FXdiv'
2022-05-18T03:59:52.9567147Z Entering 'third_party/NNPACK'
2022-05-18T03:59:52.9609030Z Entering 'third_party/QNNPACK'
2022-05-18T03:59:52.9650204Z Entering 'third_party/XNNPACK'
2022-05-18T03:59:52.9702923Z Entering 'third_party/benchmark'
2022-05-18T03:59:52.9743595Z Entering 'third_party/cpuinfo'
2022-05-18T03:59:52.9785312Z Entering 'third_party/cub'
2022-05-18T03:59:52.9826909Z Entering 'third_party/cudnn_frontend'
2022-05-18T03:59:52.9873165Z Entering 'third_party/eigen'
2022-05-18T03:59:52.9916237Z Entering 'third_party/fbgemm'
2022-05-18T03:59:52.9956938Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T03:59:52.9998753Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T03:59:53.0039449Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T03:59:53.0083870Z Entering 'third_party/flatbuffers'
2022-05-18T03:59:53.0127035Z Entering 'third_party/fmt'
2022-05-18T03:59:53.0167920Z Entering 'third_party/foxi'
2022-05-18T03:59:53.0208485Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T03:59:53.0249816Z Entering 'third_party/gloo'
2022-05-18T03:59:53.0291188Z Entering 'third_party/googletest'
2022-05-18T03:59:53.0332135Z Entering 'third_party/ideep'
2022-05-18T03:59:53.0372058Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T03:59:53.0472342Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T03:59:53.0473295Z Entering 'third_party/ios-cmake'
2022-05-18T03:59:53.0503264Z Entering 'third_party/kineto'
2022-05-18T03:59:53.0544074Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T03:59:53.0585853Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T03:59:53.0628636Z Entering 'third_party/nccl/nccl'
2022-05-18T03:59:53.0670086Z Entering 'third_party/neon2sse'
2022-05-18T03:59:53.0710934Z Entering 'third_party/onnx'
2022-05-18T03:59:53.0763926Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T03:59:53.0804765Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T03:59:53.0847728Z Entering 'third_party/onnx-tensorrt'
2022-05-18T03:59:53.0887563Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T03:59:53.0932809Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T03:59:53.0976402Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T03:59:53.1016890Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T03:59:53.1062440Z Entering 'third_party/pocketfft'
2022-05-18T03:59:53.1103569Z Entering 'third_party/protobuf'
2022-05-18T03:59:53.1147517Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T03:59:53.1189351Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T03:59:53.1233310Z Entering 'third_party/psimd'
2022-05-18T03:59:53.1275329Z Entering 'third_party/pthreadpool'
2022-05-18T03:59:53.1316779Z Entering 'third_party/pybind11'
2022-05-18T03:59:53.1359042Z Entering 'third_party/python-enum'
2022-05-18T03:59:53.1400143Z Entering 'third_party/python-peachpy'
2022-05-18T03:59:53.1440546Z Entering 'third_party/python-six'
2022-05-18T03:59:53.1482955Z Entering 'third_party/sleef'
2022-05-18T03:59:53.1524742Z Entering 'third_party/tbb'
2022-05-18T03:59:53.1568231Z Entering 'third_party/tensorpipe'
2022-05-18T03:59:53.1609698Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T03:59:53.1651488Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T03:59:53.1693107Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T03:59:53.1734273Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T03:59:53.1774625Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T03:59:53.1817593Z Entering 'third_party/zstd'
2022-05-18T03:59:53.1872955Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-05-18T03:59:53.2192726Z Entering 'android/libs/fbjni'
2022-05-18T03:59:53.2231803Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-05-18T03:59:53.2249205Z Entering 'third_party/FP16'
2022-05-18T03:59:53.2289220Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-05-18T03:59:53.2306275Z Entering 'third_party/FXdiv'
2022-05-18T03:59:53.2344918Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-05-18T03:59:53.2362540Z Entering 'third_party/NNPACK'
2022-05-18T03:59:53.2400572Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-05-18T03:59:53.2417559Z Entering 'third_party/QNNPACK'
2022-05-18T03:59:53.2456196Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-05-18T03:59:53.2473753Z Entering 'third_party/XNNPACK'
2022-05-18T03:59:53.2512889Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-05-18T03:59:53.2540916Z Entering 'third_party/benchmark'
2022-05-18T03:59:53.2580838Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T03:59:53.2598138Z Entering 'third_party/cpuinfo'
2022-05-18T03:59:53.2636833Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-05-18T03:59:53.2654749Z Entering 'third_party/cub'
2022-05-18T03:59:53.2693393Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-05-18T03:59:53.2710154Z Entering 'third_party/cudnn_frontend'
2022-05-18T03:59:53.2749165Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-05-18T03:59:53.2772983Z Entering 'third_party/eigen'
2022-05-18T03:59:53.2811326Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-05-18T03:59:53.2830424Z Entering 'third_party/fbgemm'
2022-05-18T03:59:53.2869064Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-05-18T03:59:53.2886303Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T03:59:53.2924629Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-05-18T03:59:53.2941575Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T03:59:53.2980391Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-05-18T03:59:53.2997530Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T03:59:53.3035385Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-05-18T03:59:53.3055014Z Entering 'third_party/flatbuffers'
2022-05-18T03:59:53.3093328Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-05-18T03:59:53.3112491Z Entering 'third_party/fmt'
2022-05-18T03:59:53.3151434Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-05-18T03:59:53.3168879Z Entering 'third_party/foxi'
2022-05-18T03:59:53.3207205Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-05-18T03:59:53.3224216Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T03:59:53.3262269Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-05-18T03:59:53.3279320Z Entering 'third_party/gloo'
2022-05-18T03:59:53.3316762Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-05-18T03:59:53.3334026Z Entering 'third_party/googletest'
2022-05-18T03:59:53.3372369Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-05-18T03:59:53.3389755Z Entering 'third_party/ideep'
2022-05-18T03:59:53.3428300Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-05-18T03:59:53.3445442Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T03:59:53.3483612Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-05-18T03:59:53.3502653Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T03:59:53.3541148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-05-18T03:59:53.3565143Z Entering 'third_party/ios-cmake'
2022-05-18T03:59:53.3603346Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-05-18T03:59:53.3620380Z Entering 'third_party/kineto'
2022-05-18T03:59:53.3658832Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-05-18T03:59:53.3675250Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T03:59:53.3713385Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-05-18T03:59:53.3731143Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T03:59:53.3769060Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-05-18T03:59:53.3787178Z Entering 'third_party/nccl/nccl'
2022-05-18T03:59:53.3827048Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-05-18T03:59:53.3844484Z Entering 'third_party/neon2sse'
2022-05-18T03:59:53.3885156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-05-18T03:59:53.3901949Z Entering 'third_party/onnx'
2022-05-18T03:59:53.3940635Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-05-18T03:59:53.3970638Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T03:59:53.4009866Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T03:59:53.4026961Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T03:59:53.4068005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T03:59:53.4087930Z Entering 'third_party/onnx-tensorrt'
2022-05-18T03:59:53.4128309Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-05-18T03:59:53.4145809Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T03:59:53.4185938Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-05-18T03:59:53.4211098Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T03:59:53.4383969Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T03:59:53.4402586Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T03:59:53.4494564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T03:59:53.4511694Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T03:59:53.4552199Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-05-18T03:59:53.4575120Z Entering 'third_party/pocketfft'
2022-05-18T03:59:53.4614412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-05-18T03:59:53.4631775Z Entering 'third_party/protobuf'
2022-05-18T03:59:53.4672155Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-05-18T03:59:53.4694277Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T03:59:53.4732617Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T03:59:53.4749930Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T03:59:53.4789641Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-05-18T03:59:53.4809666Z Entering 'third_party/psimd'
2022-05-18T03:59:53.4848612Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-05-18T03:59:53.4865586Z Entering 'third_party/pthreadpool'
2022-05-18T03:59:53.4903612Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-05-18T03:59:53.4921037Z Entering 'third_party/pybind11'
2022-05-18T03:59:53.4959295Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T03:59:53.4977507Z Entering 'third_party/python-enum'
2022-05-18T03:59:53.5016215Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-05-18T03:59:53.5033273Z Entering 'third_party/python-peachpy'
2022-05-18T03:59:53.5072398Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-05-18T03:59:53.5090038Z Entering 'third_party/python-six'
2022-05-18T03:59:53.5130456Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-05-18T03:59:53.5147528Z Entering 'third_party/sleef'
2022-05-18T03:59:53.5186085Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-05-18T03:59:53.5205241Z Entering 'third_party/tbb'
2022-05-18T03:59:53.5242957Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-05-18T03:59:53.5262407Z Entering 'third_party/tensorpipe'
2022-05-18T03:59:53.5300624Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-05-18T03:59:53.5317762Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T03:59:53.5356297Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-05-18T03:59:53.5374296Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T03:59:53.5412208Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-05-18T03:59:53.5428837Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T03:59:53.5468037Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-05-18T03:59:53.5486659Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T03:59:53.5524552Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T03:59:53.5540881Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T03:59:53.5580535Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-05-18T03:59:53.5600675Z Entering 'third_party/zstd'
2022-05-18T03:59:53.5639378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-05-18T03:59:53.6445185Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-05-18T03:59:53.6762881Z Entering 'android/libs/fbjni'
2022-05-18T03:59:53.6805722Z Entering 'third_party/FP16'
2022-05-18T03:59:53.6848183Z Entering 'third_party/FXdiv'
2022-05-18T03:59:53.6890373Z Entering 'third_party/NNPACK'
2022-05-18T03:59:53.6933313Z Entering 'third_party/QNNPACK'
2022-05-18T03:59:53.6975914Z Entering 'third_party/XNNPACK'
2022-05-18T03:59:53.7029779Z Entering 'third_party/benchmark'
2022-05-18T03:59:53.7072643Z Entering 'third_party/cpuinfo'
2022-05-18T03:59:53.7115938Z Entering 'third_party/cub'
2022-05-18T03:59:53.7157664Z Entering 'third_party/cudnn_frontend'
2022-05-18T03:59:53.7206070Z Entering 'third_party/eigen'
2022-05-18T03:59:53.7251417Z Entering 'third_party/fbgemm'
2022-05-18T03:59:53.7293802Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T03:59:53.7335002Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T03:59:53.7377523Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T03:59:53.7421575Z Entering 'third_party/flatbuffers'
2022-05-18T03:59:53.7466020Z Entering 'third_party/fmt'
2022-05-18T03:59:53.7507567Z Entering 'third_party/foxi'
2022-05-18T03:59:53.7550113Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T03:59:53.7592454Z Entering 'third_party/gloo'
2022-05-18T03:59:53.7634159Z Entering 'third_party/googletest'
2022-05-18T03:59:53.7677009Z Entering 'third_party/ideep'
2022-05-18T03:59:53.7719241Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T03:59:53.7763541Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T03:59:53.7811780Z Entering 'third_party/ios-cmake'
2022-05-18T03:59:53.7853518Z Entering 'third_party/kineto'
2022-05-18T03:59:53.7895809Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T03:59:53.7938542Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T03:59:53.7981436Z Entering 'third_party/nccl/nccl'
2022-05-18T03:59:53.8023445Z Entering 'third_party/neon2sse'
2022-05-18T03:59:53.8066130Z Entering 'third_party/onnx'
2022-05-18T03:59:53.8118673Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T03:59:53.8161159Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T03:59:53.8205518Z Entering 'third_party/onnx-tensorrt'
2022-05-18T03:59:53.8247367Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T03:59:53.8295311Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T03:59:53.8337999Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T03:59:53.8380165Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T03:59:53.8427580Z Entering 'third_party/pocketfft'
2022-05-18T03:59:53.8470196Z Entering 'third_party/protobuf'
2022-05-18T03:59:53.8516110Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T03:59:53.8560092Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T03:59:53.8604334Z Entering 'third_party/psimd'
2022-05-18T03:59:53.8647177Z Entering 'third_party/pthreadpool'
2022-05-18T03:59:53.8690380Z Entering 'third_party/pybind11'
2022-05-18T03:59:53.8732690Z Entering 'third_party/python-enum'
2022-05-18T03:59:53.8775033Z Entering 'third_party/python-peachpy'
2022-05-18T03:59:53.8818206Z Entering 'third_party/python-six'
2022-05-18T03:59:53.8860652Z Entering 'third_party/sleef'
2022-05-18T03:59:53.8903142Z Entering 'third_party/tbb'
2022-05-18T03:59:53.8948089Z Entering 'third_party/tensorpipe'
2022-05-18T03:59:53.8991821Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T03:59:53.9033074Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T03:59:53.9074896Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T03:59:53.9116654Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T03:59:53.9157334Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T03:59:53.9202842Z Entering 'third_party/zstd'
2022-05-18T03:59:53.9258446Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-05-18T03:59:53.9578985Z Entering 'android/libs/fbjni'
2022-05-18T03:59:53.9621451Z Entering 'third_party/FP16'
2022-05-18T03:59:53.9663450Z Entering 'third_party/FXdiv'
2022-05-18T03:59:53.9705926Z Entering 'third_party/NNPACK'
2022-05-18T03:59:53.9750581Z Entering 'third_party/QNNPACK'
2022-05-18T03:59:53.9793110Z Entering 'third_party/XNNPACK'
2022-05-18T03:59:53.9848148Z Entering 'third_party/benchmark'
2022-05-18T03:59:53.9890750Z Entering 'third_party/cpuinfo'
2022-05-18T03:59:53.9933234Z Entering 'third_party/cub'
2022-05-18T03:59:53.9976491Z Entering 'third_party/cudnn_frontend'
2022-05-18T03:59:54.0024199Z Entering 'third_party/eigen'
2022-05-18T03:59:54.0069141Z Entering 'third_party/fbgemm'
2022-05-18T03:59:54.0111741Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T03:59:54.0153786Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T03:59:54.0196347Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T03:59:54.0239120Z Entering 'third_party/flatbuffers'
2022-05-18T03:59:54.0283251Z Entering 'third_party/fmt'
2022-05-18T03:59:54.0326441Z Entering 'third_party/foxi'
2022-05-18T03:59:54.0368219Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T03:59:54.0411682Z Entering 'third_party/gloo'
2022-05-18T03:59:54.0453268Z Entering 'third_party/googletest'
2022-05-18T03:59:54.0495646Z Entering 'third_party/ideep'
2022-05-18T03:59:54.0537712Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T03:59:54.0581804Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T03:59:54.0630235Z Entering 'third_party/ios-cmake'
2022-05-18T03:59:54.0671969Z Entering 'third_party/kineto'
2022-05-18T03:59:54.0715171Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T03:59:54.0757918Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T03:59:54.0801483Z Entering 'third_party/nccl/nccl'
2022-05-18T03:59:54.0844275Z Entering 'third_party/neon2sse'
2022-05-18T03:59:54.0885730Z Entering 'third_party/onnx'
2022-05-18T03:59:54.0939429Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T03:59:54.0981723Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T03:59:54.1025396Z Entering 'third_party/onnx-tensorrt'
2022-05-18T03:59:54.1067002Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T03:59:54.1113901Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T03:59:54.1155779Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T03:59:54.1197484Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T03:59:54.1245349Z Entering 'third_party/pocketfft'
2022-05-18T03:59:54.1287460Z Entering 'third_party/protobuf'
2022-05-18T03:59:54.1334467Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T03:59:54.1376860Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T03:59:54.1420775Z Entering 'third_party/psimd'
2022-05-18T03:59:54.1462220Z Entering 'third_party/pthreadpool'
2022-05-18T03:59:54.1504430Z Entering 'third_party/pybind11'
2022-05-18T03:59:54.1546188Z Entering 'third_party/python-enum'
2022-05-18T03:59:54.1588720Z Entering 'third_party/python-peachpy'
2022-05-18T03:59:54.1631248Z Entering 'third_party/python-six'
2022-05-18T03:59:54.1673429Z Entering 'third_party/sleef'
2022-05-18T03:59:54.1715749Z Entering 'third_party/tbb'
2022-05-18T03:59:54.1760441Z Entering 'third_party/tensorpipe'
2022-05-18T03:59:54.1802508Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T03:59:54.1844869Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T03:59:54.1886524Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T03:59:54.1928747Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T03:59:54.1969484Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T03:59:54.2014908Z Entering 'third_party/zstd'
2022-05-18T03:59:54.2066468Z ##[endgroup]
2022-05-18T03:59:54.2111507Z [command]/usr/bin/git log -1 --format='%H'
2022-05-18T03:59:54.2140820Z '3b2375291aab7b48442f2e6fb1ef66cebc761e24'
2022-05-18T03:59:54.2287617Z Prepare all required actions
2022-05-18T03:59:54.2317714Z ##[group]Run ./.github/actions/setup-linux
2022-05-18T03:59:54.2317993Z env:
2022-05-18T03:59:54.2319065Z   IN_CI: 1
2022-05-18T03:59:54.2319297Z   IS_GHA: 1
2022-05-18T03:59:54.2319550Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:59:54.2319791Z ##[endgroup]
2022-05-18T03:59:54.2336731Z ##[group]Run set -euo pipefail
2022-05-18T03:59:54.2337045Z set -euo pipefail
2022-05-18T03:59:54.2337339Z function get_ec2_metadata() {
2022-05-18T03:59:54.2337661Z   # Pulled from instance metadata endpoint for EC2
2022-05-18T03:59:54.2338145Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-05-18T03:59:54.2338562Z   category=$1
2022-05-18T03:59:54.2338880Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-05-18T03:59:54.2339192Z }
2022-05-18T03:59:54.2339507Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-05-18T03:59:54.2339845Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-05-18T03:59:54.2340224Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-05-18T03:59:54.2340569Z echo "system info $(uname -a)"
2022-05-18T03:59:54.2354129Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:59:54.2354435Z env:
2022-05-18T03:59:54.2354659Z   IN_CI: 1
2022-05-18T03:59:54.2354870Z   IS_GHA: 1
2022-05-18T03:59:54.2355126Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:59:54.2355388Z ##[endgroup]
2022-05-18T03:59:54.2457899Z ami-id: ami-096198a0bccc6bad4
2022-05-18T03:59:54.2522279Z instance-id: i-08f720eed7f2b488f
2022-05-18T03:59:54.2583715Z instance-type: g3.8xlarge
2022-05-18T03:59:54.2592292Z system info Linux ip-10-0-4-36.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-05-18T03:59:54.2610247Z ##[group]Run if systemctl is-active --quiet docker; then
2022-05-18T03:59:54.2610653Z if systemctl is-active --quiet docker; then
2022-05-18T03:59:54.2610974Z     echo "Docker daemon is running...";
2022-05-18T03:59:54.2611253Z else
2022-05-18T03:59:54.2611572Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-05-18T03:59:54.2611864Z fi
2022-05-18T03:59:54.2623427Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:59:54.2623729Z env:
2022-05-18T03:59:54.2623931Z   IN_CI: 1
2022-05-18T03:59:54.2624153Z   IS_GHA: 1
2022-05-18T03:59:54.2624401Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:59:54.2624643Z ##[endgroup]
2022-05-18T03:59:54.2675470Z Docker daemon is running...
2022-05-18T03:59:54.2693254Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-05-18T03:59:54.2693724Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-05-18T03:59:54.2694095Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-05-18T03:59:54.2694579Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-05-18T03:59:54.2695038Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-05-18T03:59:54.2706657Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:59:54.2706937Z env:
2022-05-18T03:59:54.2707154Z   IN_CI: 1
2022-05-18T03:59:54.2707376Z   IS_GHA: 1
2022-05-18T03:59:54.2707607Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:59:54.2707874Z   AWS_RETRY_MODE: standard
2022-05-18T03:59:54.2708134Z   AWS_MAX_ATTEMPTS: 5
2022-05-18T03:59:54.2708386Z   AWS_DEFAULT_REGION: us-east-1
2022-05-18T03:59:54.2708652Z ##[endgroup]
2022-05-18T03:59:55.2586144Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-05-18T03:59:55.2586637Z Configure a credential helper to remove this warning. See
2022-05-18T03:59:55.2587638Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-05-18T03:59:55.2588197Z 
2022-05-18T03:59:55.2588425Z Login Succeeded
2022-05-18T03:59:55.2625711Z ##[group]Run env | grep '^GITHUB' > "/tmp/github_env_${GITHUB_RUN_ID}"
2022-05-18T03:59:55.2626100Z env | grep '^GITHUB' > "/tmp/github_env_${GITHUB_RUN_ID}"
2022-05-18T03:59:55.2639625Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:59:55.2639930Z env:
2022-05-18T03:59:55.2640150Z   IN_CI: 1
2022-05-18T03:59:55.2640359Z   IS_GHA: 1
2022-05-18T03:59:55.2640607Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:59:55.2640868Z ##[endgroup]
2022-05-18T03:59:55.2703264Z Prepare all required actions
2022-05-18T03:59:55.2703636Z Getting action download info
2022-05-18T03:59:55.4380630Z Download action repository 'seemethere/add-github-ssh-key@v1' (SHA:1ecffedb1e192a50aa67dba2f0e048e5d3bfa144)
2022-05-18T03:59:55.5595396Z ##[group]Run ./.github/actions/setup-ssh
2022-05-18T03:59:55.5595692Z with:
2022-05-18T03:59:55.5596125Z   github-secret: ***
2022-05-18T03:59:55.5596394Z env:
2022-05-18T03:59:55.5596634Z   IN_CI: 1
2022-05-18T03:59:55.5596857Z   IS_GHA: 1
2022-05-18T03:59:55.5597123Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:59:55.5597403Z ##[endgroup]
2022-05-18T03:59:55.5624873Z ##[group]Run seemethere/add-github-ssh-key@v1
2022-05-18T03:59:55.5625183Z with:
2022-05-18T03:59:55.5625559Z   GITHUB_TOKEN: ***
2022-05-18T03:59:55.5625852Z   activate-with-label: false
2022-05-18T03:59:55.5626141Z   label: with-ssh
2022-05-18T03:59:55.5626401Z   remove-existing-keys: true
2022-05-18T03:59:55.5626675Z env:
2022-05-18T03:59:55.5626909Z   IN_CI: 1
2022-05-18T03:59:55.5627183Z   IS_GHA: 1
2022-05-18T03:59:55.5627433Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:59:55.5627724Z ##[endgroup]
2022-05-18T03:59:55.6387711Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2022-05-18T03:59:55.6439361Z Prepare all required actions
2022-05-18T03:59:55.6459945Z ##[group]Run ./.github/actions/pull-docker-image
2022-05-18T03:59:55.6460237Z with:
2022-05-18T03:59:55.6460737Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T03:59:55.6461198Z env:
2022-05-18T03:59:55.6461419Z   IN_CI: 1
2022-05-18T03:59:55.6461646Z   IS_GHA: 1
2022-05-18T03:59:55.6461879Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:59:55.6462139Z ##[endgroup]
2022-05-18T03:59:55.6477857Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-05-18T03:59:55.6478670Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-05-18T03:59:55.6479063Z retry docker pull "${DOCKER_IMAGE}"
2022-05-18T03:59:55.6491653Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:59:55.6492028Z env:
2022-05-18T03:59:55.6492249Z   IN_CI: 1
2022-05-18T03:59:55.6492501Z   IS_GHA: 1
2022-05-18T03:59:55.6492759Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:59:55.6493270Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T03:59:55.6493764Z ##[endgroup]
2022-05-18T03:59:55.8923547Z 6deab82db6a72ca54cd3e3322ee4f13864536734: Pulling from pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7
2022-05-18T03:59:55.8924547Z 11323ed2c653: Pulling fs layer
2022-05-18T03:59:55.8924845Z 9b0c32b3202c: Pulling fs layer
2022-05-18T03:59:55.8925111Z 55d4aa3df964: Pulling fs layer
2022-05-18T03:59:55.8925559Z ced0e45f533f: Pulling fs layer
2022-05-18T03:59:55.8925845Z a6d5f855f26c: Pulling fs layer
2022-05-18T03:59:55.8926130Z 532188ad0a5d: Pulling fs layer
2022-05-18T03:59:55.8926413Z 53b0132b34a2: Pulling fs layer
2022-05-18T03:59:55.8926714Z d63f711e9949: Pulling fs layer
2022-05-18T03:59:55.8926977Z 776e7a7e28b2: Pulling fs layer
2022-05-18T03:59:55.8927256Z ced0e45f533f: Waiting
2022-05-18T03:59:55.8927530Z 69004237646f: Pulling fs layer
2022-05-18T03:59:55.8927789Z 53b0132b34a2: Waiting
2022-05-18T03:59:55.8928042Z a0a6f96a62d8: Pulling fs layer
2022-05-18T03:59:55.8928322Z 7918ac79e586: Pulling fs layer
2022-05-18T03:59:55.8928597Z 517f3f32e512: Pulling fs layer
2022-05-18T03:59:55.8928839Z a6d5f855f26c: Waiting
2022-05-18T03:59:55.8929089Z 69004237646f: Waiting
2022-05-18T03:59:55.8929349Z 7c88fb71bf11: Pulling fs layer
2022-05-18T03:59:55.8929592Z 532188ad0a5d: Waiting
2022-05-18T03:59:55.8929851Z 7b920d7a1988: Pulling fs layer
2022-05-18T03:59:55.8930126Z 0ba8a6800faf: Pulling fs layer
2022-05-18T03:59:55.8930384Z 6d58a87851d7: Pulling fs layer
2022-05-18T03:59:55.8930649Z a0a6f96a62d8: Waiting
2022-05-18T03:59:55.8930899Z 517f3f32e512: Waiting
2022-05-18T03:59:55.8931127Z 776e7a7e28b2: Waiting
2022-05-18T03:59:55.8931382Z d63f711e9949: Waiting
2022-05-18T03:59:55.8931683Z b06b299e7454: Pulling fs layer
2022-05-18T03:59:55.8931944Z b046a45d4ca8: Pulling fs layer
2022-05-18T03:59:55.8932229Z acf3886a01ad: Pulling fs layer
2022-05-18T03:59:55.8932502Z 166228572fc8: Pulling fs layer
2022-05-18T03:59:55.8932741Z 6d58a87851d7: Waiting
2022-05-18T03:59:55.8933004Z 6d680b004bdb: Pulling fs layer
2022-05-18T03:59:55.8933267Z b06b299e7454: Waiting
2022-05-18T03:59:55.8933512Z b046a45d4ca8: Waiting
2022-05-18T03:59:55.8933754Z 4d9d54d04be5: Pulling fs layer
2022-05-18T03:59:55.8934028Z 55e19101ee96: Pulling fs layer
2022-05-18T03:59:55.8934289Z acf3886a01ad: Waiting
2022-05-18T03:59:55.8934531Z d57378452c6c: Pulling fs layer
2022-05-18T03:59:55.8934790Z 166228572fc8: Waiting
2022-05-18T03:59:55.8935035Z 0ba8a6800faf: Waiting
2022-05-18T03:59:55.8935274Z 4097195e70a4: Pulling fs layer
2022-05-18T03:59:55.8935536Z 6d680b004bdb: Waiting
2022-05-18T03:59:55.8935794Z e90775d597ae: Pulling fs layer
2022-05-18T03:59:55.8936034Z 4d9d54d04be5: Waiting
2022-05-18T03:59:55.8936287Z 7c88fb71bf11: Waiting
2022-05-18T03:59:55.8936546Z 342cb5b8793f: Pulling fs layer
2022-05-18T03:59:55.8936972Z d57378452c6c: Waiting
2022-05-18T03:59:55.8937268Z ec9f4694245d: Pulling fs layer
2022-05-18T03:59:55.8937544Z 5ff41a564c23: Pulling fs layer
2022-05-18T03:59:55.8937801Z 5e9e1c5c2b02: Pulling fs layer
2022-05-18T03:59:55.8938075Z 85cae8860e8b: Pulling fs layer
2022-05-18T03:59:55.8938341Z 7b920d7a1988: Waiting
2022-05-18T03:59:55.8938569Z 342cb5b8793f: Waiting
2022-05-18T03:59:55.8938817Z 5e9e1c5c2b02: Waiting
2022-05-18T03:59:55.8939064Z 85cae8860e8b: Waiting
2022-05-18T03:59:55.8939308Z 7bd074c80c3f: Pulling fs layer
2022-05-18T03:59:55.8939587Z 7ebce38575d6: Pulling fs layer
2022-05-18T03:59:55.8939849Z 5ff41a564c23: Waiting
2022-05-18T03:59:55.8940094Z 3dcf0fc78ba8: Pulling fs layer
2022-05-18T03:59:55.8940360Z 7bd074c80c3f: Waiting
2022-05-18T03:59:55.8940629Z de93ffc12e40: Pulling fs layer
2022-05-18T03:59:55.8940889Z fd0f553736b3: Pulling fs layer
2022-05-18T03:59:55.8941469Z 6b52bc4fc524: Pulling fs layer
2022-05-18T03:59:55.8941757Z f709baccd3f5: Pulling fs layer
2022-05-18T03:59:55.8942027Z 25dff8b9a054: Pulling fs layer
2022-05-18T03:59:55.8942310Z bcd88fe424d2: Pulling fs layer
2022-05-18T03:59:55.8942585Z 8710652e57c7: Pulling fs layer
2022-05-18T03:59:55.8942836Z 050758b5b900: Pulling fs layer
2022-05-18T03:59:55.8943229Z 7ebce38575d6: Waiting
2022-05-18T03:59:55.8943488Z de93ffc12e40: Waiting
2022-05-18T03:59:55.8943715Z 6b52bc4fc524: Waiting
2022-05-18T03:59:55.8943980Z e104e8ddd08b: Pulling fs layer
2022-05-18T03:59:55.8944305Z b0c972c96382: Pulling fs layer
2022-05-18T03:59:55.8944583Z 053d59c76970: Pulling fs layer
2022-05-18T03:59:55.8944825Z f709baccd3f5: Waiting
2022-05-18T03:59:55.8945078Z fd0f553736b3: Waiting
2022-05-18T03:59:55.8945342Z 30dcacd2ffe2: Pulling fs layer
2022-05-18T03:59:55.8945583Z 8710652e57c7: Waiting
2022-05-18T03:59:55.8945844Z 1c1fd12e267d: Pulling fs layer
2022-05-18T03:59:55.8946109Z bcd88fe424d2: Waiting
2022-05-18T03:59:55.8946348Z 25dff8b9a054: Waiting
2022-05-18T03:59:55.8946598Z e104e8ddd08b: Waiting
2022-05-18T03:59:55.8946846Z b0c972c96382: Waiting
2022-05-18T03:59:55.8947074Z 30dcacd2ffe2: Waiting
2022-05-18T03:59:55.8947322Z 053d59c76970: Waiting
2022-05-18T03:59:55.8947559Z 050758b5b900: Waiting
2022-05-18T03:59:56.0387326Z 9b0c32b3202c: Verifying Checksum
2022-05-18T03:59:56.0387837Z 9b0c32b3202c: Download complete
2022-05-18T03:59:56.0514495Z 55d4aa3df964: Verifying Checksum
2022-05-18T03:59:56.0515047Z 55d4aa3df964: Download complete
2022-05-18T03:59:56.1004827Z ced0e45f533f: Verifying Checksum
2022-05-18T03:59:56.1005149Z ced0e45f533f: Download complete
2022-05-18T03:59:56.1198314Z a6d5f855f26c: Verifying Checksum
2022-05-18T03:59:56.1198885Z a6d5f855f26c: Download complete
2022-05-18T03:59:56.1956302Z 53b0132b34a2: Download complete
2022-05-18T03:59:56.2174239Z 11323ed2c653: Verifying Checksum
2022-05-18T03:59:56.2174531Z 11323ed2c653: Download complete
2022-05-18T03:59:56.2967243Z 776e7a7e28b2: Download complete
2022-05-18T03:59:57.0373237Z 11323ed2c653: Pull complete
2022-05-18T03:59:57.3247995Z 9b0c32b3202c: Pull complete
2022-05-18T03:59:57.6261294Z 55d4aa3df964: Pull complete
2022-05-18T03:59:57.7521254Z ced0e45f533f: Pull complete
2022-05-18T03:59:57.8526334Z a6d5f855f26c: Pull complete
2022-05-18T04:00:02.2005615Z 69004237646f: Verifying Checksum
2022-05-18T04:00:02.2005962Z 69004237646f: Download complete
2022-05-18T04:00:02.3125348Z a0a6f96a62d8: Download complete
2022-05-18T04:00:04.2450944Z 7918ac79e586: Verifying Checksum
2022-05-18T04:00:04.2451304Z 7918ac79e586: Download complete
2022-05-18T04:00:04.3142960Z 532188ad0a5d: Verifying Checksum
2022-05-18T04:00:04.3143298Z 532188ad0a5d: Download complete
2022-05-18T04:00:04.3334292Z 517f3f32e512: Verifying Checksum
2022-05-18T04:00:04.3334645Z 517f3f32e512: Download complete
2022-05-18T04:00:04.3974465Z 7c88fb71bf11: Verifying Checksum
2022-05-18T04:00:04.3974927Z 7c88fb71bf11: Download complete
2022-05-18T04:00:04.4226468Z 7b920d7a1988: Verifying Checksum
2022-05-18T04:00:04.4226816Z 7b920d7a1988: Download complete
2022-05-18T04:00:04.4640046Z 0ba8a6800faf: Verifying Checksum
2022-05-18T04:00:04.4640560Z 0ba8a6800faf: Download complete
2022-05-18T04:00:04.5112573Z 6d58a87851d7: Verifying Checksum
2022-05-18T04:00:04.5112901Z 6d58a87851d7: Download complete
2022-05-18T04:00:04.5415724Z b06b299e7454: Verifying Checksum
2022-05-18T04:00:04.5416084Z b06b299e7454: Download complete
2022-05-18T04:00:04.6123422Z acf3886a01ad: Download complete
2022-05-18T04:00:04.6873671Z 166228572fc8: Download complete
2022-05-18T04:00:04.7927866Z 6d680b004bdb: Verifying Checksum
2022-05-18T04:00:04.7928226Z 6d680b004bdb: Download complete
2022-05-18T04:00:04.8659005Z 4d9d54d04be5: Download complete
2022-05-18T04:00:04.9496229Z 55e19101ee96: Verifying Checksum
2022-05-18T04:00:04.9496578Z 55e19101ee96: Download complete
2022-05-18T04:00:05.0350515Z d57378452c6c: Verifying Checksum
2022-05-18T04:00:05.0350831Z d57378452c6c: Download complete
2022-05-18T04:00:05.0815804Z d63f711e9949: Verifying Checksum
2022-05-18T04:00:05.0816135Z d63f711e9949: Download complete
2022-05-18T04:00:05.1479846Z e90775d597ae: Verifying Checksum
2022-05-18T04:00:05.1480178Z e90775d597ae: Download complete
2022-05-18T04:00:05.2101700Z 342cb5b8793f: Verifying Checksum
2022-05-18T04:00:05.2101997Z 342cb5b8793f: Download complete
2022-05-18T04:00:05.2905605Z ec9f4694245d: Verifying Checksum
2022-05-18T04:00:05.2905938Z ec9f4694245d: Download complete
2022-05-18T04:00:05.3723221Z 5ff41a564c23: Verifying Checksum
2022-05-18T04:00:05.3723527Z 5ff41a564c23: Download complete
2022-05-18T04:00:05.4442594Z 5e9e1c5c2b02: Verifying Checksum
2022-05-18T04:00:05.4443275Z 5e9e1c5c2b02: Download complete
2022-05-18T04:00:05.9778225Z 4097195e70a4: Verifying Checksum
2022-05-18T04:00:05.9778587Z 4097195e70a4: Download complete
2022-05-18T04:00:06.0628699Z 7bd074c80c3f: Download complete
2022-05-18T04:00:06.1440165Z 7ebce38575d6: Download complete
2022-05-18T04:00:06.3957557Z 3dcf0fc78ba8: Verifying Checksum
2022-05-18T04:00:06.3957945Z 3dcf0fc78ba8: Download complete
2022-05-18T04:00:06.4952038Z de93ffc12e40: Download complete
2022-05-18T04:00:06.6580562Z 6b52bc4fc524: Verifying Checksum
2022-05-18T04:00:06.6580922Z 6b52bc4fc524: Download complete
2022-05-18T04:00:07.4940306Z 85cae8860e8b: Verifying Checksum
2022-05-18T04:00:07.4940687Z 85cae8860e8b: Download complete
2022-05-18T04:00:07.5705247Z 25dff8b9a054: Download complete
2022-05-18T04:00:07.6487980Z bcd88fe424d2: Verifying Checksum
2022-05-18T04:00:07.6488336Z bcd88fe424d2: Download complete
2022-05-18T04:00:07.7243713Z 8710652e57c7: Verifying Checksum
2022-05-18T04:00:07.7244040Z 8710652e57c7: Download complete
2022-05-18T04:00:07.8074469Z 050758b5b900: Verifying Checksum
2022-05-18T04:00:07.8074825Z 050758b5b900: Download complete
2022-05-18T04:00:08.0063307Z e104e8ddd08b: Verifying Checksum
2022-05-18T04:00:08.0063789Z e104e8ddd08b: Download complete
2022-05-18T04:00:08.0798281Z b0c972c96382: Verifying Checksum
2022-05-18T04:00:08.0798582Z b0c972c96382: Download complete
2022-05-18T04:00:08.6725376Z 053d59c76970: Download complete
2022-05-18T04:00:08.7452573Z 30dcacd2ffe2: Download complete
2022-05-18T04:00:08.8186170Z 1c1fd12e267d: Verifying Checksum
2022-05-18T04:00:08.8186507Z 1c1fd12e267d: Download complete
2022-05-18T04:00:10.1089012Z f709baccd3f5: Verifying Checksum
2022-05-18T04:00:10.1089691Z f709baccd3f5: Download complete
2022-05-18T04:00:13.5081862Z 532188ad0a5d: Pull complete
2022-05-18T04:00:13.6131042Z 53b0132b34a2: Pull complete
2022-05-18T04:00:25.5523003Z b046a45d4ca8: Verifying Checksum
2022-05-18T04:00:25.5523374Z b046a45d4ca8: Download complete
2022-05-18T04:00:26.4087562Z d63f711e9949: Pull complete
2022-05-18T04:00:26.5373082Z 776e7a7e28b2: Pull complete
2022-05-18T04:00:33.2861770Z 69004237646f: Pull complete
2022-05-18T04:00:33.4034104Z a0a6f96a62d8: Pull complete
2022-05-18T04:00:39.9936746Z 7918ac79e586: Pull complete
2022-05-18T04:00:41.8723291Z 517f3f32e512: Pull complete
2022-05-18T04:00:43.7179640Z 7c88fb71bf11: Pull complete
2022-05-18T04:00:45.5976020Z 7b920d7a1988: Pull complete
2022-05-18T04:00:47.4752737Z 0ba8a6800faf: Pull complete
2022-05-18T04:00:49.3473038Z 6d58a87851d7: Pull complete
2022-05-18T04:00:51.2007323Z b06b299e7454: Pull complete
2022-05-18T04:01:27.5542530Z b046a45d4ca8: Pull complete
2022-05-18T04:01:29.8807687Z acf3886a01ad: Pull complete
2022-05-18T04:01:31.7926296Z 166228572fc8: Pull complete
2022-05-18T04:01:34.3691935Z 6d680b004bdb: Pull complete
2022-05-18T04:01:37.1884435Z 4d9d54d04be5: Pull complete
2022-05-18T04:01:39.4933488Z 55e19101ee96: Pull complete
2022-05-18T04:01:41.5805872Z d57378452c6c: Pull complete
2022-05-18T04:01:44.3649234Z 4097195e70a4: Pull complete
2022-05-18T04:01:44.4822426Z e90775d597ae: Pull complete
2022-05-18T04:01:44.5891688Z 342cb5b8793f: Pull complete
2022-05-18T04:01:44.7438000Z ec9f4694245d: Pull complete
2022-05-18T04:01:44.8565084Z 5ff41a564c23: Pull complete
2022-05-18T04:01:44.9796080Z 5e9e1c5c2b02: Pull complete
2022-05-18T04:01:53.6480882Z 85cae8860e8b: Pull complete
2022-05-18T04:01:56.2009921Z 7bd074c80c3f: Pull complete
2022-05-18T04:01:58.3687300Z 7ebce38575d6: Pull complete
2022-05-18T04:02:01.1713588Z 3dcf0fc78ba8: Pull complete
2022-05-18T04:02:04.2029768Z de93ffc12e40: Pull complete
2022-05-18T04:02:07.6990884Z fd0f553736b3: Pull complete
2022-05-18T04:02:09.6267418Z 6b52bc4fc524: Pull complete
2022-05-18T04:02:15.1659525Z f709baccd3f5: Pull complete
2022-05-18T04:02:16.4840975Z 25dff8b9a054: Pull complete
2022-05-18T04:02:17.9534916Z bcd88fe424d2: Pull complete
2022-05-18T04:02:19.9931378Z 8710652e57c7: Pull complete
2022-05-18T04:02:22.6584104Z 050758b5b900: Pull complete
2022-05-18T04:02:23.5270446Z e104e8ddd08b: Pull complete
2022-05-18T04:02:23.6423571Z b0c972c96382: Pull complete
2022-05-18T04:02:25.5910443Z 053d59c76970: Pull complete
2022-05-18T04:02:25.7128575Z 30dcacd2ffe2: Pull complete
2022-05-18T04:02:25.8368500Z 1c1fd12e267d: Pull complete
2022-05-18T04:02:25.8548003Z Digest: sha256:9737b662edb86afcd12a9367db6178a57889543632c0b710c5058abe14dc048f
2022-05-18T04:02:25.8622728Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:02:25.8670712Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:02:25.8784334Z ##[group]Run nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a
2022-05-18T04:02:25.8784685Z with:
2022-05-18T04:02:25.8784925Z   timeout_minutes: 10
2022-05-18T04:02:25.8785160Z   max_attempts: 3
2022-05-18T04:02:25.8785551Z   command: set -ex
bash .github/scripts/install_nvidia_utils_linux.sh
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2022-05-18T04:02:25.8785941Z   retry_wait_seconds: 10
2022-05-18T04:02:25.8786195Z   polling_interval_seconds: 1
2022-05-18T04:02:25.8786465Z   warning_on_retry: true
2022-05-18T04:02:25.8786723Z   continue_on_error: false
2022-05-18T04:02:25.8786948Z env:
2022-05-18T04:02:25.8787168Z   IN_CI: 1
2022-05-18T04:02:25.8787389Z   IS_GHA: 1
2022-05-18T04:02:25.8787618Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:02:25.8787875Z ##[endgroup]
2022-05-18T04:02:25.9238978Z 
2022-05-18T04:02:25.9314928Z == Installing nvidia container toolkit for amzn2 ==
2022-05-18T04:02:25.9318080Z + bash .github/scripts/install_nvidia_utils_linux.sh
2022-05-18T04:02:25.9318518Z + sudo yum install -y yum-utils
2022-05-18T04:02:26.4711837Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:02:27.7655436Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2022-05-18T04:02:27.7656201Z Nothing to do
2022-05-18T04:02:27.8359152Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-05-18T04:02:28.3786861Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:02:28.4127497Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-05-18T04:02:28.4128212Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2022-05-18T04:02:28.4128742Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2022-05-18T04:02:28.4270011Z + sudo yum install -y nvidia-docker2
2022-05-18T04:02:28.9484259Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:02:30.1948040Z Package nvidia-docker2-2.10.0-1.noarch already installed and latest version
2022-05-18T04:02:30.1948857Z Nothing to do
2022-05-18T04:02:30.2653274Z + sudo systemctl restart docker
2022-05-18T04:02:38.3172449Z == Installing nvidia driver NVIDIA-Linux-x86_64-510.60.02.run ==
2022-05-18T04:02:38.3173333Z + sudo yum groupinstall -y 'Development Tools'
2022-05-18T04:02:38.8638982Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:02:39.8890078Z Maybe run: yum groups mark install (see man yum)
2022-05-18T04:02:39.8890519Z No packages in any requested group available to install or update
2022-05-18T04:02:39.9586284Z ++ uname -r
2022-05-18T04:02:39.9591719Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.252-195.483.amzn2.x86_64'
2022-05-18T04:02:40.5001115Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:02:41.7767873Z Package kernel-devel-4.14.252-195.483.amzn2.x86_64 already installed and latest version
2022-05-18T04:02:41.7768291Z Nothing to do
2022-05-18T04:02:41.8449670Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-510.60.02.run
2022-05-18T04:02:45.1597305Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm
2022-05-18T04:02:46.4027835Z Verifying archive integrity... OK
2022-05-18T04:03:10.8566411Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 510.60.02..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2022-05-18T04:03:11.0914968Z 
2022-05-18T04:03:11.0915751Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver.
2022-05-18T04:03:11.0926049Z 
2022-05-18T04:03:25.9379747Z 
2022-05-18T04:03:25.9382029Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system.  If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver.
2022-05-18T04:03:25.9382727Z 
2022-05-18T04:03:34.8238580Z + sudo rm -fv /tmp/nvidia_driver
2022-05-18T04:03:34.8752994Z removed ‘/tmp/nvidia_driver’
2022-05-18T04:03:34.8769051Z + nvidia-smi
2022-05-18T04:03:39.2769192Z Wed May 18 04:03:39 2022       
2022-05-18T04:03:39.2769775Z +-----------------------------------------------------------------------------+
2022-05-18T04:03:39.2774404Z | NVIDIA-SMI 510.60.02    Driver Version: 510.60.02    CUDA Version: 11.6     |
2022-05-18T04:03:39.2774951Z |-------------------------------+----------------------+----------------------+
2022-05-18T04:03:39.2775462Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2022-05-18T04:03:39.2775966Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2022-05-18T04:03:39.2776332Z |                               |                      |               MIG M. |
2022-05-18T04:03:39.2780034Z |===============================+======================+======================|
2022-05-18T04:03:39.2819816Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |                    0 |
2022-05-18T04:03:39.2820344Z | N/A   30C    P0    40W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-05-18T04:03:39.2820678Z |                               |                      |                  N/A |
2022-05-18T04:03:39.2821194Z +-------------------------------+----------------------+----------------------+
2022-05-18T04:03:39.2869878Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |                    0 |
2022-05-18T04:03:39.2870401Z | N/A   33C    P0    40W / 150W |      0MiB /  7680MiB |    100%      Default |
2022-05-18T04:03:39.2870726Z |                               |                      |                  N/A |
2022-05-18T04:03:39.2871177Z +-------------------------------+----------------------+----------------------+
2022-05-18T04:03:39.2871550Z                                                                                
2022-05-18T04:03:39.2871992Z +-----------------------------------------------------------------------------+
2022-05-18T04:03:39.2872610Z | Processes:                                                                  |
2022-05-18T04:03:39.2872945Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2022-05-18T04:03:39.2873284Z |        ID   ID                                                   Usage      |
2022-05-18T04:03:39.2873600Z |=============================================================================|
2022-05-18T04:03:39.2874709Z |  No running processes found                                                 |
2022-05-18T04:03:39.2875482Z +-----------------------------------------------------------------------------+
2022-05-18T04:03:39.8206595Z + echo 'GPU_FLAG=--gpus all'
2022-05-18T04:03:39.9918004Z Command completed after 1 attempt(s).
2022-05-18T04:03:39.9918222Z 
2022-05-18T04:03:39.9983284Z Prepare all required actions
2022-05-18T04:03:39.9983659Z Getting action download info
2022-05-18T04:03:40.2137468Z Download action repository 'seemethere/download-artifact-s3@v3' (SHA:64048a097659c8ca71ceacbb3c01cee9ed6f1b05)
2022-05-18T04:03:40.3915930Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869)
2022-05-18T04:03:40.5151011Z ##[group]Run ./.github/actions/download-build-artifacts
2022-05-18T04:03:40.5151316Z with:
2022-05-18T04:03:40.5151579Z   name: linux-bionic-cuda10.2-py3.9-gcc7
2022-05-18T04:03:40.5151856Z env:
2022-05-18T04:03:40.5152065Z   IN_CI: 1
2022-05-18T04:03:40.5152269Z   IS_GHA: 1
2022-05-18T04:03:40.5152513Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:03:40.5152776Z   GPU_FLAG: --gpus all
2022-05-18T04:03:40.5153007Z ##[endgroup]
2022-05-18T04:03:40.5181413Z ##[group]Run seemethere/download-artifact-s3@v3
2022-05-18T04:03:40.5181704Z with:
2022-05-18T04:03:40.5182022Z   name: linux-bionic-cuda10.2-py3.9-gcc7
2022-05-18T04:03:40.5182313Z   s3-bucket: gha-artifacts
2022-05-18T04:03:40.5182574Z   region: us-east-1
2022-05-18T04:03:40.5182804Z env:
2022-05-18T04:03:40.5183001Z   IN_CI: 1
2022-05-18T04:03:40.5183233Z   IS_GHA: 1
2022-05-18T04:03:40.5183480Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:03:40.5183729Z   GPU_FLAG: --gpus all
2022-05-18T04:03:40.5183974Z ##[endgroup]
2022-05-18T04:03:41.0195262Z Found 1 objects with prefix pytorch/pytorch/2342799949/1/linux-bionic-cuda10.2-py3.9-gcc7/
2022-05-18T04:03:41.0195882Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-05-18T04:03:46.7341459Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-05-18T04:03:46.7341796Z 
2022-05-18T04:03:46.7343179Z Artifact download has finished successfully
2022-05-18T04:03:46.7475297Z ##[group]Run unzip -o artifacts.zip
2022-05-18T04:03:46.7475611Z unzip -o artifacts.zip
2022-05-18T04:03:46.7489591Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:03:46.7489895Z env:
2022-05-18T04:03:46.7490115Z   IN_CI: 1
2022-05-18T04:03:46.7490323Z   IS_GHA: 1
2022-05-18T04:03:46.7490579Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:03:46.7490844Z   GPU_FLAG: --gpus all
2022-05-18T04:03:46.7491077Z ##[endgroup]
2022-05-18T04:03:46.7534388Z Archive:  artifacts.zip
2022-05-18T04:03:46.7537683Z    creating: dist/
2022-05-18T04:03:48.5373548Z   inflating: dist/torch-1.12.0a0+git3b23752-cp39-cp39-linux_x86_64.whl  
2022-05-18T04:03:48.5374313Z    creating: build/custom_test_artifacts/
2022-05-18T04:03:48.5375155Z    creating: build/custom_test_artifacts/custom-op-build/
2022-05-18T04:03:48.5376086Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-05-18T04:03:48.5383287Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-05-18T04:03:48.5384375Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-05-18T04:03:48.5385513Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-05-18T04:03:48.5386714Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-05-18T04:03:48.5388056Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-05-18T04:03:48.5389739Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-05-18T04:03:48.5391463Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-05-18T04:03:48.5392631Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-05-18T04:03:48.5393815Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-05-18T04:03:48.5397066Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-05-18T04:03:48.5399218Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-05-18T04:03:48.5401269Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-05-18T04:03:48.5402987Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-05-18T04:03:48.5405183Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-05-18T04:03:48.5406473Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-05-18T04:03:48.5407691Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-05-18T04:03:48.5408852Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-05-18T04:03:48.5454631Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-05-18T04:03:48.5456127Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-05-18T04:03:48.5457635Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-05-18T04:03:48.5459153Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-05-18T04:03:48.5460652Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-05-18T04:03:48.5462034Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-05-18T04:03:48.5463394Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_30.cubin  
2022-05-18T04:03:48.5464877Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-05-18T04:03:48.5466385Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-05-18T04:03:48.5502849Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-05-18T04:03:48.5539259Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-05-18T04:03:48.5540746Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-05-18T04:03:48.5542091Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_30.cubin  
2022-05-18T04:03:48.5543378Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-05-18T04:03:48.5544656Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-05-18T04:03:48.5545959Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-05-18T04:03:48.5547383Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-05-18T04:03:48.5548735Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-05-18T04:03:48.5609769Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-05-18T04:03:48.5670039Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-05-18T04:03:48.5671384Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-05-18T04:03:48.5672538Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-05-18T04:03:48.5673783Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-05-18T04:03:48.5674927Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-05-18T04:03:48.5676054Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-05-18T04:03:48.5677225Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-05-18T04:03:48.5678778Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-05-18T04:03:48.5680019Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-05-18T04:03:48.5681212Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-05-18T04:03:48.5682394Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-05-18T04:03:48.5683609Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-05-18T04:03:48.5684864Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-05-18T04:03:48.5686025Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-05-18T04:03:48.5687136Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-05-18T04:03:48.5705485Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-05-18T04:03:48.5815882Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-05-18T04:03:48.5817072Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-05-18T04:03:48.5818277Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-05-18T04:03:48.5819589Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-05-18T04:03:48.5820878Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-05-18T04:03:48.5822083Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-05-18T04:03:48.5823320Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-05-18T04:03:48.5824601Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-05-18T04:03:48.5825884Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-05-18T04:03:48.5827096Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-05-18T04:03:48.5828231Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-05-18T04:03:48.5846993Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-05-18T04:03:48.5928503Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-05-18T04:03:48.5929387Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-05-18T04:03:48.5929990Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-05-18T04:03:48.5930837Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-05-18T04:03:48.5931846Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-05-18T04:03:48.5933150Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-05-18T04:03:48.5933913Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2022-05-18T04:03:48.5936884Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-05-18T04:03:48.5937554Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-05-18T04:03:48.5938296Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-05-18T04:03:48.6028571Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-05-18T04:03:48.6089779Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-05-18T04:03:48.6090239Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-05-18T04:03:48.6090697Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-05-18T04:03:48.6097177Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-05-18T04:03:48.6097715Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-05-18T04:03:48.6098243Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-05-18T04:03:48.6098807Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-05-18T04:03:48.6099359Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-05-18T04:03:48.6101487Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-05-18T04:03:48.6102628Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-05-18T04:03:48.6103189Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-05-18T04:03:48.6103747Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-05-18T04:03:48.6106554Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-05-18T04:03:48.6107731Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-05-18T04:03:48.6109666Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-05-18T04:03:48.6110287Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-05-18T04:03:48.6111621Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-05-18T04:03:48.6112743Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-05-18T04:03:48.6113322Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-05-18T04:03:48.6113885Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-05-18T04:03:48.6159443Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-05-18T04:03:48.6160146Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-05-18T04:03:48.6160871Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-05-18T04:03:48.6162251Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-05-18T04:03:48.6163854Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-05-18T04:03:48.6165199Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-05-18T04:03:48.6166074Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_30.cubin  
2022-05-18T04:03:48.6166819Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-05-18T04:03:48.6167640Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-05-18T04:03:48.6203689Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-05-18T04:03:48.6238217Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-05-18T04:03:48.6240390Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-05-18T04:03:48.6241824Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_30.cubin  
2022-05-18T04:03:48.6243260Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-05-18T04:03:48.6244132Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-05-18T04:03:48.6245485Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-05-18T04:03:48.6246186Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-05-18T04:03:48.6246820Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-05-18T04:03:48.6304822Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-05-18T04:03:48.6364187Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-05-18T04:03:48.6365540Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-05-18T04:03:48.6366835Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-05-18T04:03:48.6368015Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-05-18T04:03:48.6369179Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-05-18T04:03:48.6370335Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-05-18T04:03:48.6371636Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-05-18T04:03:48.6373012Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-05-18T04:03:48.6374388Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-05-18T04:03:48.6375072Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-05-18T04:03:48.6375663Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-05-18T04:03:48.6376232Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-05-18T04:03:48.6376829Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-05-18T04:03:48.6377429Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-05-18T04:03:48.6378017Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-05-18T04:03:48.6393957Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-05-18T04:03:48.6456817Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-05-18T04:03:48.6458226Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-05-18T04:03:48.6459577Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-05-18T04:03:48.6460641Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-05-18T04:03:48.6461829Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-05-18T04:03:48.6462488Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-05-18T04:03:48.6463037Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2022-05-18T04:03:48.6464372Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-05-18T04:03:48.6465484Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-05-18T04:03:48.6466492Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-05-18T04:03:48.6514489Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-05-18T04:03:48.6515529Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-05-18T04:03:48.6516059Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-05-18T04:03:48.6522467Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-05-18T04:03:48.6523752Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-05-18T04:03:48.6524535Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-05-18T04:03:48.6525117Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-05-18T04:03:48.6525715Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-05-18T04:03:48.6527049Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-05-18T04:03:48.6528369Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-05-18T04:03:48.6529443Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-05-18T04:03:48.6530036Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-05-18T04:03:48.6531639Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-05-18T04:03:48.6533167Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-05-18T04:03:48.6534674Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-05-18T04:03:48.6535965Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-05-18T04:03:48.6537346Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-05-18T04:03:48.6538787Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-05-18T04:03:48.6539518Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-05-18T04:03:48.6540109Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-05-18T04:03:48.6584560Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-05-18T04:03:48.6586211Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-05-18T04:03:48.6588097Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-05-18T04:03:48.6589826Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-05-18T04:03:48.6590863Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-05-18T04:03:48.6591567Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-05-18T04:03:48.6592392Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_30.cubin  
2022-05-18T04:03:48.6593129Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-05-18T04:03:48.6593859Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-05-18T04:03:48.6627868Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-05-18T04:03:48.6662839Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-05-18T04:03:48.6664433Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-05-18T04:03:48.6665957Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_30.cubin  
2022-05-18T04:03:48.6667507Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-05-18T04:03:48.6668567Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-05-18T04:03:48.6670016Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-05-18T04:03:48.6670716Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-05-18T04:03:48.6671379Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-05-18T04:03:48.6729459Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-05-18T04:03:48.6788772Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-05-18T04:03:48.6790292Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-05-18T04:03:48.6791541Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-05-18T04:03:48.6792808Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-05-18T04:03:48.6794060Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-05-18T04:03:48.6795348Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-05-18T04:03:48.6796682Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-05-18T04:03:48.6798429Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-05-18T04:03:48.6799277Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-05-18T04:03:48.6799900Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-05-18T04:03:48.6800529Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-05-18T04:03:48.6801281Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-05-18T04:03:48.6801912Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-05-18T04:03:48.6802539Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-05-18T04:03:48.6803142Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-05-18T04:03:48.6803807Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-05-18T04:03:48.6948655Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-05-18T04:03:48.6949350Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-05-18T04:03:48.6950012Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-05-18T04:03:48.6950676Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-05-18T04:03:48.6951338Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-05-18T04:03:48.6951971Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-05-18T04:03:48.6952613Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-05-18T04:03:48.6953241Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-05-18T04:03:48.6953896Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-05-18T04:03:48.6954542Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-05-18T04:03:48.6955183Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-05-18T04:03:48.6974887Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-05-18T04:03:48.7031663Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-05-18T04:03:48.7032340Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-05-18T04:03:48.7032970Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-05-18T04:03:48.7033546Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-05-18T04:03:48.7034351Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-05-18T04:03:48.7035363Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-05-18T04:03:48.7035921Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2022-05-18T04:03:48.7038877Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-05-18T04:03:48.7039828Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-05-18T04:03:48.7040636Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-05-18T04:03:48.7157556Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-05-18T04:03:48.7202136Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-05-18T04:03:48.7202510Z    creating: build/lib/
2022-05-18T04:03:48.7202997Z   inflating: build/lib/libclog.a     
2022-05-18T04:03:48.7269194Z   inflating: build/lib/libgtest.a    
2022-05-18T04:03:48.7279579Z   inflating: build/lib/libpthreadpool.a  
2022-05-18T04:03:48.7369105Z   inflating: build/lib/libbenchmark.a  
2022-05-18T04:03:48.7475119Z   inflating: build/lib/libprotobuf-lite.a  
2022-05-18T04:03:48.7506969Z   inflating: build/lib/libtensorpipe_uv.a  
2022-05-18T04:03:48.7562823Z   inflating: build/lib/libasmjit.a   
2022-05-18T04:03:48.7695307Z   inflating: build/lib/libgloo.a     
2022-05-18T04:03:48.8227862Z   inflating: build/lib/libprotobuf.a  
2022-05-18T04:03:48.8247571Z   inflating: build/lib/libfmt.a      
2022-05-18T04:03:48.8249447Z   inflating: build/lib/libcaffe2_nvrtc.so  
2022-05-18T04:03:48.8250050Z   inflating: build/lib/libfoxi_loader.a  
2022-05-18T04:03:48.8316401Z   inflating: build/lib/libc10.so     
2022-05-18T04:03:48.8317355Z   inflating: build/lib/libtorch_global_deps.so  
2022-05-18T04:03:48.8327591Z   inflating: build/lib/libcpuinfo.a  
2022-05-18T04:03:48.8336582Z   inflating: build/lib/libcpuinfo_internals.a  
2022-05-18T04:03:48.8352386Z   inflating: build/lib/libqnnpack.a  
2022-05-18T04:03:48.8921464Z   inflating: build/lib/libprotoc.a   
2022-05-18T04:03:48.8924037Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-05-18T04:03:48.8947717Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-05-18T04:03:48.8966860Z   inflating: build/lib/libgmock.a    
2022-05-18T04:03:48.8967384Z   inflating: build/lib/libgtest_main.a  
2022-05-18T04:03:48.8968462Z   inflating: build/lib/libbenchmark_main.a  
2022-05-18T04:03:48.8990781Z   inflating: build/lib/libnnpack.a   
2022-05-18T04:03:49.7087930Z   inflating: build/lib/libdnnl.a     
2022-05-18T04:03:49.7741310Z   inflating: build/lib/libtensorpipe.a  
2022-05-18T04:03:49.7784269Z   inflating: build/lib/libc10_cuda.so  
2022-05-18T04:03:49.9299502Z   inflating: build/lib/libfbgemm.a   
2022-05-18T04:03:49.9300346Z   inflating: build/lib/libgmock_main.a  
2022-05-18T04:03:50.0426962Z   inflating: build/lib/libdnnl_graph.a  
2022-05-18T04:03:50.0853602Z   inflating: build/lib/libkineto.a   
2022-05-18T04:03:50.1142891Z   inflating: build/lib/libtensorpipe_cuda.a  
2022-05-18T04:03:50.1188239Z   inflating: build/lib/libcaffe2_protos.a  
2022-05-18T04:03:50.1236112Z   inflating: build/lib/libonnx_proto.a  
2022-05-18T04:03:50.1377519Z   inflating: build/lib/libXNNPACK.a  
2022-05-18T04:03:50.2039895Z   inflating: build/lib/libonnx.a     
2022-05-18T04:03:50.2468390Z   inflating: build/lib/libgloo_cuda.a  
2022-05-18T04:03:52.3400346Z   inflating: build/lib/libtorch_cpu.so  
2022-05-18T04:03:54.3223980Z   inflating: build/lib/libtorch_cuda.so  
2022-05-18T04:03:54.3224960Z   inflating: build/lib/libtorch.so   
2022-05-18T04:03:54.3228978Z   inflating: build/lib/libc10d_cuda_test.so  
2022-05-18T04:03:54.9361412Z   inflating: build/lib/libtorch_cuda_linalg.so  
2022-05-18T04:03:54.9384691Z   inflating: build/lib/libjitbackend_test.so  
2022-05-18T04:03:54.9415505Z   inflating: build/lib/libbackend_with_compiler.so  
2022-05-18T04:03:54.9468340Z   inflating: build/lib/libtorchbind_test.so  
2022-05-18T04:03:54.9473061Z   inflating: build/lib/libshm.so     
2022-05-18T04:03:55.1050286Z   inflating: build/lib/libtorch_python.so  
2022-05-18T04:03:55.1088267Z   inflating: build/lib/libnnapi_backend.so  
2022-05-18T04:03:55.1088574Z    creating: build/bin/
2022-05-18T04:03:55.1140896Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-05-18T04:03:55.1195983Z   inflating: build/bin/c10_DeviceGuard_test  
2022-05-18T04:03:55.1249487Z   inflating: build/bin/c10_Device_test  
2022-05-18T04:03:55.1311275Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-05-18T04:03:55.1362450Z   inflating: build/bin/c10_StreamGuard_test  
2022-05-18T04:03:55.1421649Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-05-18T04:03:55.1481333Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-05-18T04:03:55.1542153Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-05-18T04:03:55.1593345Z   inflating: build/bin/c10_Array_test  
2022-05-18T04:03:55.1649978Z   inflating: build/bin/c10_Bitset_test  
2022-05-18T04:03:55.1704371Z   inflating: build/bin/c10_C++17_test  
2022-05-18T04:03:55.1825184Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-05-18T04:03:55.1877602Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-05-18T04:03:55.1930734Z   inflating: build/bin/c10_Half_test  
2022-05-18T04:03:55.1991187Z   inflating: build/bin/c10_LeftRight_test  
2022-05-18T04:03:55.2058429Z   inflating: build/bin/c10_Metaprogramming_test  
2022-05-18T04:03:55.2214477Z   inflating: build/bin/c10_SmallVectorTest  
2022-05-18T04:03:55.2267913Z   inflating: build/bin/c10_Synchronized_test  
2022-05-18T04:03:55.2329304Z   inflating: build/bin/c10_ThreadLocal_test  
2022-05-18T04:03:55.2385337Z   inflating: build/bin/c10_TypeIndex_test  
2022-05-18T04:03:55.2439340Z   inflating: build/bin/c10_TypeList_test  
2022-05-18T04:03:55.2491375Z   inflating: build/bin/c10_TypeTraits_test  
2022-05-18T04:03:55.2546158Z   inflating: build/bin/c10_accumulate_test  
2022-05-18T04:03:55.2606456Z   inflating: build/bin/c10_bfloat16_test  
2022-05-18T04:03:55.2664147Z   inflating: build/bin/c10_complex_math_test  
2022-05-18T04:03:55.2723547Z   inflating: build/bin/c10_complex_test  
2022-05-18T04:03:55.2841228Z   inflating: build/bin/c10_either_test  
2022-05-18T04:03:55.2897182Z   inflating: build/bin/c10_exception_test  
2022-05-18T04:03:55.2950497Z   inflating: build/bin/c10_flags_test  
2022-05-18T04:03:55.3132474Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-05-18T04:03:55.3186299Z   inflating: build/bin/c10_irange_test  
2022-05-18T04:03:55.3248079Z   inflating: build/bin/c10_logging_test  
2022-05-18T04:03:55.3314472Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-05-18T04:03:55.3394541Z   inflating: build/bin/c10_optional_test  
2022-05-18T04:03:55.3453336Z   inflating: build/bin/c10_registry_test  
2022-05-18T04:03:55.3516485Z   inflating: build/bin/c10_string_view_test  
2022-05-18T04:03:55.3571955Z   inflating: build/bin/c10_tempfile_test  
2022-05-18T04:03:55.3632996Z   inflating: build/bin/c10_typeid_test  
2022-05-18T04:03:55.3692431Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-05-18T04:03:55.4214989Z   inflating: build/bin/protoc-3.13.0.0  
2022-05-18T04:03:55.4737202Z   inflating: build/bin/protoc        
2022-05-18T04:03:55.4789149Z   inflating: build/bin/c10_cuda_CUDATest  
2022-05-18T04:03:55.5107724Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-05-18T04:03:55.5462255Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-05-18T04:03:55.5519801Z   inflating: build/bin/HashStoreTest  
2022-05-18T04:03:55.5577316Z   inflating: build/bin/FileStoreTest  
2022-05-18T04:03:55.5642131Z   inflating: build/bin/TCPStoreTest  
2022-05-18T04:03:55.5657655Z   inflating: build/bin/ProcessGroupMPITest  
2022-05-18T04:03:55.5660755Z   inflating: build/bin/example_allreduce  
2022-05-18T04:03:55.5716659Z   inflating: build/bin/Dimname_test  
2022-05-18T04:03:55.5778327Z   inflating: build/bin/scalar_test   
2022-05-18T04:03:55.5842412Z   inflating: build/bin/apply_utils_test  
2022-05-18T04:03:55.5906128Z   inflating: build/bin/basic         
2022-05-18T04:03:55.5969972Z   inflating: build/bin/atest         
2022-05-18T04:03:55.6031427Z   inflating: build/bin/NamedTensor_test  
2022-05-18T04:03:55.6089186Z   inflating: build/bin/broadcast_test  
2022-05-18T04:03:55.6143468Z   inflating: build/bin/wrapdim_test  
2022-05-18T04:03:55.6222268Z   inflating: build/bin/Dict_test     
2022-05-18T04:03:55.6275077Z   inflating: build/bin/dlconvertor_test  
2022-05-18T04:03:55.6334970Z   inflating: build/bin/half_test     
2022-05-18T04:03:55.6394537Z   inflating: build/bin/native_test   
2022-05-18T04:03:55.6396129Z   inflating: build/bin/verify_api_visibility  
2022-05-18T04:03:55.6451970Z   inflating: build/bin/undefined_tensor_test  
2022-05-18T04:03:55.6454617Z   inflating: build/bin/thread_init_test  
2022-05-18T04:03:55.6514857Z   inflating: build/bin/scalar_tensor_test  
2022-05-18T04:03:55.6575243Z   inflating: build/bin/test_parallel  
2022-05-18T04:03:55.6629641Z   inflating: build/bin/weakref_test  
2022-05-18T04:03:55.6682251Z   inflating: build/bin/lazy_tensor_test  
2022-05-18T04:03:55.6743314Z   inflating: build/bin/quantized_test  
2022-05-18T04:03:55.6796951Z   inflating: build/bin/operators_test  
2022-05-18T04:03:55.6857329Z   inflating: build/bin/extension_backend_test  
2022-05-18T04:03:55.6913554Z   inflating: build/bin/math_kernel_test  
2022-05-18T04:03:55.6968669Z   inflating: build/bin/memory_overlapping_test  
2022-05-18T04:03:55.7021473Z   inflating: build/bin/variant_test  
2022-05-18T04:03:55.7105090Z   inflating: build/bin/tensor_iterator_test  
2022-05-18T04:03:55.7160658Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-05-18T04:03:55.7223752Z   inflating: build/bin/cpu_generator_test  
2022-05-18T04:03:55.7277915Z   inflating: build/bin/reportMemoryUsage_test  
2022-05-18T04:03:55.7330642Z   inflating: build/bin/reduce_ops_test  
2022-05-18T04:03:55.7386559Z   inflating: build/bin/memory_format_test  
2022-05-18T04:03:55.7456027Z   inflating: build/bin/pow_test      
2022-05-18T04:03:55.7511889Z   inflating: build/bin/mobile_memory_cleanup  
2022-05-18T04:03:55.7565120Z   inflating: build/bin/dispatch_key_set_test  
2022-05-18T04:03:55.7629122Z   inflating: build/bin/IListRef_test  
2022-05-18T04:03:55.7747451Z   inflating: build/bin/List_test     
2022-05-18T04:03:55.7802822Z   inflating: build/bin/stride_properties_test  
2022-05-18T04:03:55.7875602Z   inflating: build/bin/vmap_test     
2022-05-18T04:03:55.8003499Z   inflating: build/bin/kernel_function_legacy_test  
2022-05-18T04:03:55.8106273Z   inflating: build/bin/kernel_function_test  
2022-05-18T04:03:55.8241190Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-05-18T04:03:55.8352353Z   inflating: build/bin/kernel_lambda_test  
2022-05-18T04:03:55.8416718Z   inflating: build/bin/kernel_stackbased_test  
2022-05-18T04:03:55.8518569Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-05-18T04:03:55.8572855Z   inflating: build/bin/CppSignature_test  
2022-05-18T04:03:55.8624011Z   inflating: build/bin/op_allowlist_test  
2022-05-18T04:03:55.8930024Z   inflating: build/bin/op_registration_test  
2022-05-18T04:03:55.9024217Z   inflating: build/bin/cpu_rng_test  
2022-05-18T04:03:55.9081083Z   inflating: build/bin/inline_container_test  
2022-05-18T04:03:55.9149498Z   inflating: build/bin/KernelFunction_test  
2022-05-18T04:03:55.9214450Z   inflating: build/bin/type_test     
2022-05-18T04:03:55.9276950Z   inflating: build/bin/cuda_atomic_ops_test  
2022-05-18T04:03:55.9377810Z   inflating: build/bin/ivalue_test   
2022-05-18T04:03:55.9451533Z   inflating: build/bin/cuda_complex_math_test  
2022-05-18T04:03:55.9514729Z   inflating: build/bin/cuda_complex_test  
2022-05-18T04:03:55.9570633Z   inflating: build/bin/cuda_apply_test  
2022-05-18T04:03:55.9625299Z   inflating: build/bin/cuda_integer_divider_test  
2022-05-18T04:03:55.9691296Z   inflating: build/bin/cuda_stream_test  
2022-05-18T04:03:55.9751367Z   inflating: build/bin/backend_fallback_test  
2022-05-18T04:03:55.9808820Z   inflating: build/bin/cuda_caching_host_allocator_test  
2022-05-18T04:03:55.9864839Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2022-05-18T04:03:55.9918122Z   inflating: build/bin/cuda_dlconvertor_test  
2022-05-18T04:03:55.9971409Z   inflating: build/bin/cuda_half_test  
2022-05-18T04:03:56.0027078Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2022-05-18T04:03:56.0112465Z   inflating: build/bin/cuda_cub_test  
2022-05-18T04:03:56.0164399Z   inflating: build/bin/cuda_optional_test  
2022-05-18T04:03:56.0227466Z   inflating: build/bin/cuda_distributions_test  
2022-05-18T04:03:56.0284654Z   inflating: build/bin/cuda_vectorized_test  
2022-05-18T04:03:56.0336612Z   inflating: build/bin/cuda_cudnn_test  
2022-05-18T04:03:56.0399587Z   inflating: build/bin/cuda_generator_test  
2022-05-18T04:03:56.0468787Z   inflating: build/bin/ProcessGroupGlooTest  
2022-05-18T04:03:56.0531775Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2022-05-18T04:03:56.0597953Z   inflating: build/bin/ProcessGroupNCCLTest  
2022-05-18T04:03:56.0615382Z   inflating: build/bin/tutorial_tensorexpr  
2022-05-18T04:03:56.0678360Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2022-05-18T04:03:56.0735536Z   inflating: build/bin/test_dist_autograd  
2022-05-18T04:03:56.0809827Z   inflating: build/bin/test_cpp_rpc  
2022-05-18T04:03:56.0883615Z   inflating: build/bin/test_mobile_nnc  
2022-05-18T04:03:56.0886394Z   inflating: build/bin/parallel_benchmark  
2022-05-18T04:03:56.0897581Z   inflating: build/bin/aot_model_compiler_test  
2022-05-18T04:03:56.1804935Z   inflating: build/bin/test_tensorexpr  
2022-05-18T04:03:56.2185441Z   inflating: build/bin/test_lazy     
2022-05-18T04:03:56.2191021Z   inflating: build/bin/torch_shm_manager  
2022-05-18T04:03:56.2323368Z   inflating: build/bin/nvfuser_bench  
2022-05-18T04:03:56.3617539Z   inflating: build/bin/test_api      
2022-05-18T04:03:56.4571285Z   inflating: build/bin/test_jit      
2022-05-18T04:03:56.4572508Z   inflating: .pytorch-test-times.json  
2022-05-18T04:03:56.4603122Z ##[group]Run df -H
2022-05-18T04:03:56.4603399Z df -H
2022-05-18T04:03:56.4616792Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:03:56.4617096Z env:
2022-05-18T04:03:56.4617321Z   IN_CI: 1
2022-05-18T04:03:56.4617529Z   IS_GHA: 1
2022-05-18T04:03:56.4617780Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:03:56.4618048Z   GPU_FLAG: --gpus all
2022-05-18T04:03:56.4618284Z ##[endgroup]
2022-05-18T04:03:56.4657076Z Filesystem      Size  Used Avail Use% Mounted on
2022-05-18T04:03:56.4657436Z devtmpfs        129G     0  129G   0% /dev
2022-05-18T04:03:56.4657714Z tmpfs           129G  6.8M  129G   1% /dev/shm
2022-05-18T04:03:56.4658000Z tmpfs           129G  590k  129G   1% /run
2022-05-18T04:03:56.4658643Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2022-05-18T04:03:56.4658978Z /dev/xvda1      162G   22G  140G  14% /
2022-05-18T04:03:56.4659269Z tmpfs            26G     0   26G   0% /run/user/0
2022-05-18T04:03:56.4680866Z ##[group]Run .github/scripts/parse_ref.py
2022-05-18T04:03:56.4681237Z .github/scripts/parse_ref.py
2022-05-18T04:03:56.4693229Z shell: /usr/bin/bash -e {0}
2022-05-18T04:03:56.4693473Z env:
2022-05-18T04:03:56.4693693Z   IN_CI: 1
2022-05-18T04:03:56.4693921Z   IS_GHA: 1
2022-05-18T04:03:56.4694156Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:03:56.4694430Z   GPU_FLAG: --gpus all
2022-05-18T04:03:56.4694682Z ##[endgroup]
2022-05-18T04:03:56.5043616Z ##[group]Run set -x
2022-05-18T04:03:56.5044016Z set -x
2022-05-18T04:03:56.5044250Z 
2022-05-18T04:03:56.5044529Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-05-18T04:03:56.5044874Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-05-18T04:03:56.5045232Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-05-18T04:03:56.5045561Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-05-18T04:03:56.5045821Z else
2022-05-18T04:03:56.5046106Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-05-18T04:03:56.5046384Z fi
2022-05-18T04:03:56.5046616Z 
2022-05-18T04:03:56.5046927Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-05-18T04:03:56.5047284Z export COMMIT_MESSAGES
2022-05-18T04:03:56.5047539Z 
2022-05-18T04:03:56.5047836Z # detached container should get cleaned up by teardown_ec2_linux
2022-05-18T04:03:56.5048277Z # TODO: Stop building test binaries as part of the build phase
2022-05-18T04:03:56.5048666Z # Used for GPU_FLAG since that doesn't play nice
2022-05-18T04:03:56.5048984Z # shellcheck disable=SC2086,SC2090
2022-05-18T04:03:56.5049293Z container_name=$(docker run \
2022-05-18T04:03:56.5049576Z   ${GPU_FLAG:-} \
2022-05-18T04:03:56.5049834Z   -e BUILD_ENVIRONMENT \
2022-05-18T04:03:56.5050110Z   -e PR_NUMBER \
2022-05-18T04:03:56.5050406Z   -e CUSTOM_TEST_ARTIFACT_BUILD_DIR \
2022-05-18T04:03:56.5050686Z   -e GITHUB_ACTIONS \
2022-05-18T04:03:56.5051071Z   -e IN_CI \
2022-05-18T04:03:56.5051319Z   -e IS_GHA \
2022-05-18T04:03:56.5051551Z   -e BRANCH \
2022-05-18T04:03:56.5051799Z   -e SHA1 \
2022-05-18T04:03:56.5052062Z   -e AWS_DEFAULT_REGION \
2022-05-18T04:03:56.5052341Z   -e IN_WHEEL_TEST \
2022-05-18T04:03:56.5052596Z   -e SHARD_NUMBER \
2022-05-18T04:03:56.5052861Z   -e JOB_BASE_NAME \
2022-05-18T04:03:56.5053129Z   -e TEST_CONFIG \
2022-05-18T04:03:56.5053382Z   -e NUM_TEST_SHARDS \
2022-05-18T04:03:56.5053648Z   -e PR_BODY \
2022-05-18T04:03:56.5053914Z   -e COMMIT_MESSAGES \
2022-05-18T04:03:56.5054189Z   -e PYTORCH_RETRY_TEST_CASES \
2022-05-18T04:03:56.5054472Z   -e PR_LABELS \
2022-05-18T04:03:56.5054768Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-05-18T04:03:56.5055048Z   -e SCCACHE_BUCKET \
2022-05-18T04:03:56.5055312Z   -e XLA_CUDA \
2022-05-18T04:03:56.5055601Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-05-18T04:03:56.5055930Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-05-18T04:03:56.5056259Z   --ulimit stack=10485760:83886080 \
2022-05-18T04:03:56.5056575Z   --security-opt seccomp=unconfined \
2022-05-18T04:03:56.5056888Z   --cap-add=SYS_PTRACE \
2022-05-18T04:03:56.5057149Z   --ipc=host \
2022-05-18T04:03:56.5057422Z   --shm-size="${SHM_SIZE}" \
2022-05-18T04:03:56.5057691Z   --tty \
2022-05-18T04:03:56.5057921Z   --detach \
2022-05-18T04:03:56.5058193Z   --name="${container_name}" \
2022-05-18T04:03:56.5058471Z   --user jenkins \
2022-05-18T04:03:56.5058780Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-05-18T04:03:56.5059127Z   -w /var/lib/jenkins/workspace \
2022-05-18T04:03:56.5059414Z   "${DOCKER_IMAGE}"
2022-05-18T04:03:56.5059643Z )
2022-05-18T04:03:56.5059988Z docker exec -t "${container_name}" sh -c "pip install dist/*.whl && ${TEST_COMMAND}"
2022-05-18T04:03:56.5072326Z shell: /usr/bin/bash -e {0}
2022-05-18T04:03:56.5072571Z env:
2022-05-18T04:03:56.5072792Z   IN_CI: 1
2022-05-18T04:03:56.5073018Z   IS_GHA: 1
2022-05-18T04:03:56.5073255Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:03:56.5073605Z   GPU_FLAG: --gpus all
2022-05-18T04:03:56.5073949Z   BUILD_ENVIRONMENT: linux-bionic-cuda10.2-py3.9-gcc7
2022-05-18T04:03:56.5074267Z   PR_NUMBER: 
2022-05-18T04:03:56.5074491Z   BRANCH: master
2022-05-18T04:03:56.5074802Z   CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts
2022-05-18T04:03:56.5075153Z   SHA1: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T04:03:56.5075438Z   PYTORCH_RETRY_TEST_CASES: 1
2022-05-18T04:03:56.5075776Z   JOB_BASE_NAME: linux-bionic-cuda10.2-py3.9-gcc7-test
2022-05-18T04:03:56.5076102Z   TEST_CONFIG: distributed
2022-05-18T04:03:56.5076345Z   SHARD_NUMBER: 1
2022-05-18T04:03:56.5076591Z   NUM_TEST_SHARDS: 2
2022-05-18T04:03:56.5076840Z   PR_BODY: 
2022-05-18T04:03:56.5077130Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-05-18T04:03:56.5077436Z   SHM_SIZE: 2g
2022-05-18T04:03:56.5077941Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:03:56.5078688Z   XLA_CUDA: 
2022-05-18T04:03:56.5079028Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-05-18T04:03:56.5079386Z ##[endgroup]
2022-05-18T04:03:56.5107096Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-05-18T04:03:56.5107591Z + [[ linux-bionic-cuda10.2-py3.9-gcc7 == *onnx* ]]
2022-05-18T04:03:56.5107926Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-05-18T04:03:56.5110767Z ++ git cherry -v origin/master
2022-05-18T04:03:56.5144641Z + COMMIT_MESSAGES=
2022-05-18T04:03:56.5144904Z + export COMMIT_MESSAGES
2022-05-18T04:03:56.5154052Z +++ nproc --ignore=2
2022-05-18T04:03:56.5167236Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e CUSTOM_TEST_ARTIFACT_BUILD_DIR -e GITHUB_ACTIONS -e IN_CI -e IS_GHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e JOB_BASE_NAME -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_2342799949 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:04:20.7472941Z + container_name=4e1873759d184a295db3cfb7e4bbb284ef9fc37ca3243b456bdcb3bd87dc97d9
2022-05-18T04:04:20.7475135Z + docker exec -t 4e1873759d184a295db3cfb7e4bbb284ef9fc37ca3243b456bdcb3bd87dc97d9 sh -c 'pip install dist/*.whl && .jenkins/pytorch/test.sh'
2022-05-18T04:04:21.2453083Z Processing ./dist/torch-1.12.0a0+git3b23752-cp39-cp39-linux_x86_64.whl
2022-05-18T04:04:21.3411663Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.9/site-packages (from torch==1.12.0a0+git3b23752) (4.2.0)
2022-05-18T04:04:21.8803790Z Installing collected packages: torch
2022-05-18T04:04:31.3498328Z Successfully installed torch-1.12.0a0+git3b23752
2022-05-18T04:04:31.4079644Z + COMPACT_JOB_NAME=linux-bionic-cuda10.2-py3.9-gcc7
2022-05-18T04:04:31.4082095Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-05-18T04:04:31.4364724Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.9/site-packages/torch
2022-05-18T04:04:31.4365241Z + TORCH_BIN_DIR=/opt/conda/lib/python3.9/site-packages/torch/bin
2022-05-18T04:04:31.4369928Z + TORCH_LIB_DIR=/opt/conda/lib/python3.9/site-packages/torch/lib
2022-05-18T04:04:31.4370500Z + TORCH_TEST_DIR=/opt/conda/lib/python3.9/site-packages/torch/test
2022-05-18T04:04:31.4370862Z + BUILD_DIR=build
2022-05-18T04:04:31.4371141Z + BUILD_RENAMED_DIR=build_renamed
2022-05-18T04:04:31.4371408Z + BUILD_BIN_DIR=build/bin
2022-05-18T04:04:31.4371716Z + [[ -n distributed ]]
2022-05-18T04:04:31.4372156Z + BUILD_ENVIRONMENT=linux-bionic-cuda10.2-py3.9-gcc7-distributed
2022-05-18T04:04:31.4372861Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed != *bazel* ]]
2022-05-18T04:04:31.4374128Z ++ realpath build/custom_test_artifacts
2022-05-18T04:04:31.4379331Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-05-18T04:04:31.4383228Z ++ dirname .jenkins/pytorch/test.sh
2022-05-18T04:04:31.4390351Z + source .jenkins/pytorch/common.sh
2022-05-18T04:04:31.4394633Z +++ dirname .jenkins/pytorch/common.sh
2022-05-18T04:04:31.4405289Z ++ source .jenkins/pytorch/common_utils.sh
2022-05-18T04:04:31.4408627Z +++ TORCHVISION_COMMIT=8a2dc6f22ac4389ccba8859aa1e1cb14f1ee53db
2022-05-18T04:04:31.4409512Z ++ set -ex
2022-05-18T04:04:31.4416799Z ++++ dirname .jenkins/pytorch/common.sh
2022-05-18T04:04:31.4427221Z +++ cd .jenkins/pytorch
2022-05-18T04:04:31.4427521Z +++ pwd -P
2022-05-18T04:04:31.4430796Z ++ SCRIPT_DIR=/var/lib/jenkins/workspace/.jenkins/pytorch
2022-05-18T04:04:31.4431290Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *linux* ]]
2022-05-18T04:04:31.4435802Z +++ find /etc/apt/ -type f -name '*.list'
2022-05-18T04:04:31.4455840Z ++ sudo sed -i 's/.*nvidia.*/# &/' /etc/apt/sources.list /etc/apt/sources.list.d/cuda.list /etc/apt/sources.list.d/nvidia-ml.list /etc/apt/sources.list.d/ubuntu-toolchain-r-ubuntu-test-bionic.list
2022-05-18T04:04:31.4521789Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *rocm* ]]
2022-05-18T04:04:31.4522155Z ++ echo ENTERED_USER_LAND
2022-05-18T04:04:31.4522423Z ENTERED_USER_LAND
2022-05-18T04:04:31.4522677Z ++ export IN_CI=1
2022-05-18T04:04:31.4522897Z ++ IN_CI=1
2022-05-18T04:04:31.4523933Z ++ declare -f -t trap_add
2022-05-18T04:04:31.4524210Z ++ trap_add cleanup EXIT
2022-05-18T04:04:31.4524677Z ++ trap_add_cmd=cleanup
2022-05-18T04:04:31.4524907Z ++ shift
2022-05-18T04:04:31.4525154Z ++ for trap_add_name in "$@"
2022-05-18T04:04:31.4533260Z ++++ trap -p EXIT
2022-05-18T04:04:31.4536187Z +++ eval 'extract_trap_cmd '
2022-05-18T04:04:31.4536469Z ++++ extract_trap_cmd
2022-05-18T04:04:31.4536767Z ++++ printf '%s\n' ''
2022-05-18T04:04:31.4537058Z +++ printf '%s\n' cleanup
2022-05-18T04:04:31.4539873Z ++ trap -- '
2022-05-18T04:04:31.4540160Z cleanup' EXIT
2022-05-18T04:04:31.4542741Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed != *win-* ]]
2022-05-18T04:04:31.4543074Z ++ which sccache
2022-05-18T04:04:31.4553742Z ++ sccache --stop-server
2022-05-18T04:04:31.4581643Z ++ true
2022-05-18T04:04:31.4581987Z ++ rm -f /var/lib/jenkins/sccache_error.log
2022-05-18T04:04:31.4590739Z ++ [[ -n '' ]]
2022-05-18T04:04:31.4591178Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *rocm* ]]
2022-05-18T04:04:31.4591566Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
2022-05-18T04:04:31.4591892Z ++ SCCACHE_IDLE_TIMEOUT=1200
2022-05-18T04:04:31.4608585Z ++ RUST_LOG=sccache::server=error
2022-05-18T04:04:31.4608989Z ++ sccache --start-server
2022-05-18T04:04:31.4612798Z sccache: Starting the server...
2022-05-18T04:04:31.4802235Z ++ sccache --zero-stats
2022-05-18T04:04:31.4823585Z Compile requests                      0
2022-05-18T04:04:31.4823916Z Compile requests executed             0
2022-05-18T04:04:31.4824196Z Cache hits                            0
2022-05-18T04:04:31.4824484Z Cache misses                          0
2022-05-18T04:04:31.4824758Z Cache timeouts                        0
2022-05-18T04:04:31.4825019Z Cache read errors                     0
2022-05-18T04:04:31.4825302Z Forced recaches                       0
2022-05-18T04:04:31.4825582Z Cache write errors                    0
2022-05-18T04:04:31.4825852Z Compilation failures                  0
2022-05-18T04:04:31.4826136Z Cache errors                          0
2022-05-18T04:04:31.4826500Z Non-cacheable compilations            0
2022-05-18T04:04:31.4826855Z Non-cacheable calls                   0
2022-05-18T04:04:31.4827201Z Non-compilation calls                 0
2022-05-18T04:04:31.4827514Z Unsupported compiler calls            0
2022-05-18T04:04:31.4827820Z Average cache write               0.000 s
2022-05-18T04:04:31.4828099Z Average cache read miss           0.000 s
2022-05-18T04:04:31.4828561Z Average cache read hit            0.000 s
2022-05-18T04:04:31.4828899Z Failed distributed compilations       0
2022-05-18T04:04:31.4829620Z Cache location                  S3, bucket: Bucket(name=ossci-compiler-cache-circleci-v2, base_url=http://ossci-compiler-cache-circleci-v2.s3.amazonaws.com/)
2022-05-18T04:04:31.4830293Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7-test == *-build ]]
2022-05-18T04:04:31.4830613Z ++ which ccache
2022-05-18T04:04:31.4837686Z ++ '[' -z linux-bionic-cuda10.2-py3.9-gcc7 ']'
2022-05-18T04:04:31.4838567Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *linux-trusty-py3.6-gcc7* ]]
2022-05-18T04:04:31.4838981Z ++ BUILD_TEST_LIBTORCH=0
2022-05-18T04:04:31.4839271Z ++ [[ distributed == *xla* ]]
2022-05-18T04:04:31.4839698Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *centos* ]]
2022-05-18T04:04:31.4840234Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *linux-bionic* ]]
2022-05-18T04:04:31.4840587Z ++ which conda
2022-05-18T04:04:31.4849039Z /opt/conda/bin/conda
2022-05-18T04:04:31.4849711Z ++ conda install -q -y cmake
2022-05-18T04:04:37.3899063Z Collecting package metadata (current_repodata.json): ...working... done
2022-05-18T04:04:37.8755387Z Solving environment: ...working... done
2022-05-18T04:04:37.9599612Z 
2022-05-18T04:04:37.9600079Z ## Package Plan ##
2022-05-18T04:04:37.9600278Z 
2022-05-18T04:04:37.9600430Z   environment location: /opt/conda
2022-05-18T04:04:37.9600633Z 
2022-05-18T04:04:37.9600981Z   added / updated specs:
2022-05-18T04:04:37.9601679Z     - cmake
2022-05-18T04:04:37.9601950Z 
2022-05-18T04:04:37.9601979Z 
2022-05-18T04:04:37.9602296Z The following packages will be downloaded:
2022-05-18T04:04:37.9602890Z 
2022-05-18T04:04:37.9603021Z     package                    |            build
2022-05-18T04:04:37.9603420Z     ---------------------------|-----------------
2022-05-18T04:04:37.9603848Z     bzip2-1.0.8                |       h7b6447c_0          78 KB
2022-05-18T04:04:37.9604240Z     c-ares-1.18.1              |       h7f8727e_0         114 KB
2022-05-18T04:04:37.9604649Z     cmake-3.22.1               |       h1fce559_0         7.3 MB
2022-05-18T04:04:37.9605040Z     expat-2.4.4                |       h295c915_0         169 KB
2022-05-18T04:04:37.9605437Z     krb5-1.19.2                |       hac12032_0         1.2 MB
2022-05-18T04:04:37.9605818Z     libcurl-7.82.0             |       h0b77cf5_0         342 KB
2022-05-18T04:04:37.9606224Z     libedit-3.1.20210910       |       h7f8727e_0         166 KB
2022-05-18T04:04:37.9606616Z     libev-4.33                 |       h7f8727e_1         111 KB
2022-05-18T04:04:37.9606996Z     libnghttp2-1.46.0          |       hce63b2e_0         680 KB
2022-05-18T04:04:37.9607402Z     libssh2-1.10.0             |       h8f2d780_0         274 KB
2022-05-18T04:04:37.9607792Z     libuv-1.40.0               |       h7b6447c_0         736 KB
2022-05-18T04:04:37.9608157Z     lz4-c-1.9.3                |       h295c915_1         185 KB
2022-05-18T04:04:37.9608550Z     rhash-1.4.1                |       h3c74f83_1         203 KB
2022-05-18T04:04:37.9608937Z     zstd-1.5.2                 |       ha4553b6_0         488 KB
2022-05-18T04:04:37.9609351Z     ------------------------------------------------------------
2022-05-18T04:04:37.9609678Z                                            Total:        12.0 MB
2022-05-18T04:04:37.9609856Z 
2022-05-18T04:04:37.9610020Z The following NEW packages will be INSTALLED:
2022-05-18T04:04:37.9610245Z 
2022-05-18T04:04:37.9610597Z   bzip2              pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0
2022-05-18T04:04:37.9611086Z   c-ares             pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0
2022-05-18T04:04:37.9611548Z   cmake              pkgs/main/linux-64::cmake-3.22.1-h1fce559_0
2022-05-18T04:04:37.9612031Z   expat              pkgs/main/linux-64::expat-2.4.4-h295c915_0
2022-05-18T04:04:37.9612500Z   krb5               pkgs/main/linux-64::krb5-1.19.2-hac12032_0
2022-05-18T04:04:37.9612962Z   libcurl            pkgs/main/linux-64::libcurl-7.82.0-h0b77cf5_0
2022-05-18T04:04:37.9613611Z   libedit            pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0
2022-05-18T04:04:37.9614129Z   libev              pkgs/main/linux-64::libev-4.33-h7f8727e_1
2022-05-18T04:04:37.9614634Z   libnghttp2         pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0
2022-05-18T04:04:37.9615121Z   libssh2            pkgs/main/linux-64::libssh2-1.10.0-h8f2d780_0
2022-05-18T04:04:37.9615607Z   libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0
2022-05-18T04:04:37.9616082Z   lz4-c              pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1
2022-05-18T04:04:37.9616551Z   rhash              pkgs/main/linux-64::rhash-1.4.1-h3c74f83_1
2022-05-18T04:04:37.9617012Z   zstd               pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0
2022-05-18T04:04:37.9617221Z 
2022-05-18T04:04:37.9617509Z The following packages will be SUPERSEDED by a higher-priority channel:
2022-05-18T04:04:37.9617763Z 
2022-05-18T04:04:37.9618172Z   certifi            conda-forge::certifi-2021.10.8-py39hf~ --> pkgs/main::certifi-2021.10.8-py39h06a4308_2
2022-05-18T04:04:37.9618820Z   conda              conda-forge::conda-4.12.0-py39hf3d152~ --> pkgs/main::conda-4.12.0-py39h06a4308_0
2022-05-18T04:04:37.9619069Z 
2022-05-18T04:04:37.9619109Z 
2022-05-18T04:04:38.9766417Z Preparing transaction: ...working... done
2022-05-18T04:04:39.5002523Z Verifying transaction: ...working... done
2022-05-18T04:04:41.6845506Z Executing transaction: ...working... done
2022-05-18T04:04:42.4207311Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *centos* ]]
2022-05-18T04:04:42.4210542Z + echo 'Testing pytorch'
2022-05-18T04:04:42.4210977Z Testing pytorch
2022-05-18T04:04:42.4211259Z + export LANG=C.UTF-8
2022-05-18T04:04:42.4211760Z + LANG=C.UTF-8
2022-05-18T04:04:42.4213838Z + PR_NUMBER=
2022-05-18T04:04:42.4214307Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-05-18T04:04:42.4214631Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-05-18T04:04:42.4215315Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *rocm* ]]
2022-05-18T04:04:42.4216257Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *-slow-* ]]
2022-05-18T04:04:42.4216777Z + [[ distributed == \s\l\o\w ]]
2022-05-18T04:04:42.4217470Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *slow-gradcheck* ]]
2022-05-18T04:04:42.4218009Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *cuda* ]]
2022-05-18T04:04:42.4218674Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-05-18T04:04:42.4219211Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-05-18T04:04:42.4219681Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *cuda11* ]]
2022-05-18T04:04:42.4220205Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *crossref* ]]
2022-05-18T04:04:42.4220570Z + [[ -n '' ]]
2022-05-18T04:04:42.4220869Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=0
2022-05-18T04:04:42.4221204Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=0
2022-05-18T04:04:42.4221639Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *rocm* ]]
2022-05-18T04:04:42.4222160Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed != *ppc64le* ]]
2022-05-18T04:04:42.4222676Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed != *-bazel-* ]]
2022-05-18T04:04:42.4223057Z + pip_install --user ninja
2022-05-18T04:04:42.4223433Z + pip install --progress-bar off --user ninja
2022-05-18T04:04:42.9488851Z Collecting ninja
2022-05-18T04:04:42.9700237Z   Downloading ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2022-05-18T04:04:42.9772563Z [?25l
2022-05-18T04:04:43.4771395Z [?25hInstalling collected packages: ninja
2022-05-18T04:04:43.4885411Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-05-18T04:04:43.4886086Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-05-18T04:04:43.4944830Z Successfully installed ninja-1.10.2.3
2022-05-18T04:04:43.5481541Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-05-18T04:04:43.5482232Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-05-18T04:04:43.5483979Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *asan* ]]
2022-05-18T04:04:43.5484624Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *-NO_AVX-* ]]
2022-05-18T04:04:43.5485033Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X ]]
2022-05-18T04:04:43.5485503Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *-NO_AVX2-* ]]
2022-05-18T04:04:43.5485877Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-05-18T04:04:43.5486371Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *-NO_AVX512-* ]]
2022-05-18T04:04:43.5486777Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\5\1\2 ]]
2022-05-18T04:04:43.5489448Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *tbb* ]]
2022-05-18T04:04:43.5503186Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *libtorch* ]]
2022-05-18T04:04:43.5503747Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *-bazel-* ]]
2022-05-18T04:04:43.5506257Z + cd test
2022-05-18T04:04:43.5506782Z + python -c 'import torch; print(torch.__config__.show())'
2022-05-18T04:04:48.0876469Z PyTorch built with:
2022-05-18T04:04:48.0877026Z   - GCC 7.5
2022-05-18T04:04:48.0877504Z   - C++ Version: 201402
2022-05-18T04:04:48.0878390Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-05-18T04:04:48.0878998Z   - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-05-18T04:04:48.0879705Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-05-18T04:04:48.0880087Z   - LAPACK is enabled (usually provided by MKL)
2022-05-18T04:04:48.0880435Z   - NNPACK is enabled
2022-05-18T04:04:48.0880752Z   - CPU capability usage: AVX2
2022-05-18T04:04:48.0881044Z   - CUDA Runtime 10.2
2022-05-18T04:04:48.0881445Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2022-05-18T04:04:48.0881792Z   - CuDNN 7.6.5
2022-05-18T04:04:48.0882045Z   - Magma 2.5.2
2022-05-18T04:04:48.0885008Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-05-18T04:04:48.0887244Z 
2022-05-18T04:04:48.6856209Z + cd test
2022-05-18T04:04:48.6856790Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-05-18T04:04:49.4936079Z ATen/Parallel:
2022-05-18T04:04:49.4936465Z 	at::get_num_threads() : 16
2022-05-18T04:04:49.4936782Z 	at::get_num_interop_threads() : 16
2022-05-18T04:04:49.4937110Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-05-18T04:04:49.4937381Z 	omp_get_max_threads() : 16
2022-05-18T04:04:49.4938034Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-05-18T04:04:49.4938433Z 	mkl_get_max_threads() : 16
2022-05-18T04:04:49.4939168Z Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-05-18T04:04:49.4939569Z std::thread::hardware_concurrency() : 32
2022-05-18T04:04:49.4939867Z Environment variables:
2022-05-18T04:04:49.4940125Z 	OMP_NUM_THREADS : [not set]
2022-05-18T04:04:49.4940401Z 	MKL_NUM_THREADS : [not set]
2022-05-18T04:04:49.4940683Z ATen parallel backend: OpenMP
2022-05-18T04:04:49.4940866Z 
2022-05-18T04:04:49.6010974Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *deploy* ]]
2022-05-18T04:04:49.6011598Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *backward* ]]
2022-05-18T04:04:49.6011965Z + [[ distributed == *xla* ]]
2022-05-18T04:04:49.6012440Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *jit_legacy-test ]]
2022-05-18T04:04:49.6012946Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-test == *jit_legacy-test ]]
2022-05-18T04:04:49.6013321Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-05-18T04:04:49.6013783Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *libtorch* ]]
2022-05-18T04:04:49.6014301Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *distributed* ]]
2022-05-18T04:04:49.6014657Z + test_distributed
2022-05-18T04:04:49.6014998Z + echo 'Testing distributed python tests'
2022-05-18T04:04:49.6015294Z Testing distributed python tests
2022-05-18T04:04:49.6015728Z + python test/run_test.py --distributed-tests --shard 1 2 --verbose
2022-05-18T04:04:55.8902988Z Ignoring disabled issues:  []
2022-05-18T04:04:55.9034869Z /var/lib/jenkins/workspace/test/run_test.py:894: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-05-18T04:04:55.9035469Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) == "11.6":
2022-05-18T04:04:55.9107914Z Found stats for current commit: 3b2375291aab7b48442f2e6fb1ef66cebc761e24 and job: linux-bionic-cuda10.2-py3.9-gcc7. Proceeding with those values.
2022-05-18T04:04:55.9110229Z Selected tests:
2022-05-18T04:04:55.9110554Z  distributed/test_distributed_spawn
2022-05-18T04:04:55.9110910Z  distributed/fsdp/test_fsdp_state_dict
2022-05-18T04:04:55.9111240Z  distributed/fsdp/test_fsdp_optim_state
2022-05-18T04:04:55.9111564Z  distributed/test_store
2022-05-18T04:04:55.9111886Z  distributed/fsdp/test_fsdp_clip_grad_norm
2022-05-18T04:04:55.9112215Z  distributed/fsdp/test_wrap
2022-05-18T04:04:55.9112525Z  distributed/fsdp/test_fsdp_freezing_weights
2022-05-18T04:04:55.9112864Z  distributed/test_c10d_spawn_nccl
2022-05-18T04:04:55.9113180Z  distributed/test_c10d_common
2022-05-18T04:04:55.9113477Z  distributed/fsdp/test_fsdp_exec_order
2022-05-18T04:04:55.9113799Z  distributed/fsdp/test_fsdp_meta
2022-05-18T04:04:55.9114154Z  distributed/elastic/multiprocessing/api_test
2022-05-18T04:04:55.9114489Z  distributed/fsdp/test_fsdp_checkpoint
2022-05-18T04:04:55.9114820Z  distributed/fsdp/test_fsdp_overlap
2022-05-18T04:04:55.9115292Z  distributed/fsdp/test_fsdp_ignored_modules
2022-05-18T04:04:55.9115629Z  distributed/fsdp/test_fsdp_apply
2022-05-18T04:04:55.9115947Z  distributed/fsdp/test_distributed_checkpoint
2022-05-18T04:04:55.9116331Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-05-18T04:04:55.9116715Z  distributed/_shard/sharding_spec/test_sharding_spec
2022-05-18T04:04:55.9117092Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-05-18T04:04:55.9117473Z  distributed/_shard/sharded_tensor/ops/test_init
2022-05-18T04:04:55.9117832Z  distributed/elastic/utils/distributed_test
2022-05-18T04:04:55.9118696Z  distributed/fsdp/test_fsdp_multiple_wrapping
2022-05-18T04:04:55.9119079Z  distributed/fsdp/test_fsdp_multiple_forward
2022-05-18T04:04:55.9119454Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-05-18T04:04:55.9119840Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2022-05-18T04:04:55.9120233Z  distributed/_shard/sharded_optim/test_sharded_optim
2022-05-18T04:04:55.9120625Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-05-18T04:04:55.9120965Z  distributed/test_launcher
2022-05-18T04:04:55.9121438Z  distributed/elastic/utils/util_test
2022-05-18T04:04:55.9121784Z  distributed/fsdp/test_utils
2022-05-18T04:04:55.9122097Z  distributed/nn/jit/test_instantiator
2022-05-18T04:04:55.9122427Z  distributed/_shard/sharding_plan/test_sharding_plan
2022-05-18T04:04:55.9122770Z  distributed/_shard/test_sharder
2022-05-18T04:04:55.9123088Z  distributed/elastic/timer/api_test
2022-05-18T04:04:55.9123400Z  distributed/pipeline/sync/skip/test_api
2022-05-18T04:04:55.9123773Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-05-18T04:04:55.9124147Z  distributed/pipeline/sync/skip/test_portal
2022-05-18T04:04:55.9124478Z  distributed/pipeline/sync/skip/test_tracker
2022-05-18T04:04:55.9124831Z  distributed/pipeline/sync/test_balance
2022-05-18T04:04:55.9125177Z  distributed/pipeline/sync/test_checkpoint
2022-05-18T04:04:55.9125517Z  distributed/pipeline/sync/test_deferred_batch_norm
2022-05-18T04:04:55.9125871Z  distributed/pipeline/sync/test_inplace
2022-05-18T04:04:55.9126209Z  distributed/pipeline/sync/test_phony
2022-05-18T04:04:55.9126523Z  distributed/pipeline/sync/test_pipeline
2022-05-18T04:04:55.9126879Z  distributed/pipeline/sync/test_transparency
2022-05-18T04:04:55.9127219Z  distributed/rpc/test_faulty_agent
2022-05-18T04:04:55.9189326Z Prioritized test from test file changes.
2022-05-18T04:04:55.9189638Z reordering tests for PR:
2022-05-18T04:04:55.9189927Z prioritized: []
2022-05-18T04:04:55.9194504Z the rest: ['distributed/test_distributed_spawn', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/fsdp/test_fsdp_optim_state', 'distributed/test_store', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/fsdp/test_wrap', 'distributed/fsdp/test_fsdp_freezing_weights', 'distributed/test_c10d_spawn_nccl', 'distributed/test_c10d_common', 'distributed/fsdp/test_fsdp_exec_order', 'distributed/fsdp/test_fsdp_meta', 'distributed/elastic/multiprocessing/api_test', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/fsdp/test_fsdp_overlap', 'distributed/fsdp/test_fsdp_ignored_modules', 'distributed/fsdp/test_fsdp_apply', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops', 'distributed/_shard/sharded_tensor/ops/test_init', 'distributed/elastic/utils/distributed_test', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard', 'distributed/_shard/sharded_optim/test_sharded_optim', 'distributed/_shard/sharded_tensor/test_megatron_prototype', 'distributed/test_launcher', 'distributed/elastic/utils/util_test', 'distributed/fsdp/test_utils', 'distributed/nn/jit/test_instantiator', 'distributed/_shard/sharding_plan/test_sharding_plan', 'distributed/_shard/test_sharder', 'distributed/elastic/timer/api_test', 'distributed/pipeline/sync/skip/test_api', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_transparency', 'distributed/rpc/test_faulty_agent']
2022-05-18T04:04:55.9197599Z 
2022-05-18T04:04:55.9696683Z Running distributed/test_distributed_spawn ... [2022-05-18 04:04:55.969256]
2022-05-18T04:04:55.9741728Z /usr/bin/mpiexec
2022-05-18T04:04:55.9742951Z MPI not available -- MPI backend tests will be skipped
2022-05-18T04:04:55.9748572Z Running distributed tests for the test backend with env init_method
2022-05-18T04:04:55.9753015Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:04:55.974949]
2022-05-18T04:04:57.1254453Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:04:57.2496756Z Running distributed tests for the test backend with file init_method
2022-05-18T04:04:57.2499124Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:04:57.249577]
2022-05-18T04:04:58.3893634Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:04:58.5091941Z Running distributed tests for the nccl backend with env init_method
2022-05-18T04:04:58.5094356Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:04:58.509087]
2022-05-18T04:04:59.6495588Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:04:59.6537794Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-05-18T04:04:59.6572281Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6572805Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6573281Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6573706Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6574161Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6574642Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6575113Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6575606Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6576239Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6576807Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6577353Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6577884Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6578419Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6578924Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6579412Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6579902Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6580372Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6580789Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6581220Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6581665Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6582134Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6582612Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6583022Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6583523Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6583941Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6584369Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6584795Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6585206Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6585618Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6586003Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6586399Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6586783Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6587171Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6587561Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6587956Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6588377Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6588794Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6589211Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6589658Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6590106Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6590550Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6590969Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6591398Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6591835Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6592257Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6592673Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6593113Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6593631Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6594047Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6594461Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6594889Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6595300Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6595707Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6596124Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6596523Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6596933Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6597329Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6597732Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6598742Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6599170Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6599550Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6599922Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6600333Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6600738Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6601115Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6601500Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6601996Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6602392Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6602771Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6603176Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6603592Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6603967Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6604343Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6604723Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6605097Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6605501Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6605904Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6606306Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6606679Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6607086Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6607522Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6607946Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6608392Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6608851Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6609309Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6609747Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6610197Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6610638Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6611058Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6611574Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6612040Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6612498Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6612944Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6613401Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6613852Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6614266Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6614667Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6615049Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6615401Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6615771Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6616157Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6616551Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6616926Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6617312Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6617713Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6618106Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6618505Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6618905Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6619368Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6619797Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6620214Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6620633Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6621039Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6621454Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6621884Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6622294Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6622710Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6623098Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6623483Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6623860Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6624250Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6624641Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6625022Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6625480Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6625994Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6626431Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6626848Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6627265Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6627704Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6628144Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6628611Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6629129Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6629553Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6629993Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6630406Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6630782Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6631156Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6631574Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6631990Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6632418Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6632861Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6633283Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6633730Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6634229Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6634793Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6635405Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6635993Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6636673Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6637281Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6637883Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6639121Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6639739Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6640289Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6640789Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6641218Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6641613Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6642012Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6642424Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6642807Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6643230Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6643675Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6644120Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6644587Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6645010Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6645406Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6645804Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6646329Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6646777Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6647179Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6647598Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6648032Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6648444Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6648870Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6649286Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6649700Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6650106Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6650513Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6650938Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6651381Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6651811Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6652194Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6652576Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6653003Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6653424Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6653899Z test_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6654249Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6654619Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6655006Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6655373Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6655744Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6656130Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6656501Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6656871Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6657238Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6657622Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6658010Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6658404Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6658777Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6659110Z test_isend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6659492Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6659898Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6660295Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6660753Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6661208Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6661626Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6662033Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6662474Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6662904Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6663313Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6663784Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6664213Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6664621Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6665014Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6665408Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6665808Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6666261Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6666756Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6667235Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6667671Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6668140Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6668595Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6669028Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6669434Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6669864Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6670304Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6670736Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6671296Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6671808Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6672275Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6672659Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6673107Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6673519Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6673895Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6674283Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6674671Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6675061Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6675422Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6675786Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6676162Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6676527Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6676905Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6677278Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6677650Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6678037Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6678929Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6679285Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6679668Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6680049Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6680439Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6680819Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6681201Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6681587Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6682027Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6682422Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6682850Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6683337Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6683754Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6684157Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6684563Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6684982Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6685402Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6685804Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6686202Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6686642Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6687059Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6687466Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6687861Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6688265Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6688655Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6689058Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6689605Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:04:59.6690050Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:05:00.7838743Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:05:00.7853491Z 
2022-05-18T04:05:00.7853846Z Running tests...
2022-05-18T04:05:00.7854487Z ----------------------------------------------------------------------
2022-05-18T04:05:02.3861567Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:05:02.4241367Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 645
2022-05-18T04:05:02.4340640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 646
2022-05-18T04:05:03.5637453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:05:03.6115165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:05:03.6115996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:03.6144866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:03.6150959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:03.7130198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:04.8444507Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:05:04.8445523Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:05:04.9312945Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:05:04.9314160Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:05:05.9388917Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:05:05.9390295Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:05:05.9391004Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:05:05.9391852Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:05:05.9518518Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:05:05.9519766Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:05:05.9522514Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:05:05.9523548Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:05:05.9650730Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:05:05.9651733Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:05:05.9654252Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:05:05.9655264Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:05:06.3437268Z ok (5.558s)
2022-05-18T04:05:06.3437501Z 
2022-05-18T04:05:06.3438566Z ----------------------------------------------------------------------
2022-05-18T04:05:06.3439303Z Ran 1 test in 5.558s
2022-05-18T04:05:06.3439585Z 
2022-05-18T04:05:06.3439691Z OK
2022-05-18T04:05:06.3439813Z 
2022-05-18T04:05:06.3439949Z Generating XML reports...
2022-05-18T04:05:06.3481681Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040500.xml
2022-05-18T04:05:07.7563378Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:05:07.7577740Z 
2022-05-18T04:05:07.7577981Z Running tests...
2022-05-18T04:05:07.7578410Z ----------------------------------------------------------------------
2022-05-18T04:05:07.7621241Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-05-18T04:05:07.7621579Z 
2022-05-18T04:05:07.7621879Z ----------------------------------------------------------------------
2022-05-18T04:05:07.7622212Z Ran 1 test in 0.004s
2022-05-18T04:05:07.7622381Z 
2022-05-18T04:05:07.7622477Z OK (skipped=1)
2022-05-18T04:05:07.7622634Z 
2022-05-18T04:05:07.7622761Z Generating XML reports...
2022-05-18T04:05:07.7658116Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040507.xml
2022-05-18T04:05:08.9761590Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:05:08.9777614Z 
2022-05-18T04:05:08.9777782Z Running tests...
2022-05-18T04:05:08.9778906Z ----------------------------------------------------------------------
2022-05-18T04:05:10.5458760Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:05:10.5832135Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 795
2022-05-18T04:05:10.5930845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 796
2022-05-18T04:05:11.7385846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:05:11.7687003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:05:11.7688103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:11.7689484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:11.7696417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:11.8761165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:12.1984683Z ok (3.220s)
2022-05-18T04:05:12.1985000Z 
2022-05-18T04:05:12.1985851Z ----------------------------------------------------------------------
2022-05-18T04:05:12.1986250Z Ran 1 test in 3.221s
2022-05-18T04:05:12.1986434Z 
2022-05-18T04:05:12.1986513Z OK
2022-05-18T04:05:12.1986652Z 
2022-05-18T04:05:12.1986787Z Generating XML reports...
2022-05-18T04:05:12.2027023Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040508.xml
2022-05-18T04:05:13.5894628Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:05:13.5909041Z 
2022-05-18T04:05:13.5909411Z Running tests...
2022-05-18T04:05:13.5909928Z ----------------------------------------------------------------------
2022-05-18T04:05:15.1648526Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:05:15.1797217Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.588s)
2022-05-18T04:05:15.1797887Z 
2022-05-18T04:05:15.1798650Z ----------------------------------------------------------------------
2022-05-18T04:05:15.1799238Z Ran 1 test in 1.589s
2022-05-18T04:05:15.1799527Z 
2022-05-18T04:05:15.1799713Z OK (skipped=1)
2022-05-18T04:05:15.1799988Z 
2022-05-18T04:05:15.1800181Z Generating XML reports...
2022-05-18T04:05:15.1830351Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040513.xml
2022-05-18T04:05:16.5431776Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:05:16.5446453Z 
2022-05-18T04:05:16.5446759Z Running tests...
2022-05-18T04:05:16.5447188Z ----------------------------------------------------------------------
2022-05-18T04:05:16.5465039Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-05-18T04:05:16.5465393Z 
2022-05-18T04:05:16.5465684Z ----------------------------------------------------------------------
2022-05-18T04:05:16.5466022Z Ran 1 test in 0.002s
2022-05-18T04:05:16.5466167Z 
2022-05-18T04:05:16.5466276Z OK (skipped=1)
2022-05-18T04:05:16.5466431Z 
2022-05-18T04:05:16.5466557Z Generating XML reports...
2022-05-18T04:05:16.5502056Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040516.xml
2022-05-18T04:05:17.7997607Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:05:17.8011975Z 
2022-05-18T04:05:17.8012300Z Running tests...
2022-05-18T04:05:17.8012775Z ----------------------------------------------------------------------
2022-05-18T04:05:17.8030891Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-05-18T04:05:17.8031572Z 
2022-05-18T04:05:17.8031932Z ----------------------------------------------------------------------
2022-05-18T04:05:17.8032261Z Ran 1 test in 0.002s
2022-05-18T04:05:17.8032426Z 
2022-05-18T04:05:17.8032539Z OK (skipped=1)
2022-05-18T04:05:17.8032699Z 
2022-05-18T04:05:17.8032829Z Generating XML reports...
2022-05-18T04:05:17.8067602Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040517.xml
2022-05-18T04:05:19.0622222Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:05:19.0636879Z 
2022-05-18T04:05:19.0637379Z Running tests...
2022-05-18T04:05:19.0637874Z ----------------------------------------------------------------------
2022-05-18T04:05:20.6312960Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:05:20.6690026Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1002
2022-05-18T04:05:20.6788875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1003
2022-05-18T04:05:21.8389520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:05:21.8539438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:05:21.8540507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:21.8592342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:21.8598750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:21.9553508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:23.1281237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_cmy0t3l
2022-05-18T04:05:23.1282345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_cmy0t3l/_remote_module_non_scriptable.py
2022-05-18T04:05:23.2133361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmk5emdq0
2022-05-18T04:05:23.2134510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmk5emdq0/_remote_module_non_scriptable.py
2022-05-18T04:05:24.0218530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:24.0219079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:24.0433786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:24.0434449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:24.0729637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:24.0730367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:24.0943717Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:24.0944435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:24.2210045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:24.2210532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:24.2423124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:24.2423629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:24.6885730Z ok (5.625s)
2022-05-18T04:05:24.6886214Z 
2022-05-18T04:05:24.6886686Z ----------------------------------------------------------------------
2022-05-18T04:05:24.6887056Z Ran 1 test in 5.625s
2022-05-18T04:05:24.6887445Z 
2022-05-18T04:05:24.6887565Z OK
2022-05-18T04:05:24.6887688Z 
2022-05-18T04:05:24.6887828Z Generating XML reports...
2022-05-18T04:05:24.6928307Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040519.xml
2022-05-18T04:05:26.1168298Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:05:26.1183401Z 
2022-05-18T04:05:26.1183564Z Running tests...
2022-05-18T04:05:26.1184296Z ----------------------------------------------------------------------
2022-05-18T04:05:27.6992074Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:05:27.7375923Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1122
2022-05-18T04:05:27.7475169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1123
2022-05-18T04:05:28.9252081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:05:28.9278086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:05:28.9279232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:28.9353829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:28.9360122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:29.0293557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:30.1796675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp161r6dfb
2022-05-18T04:05:30.1797299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp161r6dfb/_remote_module_non_scriptable.py
2022-05-18T04:05:30.2765691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11l4lsqx
2022-05-18T04:05:30.2766543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11l4lsqx/_remote_module_non_scriptable.py
2022-05-18T04:05:30.3659336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:30.3659858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:30.3792108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:30.3792608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:30.6553222Z ok (4.537s)
2022-05-18T04:05:30.6553444Z 
2022-05-18T04:05:30.6554083Z ----------------------------------------------------------------------
2022-05-18T04:05:30.6554459Z Ran 1 test in 4.537s
2022-05-18T04:05:30.6554627Z 
2022-05-18T04:05:30.6554725Z OK
2022-05-18T04:05:30.6554860Z 
2022-05-18T04:05:30.6555019Z Generating XML reports...
2022-05-18T04:05:30.6595923Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040526.xml
2022-05-18T04:05:32.0810938Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:05:32.0825690Z 
2022-05-18T04:05:32.0826049Z Running tests...
2022-05-18T04:05:32.0826761Z ----------------------------------------------------------------------
2022-05-18T04:05:33.6609514Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:05:33.6998927Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1242
2022-05-18T04:05:33.7099040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1243
2022-05-18T04:05:34.8369406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:05:34.8490409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:05:34.8491276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:34.8571880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:34.8577921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:34.9504882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:36.1042584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj9xcivgn
2022-05-18T04:05:36.1043449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj9xcivgn/_remote_module_non_scriptable.py
2022-05-18T04:05:36.2037580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppydg90vm
2022-05-18T04:05:36.2039298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppydg90vm/_remote_module_non_scriptable.py
2022-05-18T04:05:36.3016069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:36.3016624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:36.3181973Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:36.3182492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:36.7178592Z ok (4.635s)
2022-05-18T04:05:36.7178978Z 
2022-05-18T04:05:36.7179595Z ----------------------------------------------------------------------
2022-05-18T04:05:36.7180033Z Ran 1 test in 4.635s
2022-05-18T04:05:36.7180202Z 
2022-05-18T04:05:36.7180306Z OK
2022-05-18T04:05:36.7180441Z 
2022-05-18T04:05:36.7180552Z Generating XML reports...
2022-05-18T04:05:36.7220934Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040532.xml
2022-05-18T04:05:38.1431494Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:05:38.1446994Z 
2022-05-18T04:05:38.1447447Z Running tests...
2022-05-18T04:05:38.1447919Z ----------------------------------------------------------------------
2022-05-18T04:05:39.7395327Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:05:39.7780390Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1362
2022-05-18T04:05:39.7880009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1363
2022-05-18T04:05:40.9185000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:05:40.9675489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:05:40.9676316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:40.9690786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:40.9696785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:41.0690932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:42.2153015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6fx8yaii
2022-05-18T04:05:42.2153649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6fx8yaii/_remote_module_non_scriptable.py
2022-05-18T04:05:42.3019936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuepl4o2o
2022-05-18T04:05:42.3021058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuepl4o2o/_remote_module_non_scriptable.py
2022-05-18T04:05:42.3874313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:42.3874863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:42.7961066Z ok (4.651s)
2022-05-18T04:05:42.7961277Z 
2022-05-18T04:05:42.7961678Z ----------------------------------------------------------------------
2022-05-18T04:05:42.7962018Z Ran 1 test in 4.651s
2022-05-18T04:05:42.7962189Z 
2022-05-18T04:05:42.7962269Z OK
2022-05-18T04:05:42.7962404Z 
2022-05-18T04:05:42.7962539Z Generating XML reports...
2022-05-18T04:05:42.8003198Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040538.xml
2022-05-18T04:05:44.2194792Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:05:44.2209754Z 
2022-05-18T04:05:44.2210154Z Running tests...
2022-05-18T04:05:44.2210668Z ----------------------------------------------------------------------
2022-05-18T04:05:45.7863424Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:05:45.8241439Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1482
2022-05-18T04:05:45.8340270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1483
2022-05-18T04:05:46.9803935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:05:46.9895005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:05:46.9896030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:46.9905063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:46.9910869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:47.0910011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:48.2315279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcgwz42f
2022-05-18T04:05:48.2315908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcgwz42f/_remote_module_non_scriptable.py
2022-05-18T04:05:48.3469664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaz6cea97
2022-05-18T04:05:48.3470923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaz6cea97/_remote_module_non_scriptable.py
2022-05-18T04:05:49.1649852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:49.1650393Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:49.1865368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:49.1865911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:49.5433798Z ok (5.322s)
2022-05-18T04:05:49.5434046Z 
2022-05-18T04:05:49.5434473Z ----------------------------------------------------------------------
2022-05-18T04:05:49.5434826Z Ran 1 test in 5.322s
2022-05-18T04:05:49.5434996Z 
2022-05-18T04:05:49.5435094Z OK
2022-05-18T04:05:49.5435211Z 
2022-05-18T04:05:49.5435347Z Generating XML reports...
2022-05-18T04:05:49.5476077Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040544.xml
2022-05-18T04:05:50.9427059Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:05:50.9441483Z 
2022-05-18T04:05:50.9441925Z Running tests...
2022-05-18T04:05:50.9442408Z ----------------------------------------------------------------------
2022-05-18T04:05:52.4811677Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:05:52.5186313Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1602
2022-05-18T04:05:52.5290405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1603
2022-05-18T04:05:53.6817470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:05:53.7383976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:05:53.7384773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:53.7425021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:53.7431124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:53.8398237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:54.9870235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb9q6zcm2
2022-05-18T04:05:54.9870837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb9q6zcm2/_remote_module_non_scriptable.py
2022-05-18T04:05:55.0876818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9jg1qpx
2022-05-18T04:05:55.0877682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9jg1qpx/_remote_module_non_scriptable.py
2022-05-18T04:05:55.3440019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:55.3440565Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:55.3610039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:55.3613267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:05:55.7372569Z ok (4.793s)
2022-05-18T04:05:55.7372757Z 
2022-05-18T04:05:55.7373429Z ----------------------------------------------------------------------
2022-05-18T04:05:55.7373813Z Ran 1 test in 4.793s
2022-05-18T04:05:55.7373983Z 
2022-05-18T04:05:55.7374090Z OK
2022-05-18T04:05:55.7374230Z 
2022-05-18T04:05:55.7374365Z Generating XML reports...
2022-05-18T04:05:55.7414253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040550.xml
2022-05-18T04:05:57.1261258Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:05:57.1274509Z 
2022-05-18T04:05:57.1274910Z Running tests...
2022-05-18T04:05:57.1275398Z ----------------------------------------------------------------------
2022-05-18T04:05:58.6559603Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:05:58.6940430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1722
2022-05-18T04:05:58.7040748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1723
2022-05-18T04:05:59.8444240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:05:59.8971633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:05:59.8972433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:59.9051891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:05:59.9058422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:59.9985999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:01.1842806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14lnfngm
2022-05-18T04:06:01.1843421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14lnfngm/_remote_module_non_scriptable.py
2022-05-18T04:06:01.2268802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8lr4vc1w
2022-05-18T04:06:01.2269712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8lr4vc1w/_remote_module_non_scriptable.py
2022-05-18T04:06:01.3142408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:06:01.3142963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:06:01.3281832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:06:01.3282355Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:06:01.7119471Z ok (4.584s)
2022-05-18T04:06:01.7119657Z 
2022-05-18T04:06:01.7120139Z ----------------------------------------------------------------------
2022-05-18T04:06:01.7120685Z Ran 1 test in 4.584s
2022-05-18T04:06:01.7120854Z 
2022-05-18T04:06:01.7120954Z OK
2022-05-18T04:06:01.7121090Z 
2022-05-18T04:06:01.7121203Z Generating XML reports...
2022-05-18T04:06:01.7162402Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040557.xml
2022-05-18T04:06:03.1114666Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:03.1128420Z 
2022-05-18T04:06:03.1128878Z Running tests...
2022-05-18T04:06:03.1129316Z ----------------------------------------------------------------------
2022-05-18T04:06:04.6521994Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:06:04.6671950Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.554s)
2022-05-18T04:06:04.6672541Z 
2022-05-18T04:06:04.6672827Z ----------------------------------------------------------------------
2022-05-18T04:06:04.6673164Z Ran 1 test in 1.554s
2022-05-18T04:06:04.6673331Z 
2022-05-18T04:06:04.6673441Z OK (skipped=1)
2022-05-18T04:06:04.6673581Z 
2022-05-18T04:06:04.6673706Z Generating XML reports...
2022-05-18T04:06:04.6704876Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040603.xml
2022-05-18T04:06:06.0343331Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:06.0357523Z 
2022-05-18T04:06:06.0357811Z Running tests...
2022-05-18T04:06:06.0358596Z ----------------------------------------------------------------------
2022-05-18T04:06:07.5936993Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:06:07.6314839Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1876
2022-05-18T04:06:07.6415961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1877
2022-05-18T04:06:08.7860476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:06:08.8071832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:06:08.8072659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:08.8165143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:08.8171432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:08.9083841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:09.1468460Z ok (3.111s)
2022-05-18T04:06:09.1468673Z 
2022-05-18T04:06:09.1469077Z ----------------------------------------------------------------------
2022-05-18T04:06:09.1469421Z Ran 1 test in 3.111s
2022-05-18T04:06:09.1469592Z 
2022-05-18T04:06:09.1469688Z OK
2022-05-18T04:06:09.1469807Z 
2022-05-18T04:06:09.1469942Z Generating XML reports...
2022-05-18T04:06:09.1512989Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040606.xml
2022-05-18T04:06:10.5605890Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:10.5620697Z 
2022-05-18T04:06:10.5620969Z Running tests...
2022-05-18T04:06:10.5621435Z ----------------------------------------------------------------------
2022-05-18T04:06:12.1312405Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:06:12.1468940Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.584s)
2022-05-18T04:06:12.1469516Z 
2022-05-18T04:06:12.1469797Z ----------------------------------------------------------------------
2022-05-18T04:06:12.1470431Z Ran 1 test in 1.585s
2022-05-18T04:06:12.1470601Z 
2022-05-18T04:06:12.1470696Z OK (skipped=1)
2022-05-18T04:06:12.1470853Z 
2022-05-18T04:06:12.1470982Z Generating XML reports...
2022-05-18T04:06:12.1503911Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040610.xml
2022-05-18T04:06:13.5236939Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:13.5249578Z 
2022-05-18T04:06:13.5250015Z Running tests...
2022-05-18T04:06:13.5250510Z ----------------------------------------------------------------------
2022-05-18T04:06:15.1175564Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:06:15.1561207Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2017
2022-05-18T04:06:15.1662414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2018
2022-05-18T04:06:16.2743759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:06:16.2947931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:16.2948532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:06:16.2949186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:16.2955210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:16.2955706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:17.9740357Z ok (4.449s)
2022-05-18T04:06:17.9740602Z 
2022-05-18T04:06:17.9741023Z ----------------------------------------------------------------------
2022-05-18T04:06:17.9741358Z Ran 1 test in 4.449s
2022-05-18T04:06:17.9741558Z 
2022-05-18T04:06:17.9741670Z OK
2022-05-18T04:06:17.9741813Z 
2022-05-18T04:06:17.9741953Z Generating XML reports...
2022-05-18T04:06:17.9784495Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040613.xml
2022-05-18T04:06:19.3862892Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:19.3876955Z 
2022-05-18T04:06:19.3877201Z Running tests...
2022-05-18T04:06:19.3877642Z ----------------------------------------------------------------------
2022-05-18T04:06:19.3898369Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-05-18T04:06:19.3898681Z 
2022-05-18T04:06:19.3898978Z ----------------------------------------------------------------------
2022-05-18T04:06:19.3899325Z Ran 1 test in 0.002s
2022-05-18T04:06:19.3899489Z 
2022-05-18T04:06:19.3899584Z OK (skipped=1)
2022-05-18T04:06:19.3899742Z 
2022-05-18T04:06:19.3899871Z Generating XML reports...
2022-05-18T04:06:19.3935260Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040619.xml
2022-05-18T04:06:20.6567095Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:20.6581766Z 
2022-05-18T04:06:20.6582465Z Running tests...
2022-05-18T04:06:20.6582968Z ----------------------------------------------------------------------
2022-05-18T04:06:20.6601143Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:06:22.2581840Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:06:22.2967242Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2166
2022-05-18T04:06:22.3067234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2167
2022-05-18T04:06:23.4758449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:06:23.4832590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:06:23.4833419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:23.4859905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:23.4866450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:23.5847942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:24.7258802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrhhjz93
2022-05-18T04:06:24.7259960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrhhjz93/_remote_module_non_scriptable.py
2022-05-18T04:06:24.8332843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6x75c5m
2022-05-18T04:06:24.8333771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6x75c5m/_remote_module_non_scriptable.py
2022-05-18T04:06:25.4147687Z ok (4.756s)
2022-05-18T04:06:25.4147957Z 
2022-05-18T04:06:25.4148364Z ----------------------------------------------------------------------
2022-05-18T04:06:25.4148733Z Ran 1 test in 4.757s
2022-05-18T04:06:25.4148907Z 
2022-05-18T04:06:25.4149013Z OK
2022-05-18T04:06:25.4149153Z 
2022-05-18T04:06:25.4149271Z Generating XML reports...
2022-05-18T04:06:25.4189582Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040620.xml
2022-05-18T04:06:26.8319592Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:26.8334095Z 
2022-05-18T04:06:26.8334445Z Running tests...
2022-05-18T04:06:26.8334896Z ----------------------------------------------------------------------
2022-05-18T04:06:26.8357185Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:06:28.3978146Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:06:28.4354758Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2286
2022-05-18T04:06:28.4454903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2287
2022-05-18T04:06:29.6317127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:06:29.6493185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:06:29.6494014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:29.6519521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:29.6527022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:29.7507479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:30.9153886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpor1s36ua
2022-05-18T04:06:30.9154778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpor1s36ua/_remote_module_non_scriptable.py
2022-05-18T04:06:31.0214200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3kbkiycb
2022-05-18T04:06:31.0215179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3kbkiycb/_remote_module_non_scriptable.py
2022-05-18T04:06:31.2693512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:06:31.2696303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:06:31.5535531Z ok (4.720s)
2022-05-18T04:06:31.5536247Z 
2022-05-18T04:06:31.5536650Z ----------------------------------------------------------------------
2022-05-18T04:06:31.5536981Z Ran 1 test in 4.720s
2022-05-18T04:06:31.5537151Z 
2022-05-18T04:06:31.5537248Z OK
2022-05-18T04:06:31.5537386Z 
2022-05-18T04:06:31.5537520Z Generating XML reports...
2022-05-18T04:06:31.5578620Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040626.xml
2022-05-18T04:06:32.9458623Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:32.9472722Z 
2022-05-18T04:06:32.9473095Z Running tests...
2022-05-18T04:06:32.9473592Z ----------------------------------------------------------------------
2022-05-18T04:06:32.9498153Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:06:34.4789428Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:06:34.5169435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2406
2022-05-18T04:06:34.5268323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2407
2022-05-18T04:06:35.6427433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:06:35.6604486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:06:35.6605286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:35.6630109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:35.6636061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:35.7618684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:36.9011415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmrlcc42
2022-05-18T04:06:36.9012410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmrlcc42/_remote_module_non_scriptable.py
2022-05-18T04:06:37.0094650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqw_0rzwg
2022-05-18T04:06:37.0095557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqw_0rzwg/_remote_module_non_scriptable.py
2022-05-18T04:06:37.2567527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:06:37.2573236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:06:37.5348886Z ok (4.587s)
2022-05-18T04:06:37.5349074Z 
2022-05-18T04:06:37.5349526Z ----------------------------------------------------------------------
2022-05-18T04:06:37.5350120Z Ran 1 test in 4.588s
2022-05-18T04:06:37.5350297Z 
2022-05-18T04:06:37.5350396Z OK
2022-05-18T04:06:37.5350555Z 
2022-05-18T04:06:37.5350672Z Generating XML reports...
2022-05-18T04:06:37.5393328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040632.xml
2022-05-18T04:06:38.9550655Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:38.9565672Z 
2022-05-18T04:06:38.9565976Z Running tests...
2022-05-18T04:06:38.9566422Z ----------------------------------------------------------------------
2022-05-18T04:06:38.9584591Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:06:40.5236269Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:06:40.5626215Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2526
2022-05-18T04:06:40.5727580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2527
2022-05-18T04:06:41.7252641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:06:41.7478055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:06:41.7479198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:41.7556939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:41.7563414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:41.8492242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:43.0078361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpml2zkk_g
2022-05-18T04:06:43.0079228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpml2zkk_g/_remote_module_non_scriptable.py
2022-05-18T04:06:43.1237820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkov7kx9c
2022-05-18T04:06:43.1239141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkov7kx9c/_remote_module_non_scriptable.py
2022-05-18T04:06:43.6809422Z ok (4.724s)
2022-05-18T04:06:43.6809650Z 
2022-05-18T04:06:43.6810327Z ----------------------------------------------------------------------
2022-05-18T04:06:43.6810698Z Ran 1 test in 4.724s
2022-05-18T04:06:43.6810867Z 
2022-05-18T04:06:43.6810944Z OK
2022-05-18T04:06:43.6811095Z 
2022-05-18T04:06:43.6811237Z Generating XML reports...
2022-05-18T04:06:43.6851796Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040638.xml
2022-05-18T04:06:45.0955451Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:45.0970779Z 
2022-05-18T04:06:45.0971013Z Running tests...
2022-05-18T04:06:45.0971456Z ----------------------------------------------------------------------
2022-05-18T04:06:45.0994592Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:06:45.0994881Z 
2022-05-18T04:06:45.0995368Z ----------------------------------------------------------------------
2022-05-18T04:06:45.0996069Z Ran 1 test in 0.002s
2022-05-18T04:06:45.0996260Z 
2022-05-18T04:06:45.0996373Z OK (skipped=1)
2022-05-18T04:06:45.0996540Z 
2022-05-18T04:06:45.0996648Z Generating XML reports...
2022-05-18T04:06:45.1039338Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040645.xml
2022-05-18T04:06:46.3429701Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:46.3446380Z 
2022-05-18T04:06:46.3447439Z Running tests...
2022-05-18T04:06:46.3448349Z ----------------------------------------------------------------------
2022-05-18T04:06:46.3469372Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:06:46.3470093Z 
2022-05-18T04:06:46.3470683Z ----------------------------------------------------------------------
2022-05-18T04:06:46.3471335Z Ran 1 test in 0.002s
2022-05-18T04:06:46.3471659Z 
2022-05-18T04:06:46.3471856Z OK (skipped=1)
2022-05-18T04:06:46.3472169Z 
2022-05-18T04:06:46.3472417Z Generating XML reports...
2022-05-18T04:06:46.3509454Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040646.xml
2022-05-18T04:06:47.5692586Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:47.5707086Z 
2022-05-18T04:06:47.5707432Z Running tests...
2022-05-18T04:06:47.5707862Z ----------------------------------------------------------------------
2022-05-18T04:06:47.5727431Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:06:47.5728099Z 
2022-05-18T04:06:47.5728394Z ----------------------------------------------------------------------
2022-05-18T04:06:47.5728731Z Ran 1 test in 0.002s
2022-05-18T04:06:47.5728879Z 
2022-05-18T04:06:47.5728994Z OK (skipped=1)
2022-05-18T04:06:47.5729150Z 
2022-05-18T04:06:47.5729285Z Generating XML reports...
2022-05-18T04:06:47.5764778Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040647.xml
2022-05-18T04:06:48.8293782Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:48.8308125Z 
2022-05-18T04:06:48.8308276Z Running tests...
2022-05-18T04:06:48.8308725Z ----------------------------------------------------------------------
2022-05-18T04:06:48.8328922Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:06:48.8329280Z 
2022-05-18T04:06:48.8329569Z ----------------------------------------------------------------------
2022-05-18T04:06:48.8329902Z Ran 1 test in 0.002s
2022-05-18T04:06:48.8330068Z 
2022-05-18T04:06:48.8330182Z OK (skipped=1)
2022-05-18T04:06:48.8330325Z 
2022-05-18T04:06:48.8330455Z Generating XML reports...
2022-05-18T04:06:48.8365738Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040648.xml
2022-05-18T04:06:50.0571126Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:50.0586344Z 
2022-05-18T04:06:50.0586698Z Running tests...
2022-05-18T04:06:50.0587158Z ----------------------------------------------------------------------
2022-05-18T04:06:50.0607060Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:06:50.0607427Z 
2022-05-18T04:06:50.0607724Z ----------------------------------------------------------------------
2022-05-18T04:06:50.0608084Z Ran 1 test in 0.002s
2022-05-18T04:06:50.0608251Z 
2022-05-18T04:06:50.0608367Z OK (skipped=1)
2022-05-18T04:06:50.0608524Z 
2022-05-18T04:06:50.0608632Z Generating XML reports...
2022-05-18T04:06:50.0644927Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040650.xml
2022-05-18T04:06:51.2931153Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:51.2946487Z 
2022-05-18T04:06:51.2946944Z Running tests...
2022-05-18T04:06:51.2947462Z ----------------------------------------------------------------------
2022-05-18T04:06:51.2978188Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s)
2022-05-18T04:06:51.2978541Z 
2022-05-18T04:06:51.2979272Z ----------------------------------------------------------------------
2022-05-18T04:06:51.2979652Z Ran 1 test in 0.003s
2022-05-18T04:06:51.2979821Z 
2022-05-18T04:06:51.2979940Z OK (skipped=1)
2022-05-18T04:06:51.2980081Z 
2022-05-18T04:06:51.2980211Z Generating XML reports...
2022-05-18T04:06:51.3016987Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040651.xml
2022-05-18T04:06:52.5529567Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:52.5544817Z 
2022-05-18T04:06:52.5544966Z Running tests...
2022-05-18T04:06:52.5546241Z ----------------------------------------------------------------------
2022-05-18T04:06:52.5566168Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:06:52.5566896Z 
2022-05-18T04:06:52.5567180Z ----------------------------------------------------------------------
2022-05-18T04:06:52.5567531Z Ran 1 test in 0.002s
2022-05-18T04:06:52.5567698Z 
2022-05-18T04:06:52.5567809Z OK (skipped=1)
2022-05-18T04:06:52.5568276Z 
2022-05-18T04:06:52.5568411Z Generating XML reports...
2022-05-18T04:06:52.5605013Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040652.xml
2022-05-18T04:06:53.7949071Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:06:53.7963545Z 
2022-05-18T04:06:53.7963812Z Running tests...
2022-05-18T04:06:53.7964269Z ----------------------------------------------------------------------
2022-05-18T04:06:55.3295637Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:06:55.3671852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2877
2022-05-18T04:06:55.3776772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2878
2022-05-18T04:06:56.4972480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:06:56.5140200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:06:56.5141016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:56.5175089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:06:56.5181089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:56.6154792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:58.6863659Z ok (4.889s)
2022-05-18T04:06:58.6864063Z 
2022-05-18T04:06:58.6864726Z ----------------------------------------------------------------------
2022-05-18T04:06:58.6865352Z Ran 1 test in 4.890s
2022-05-18T04:06:58.6865651Z 
2022-05-18T04:06:58.6865828Z OK
2022-05-18T04:06:58.6866082Z 
2022-05-18T04:06:58.6866329Z Generating XML reports...
2022-05-18T04:06:58.6910240Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040653.xml
2022-05-18T04:07:00.1179424Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:00.1194191Z 
2022-05-18T04:07:00.1194803Z Running tests...
2022-05-18T04:07:00.1195303Z ----------------------------------------------------------------------
2022-05-18T04:07:01.7090029Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:07:01.7479367Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2997
2022-05-18T04:07:01.7580849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2998
2022-05-18T04:07:02.8667973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:07:02.8963917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:07:02.8964764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:07:02.8971223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:07:02.8977597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:02.9977841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:04.9662535Z ok (4.846s)
2022-05-18T04:07:04.9662770Z 
2022-05-18T04:07:04.9663181Z ----------------------------------------------------------------------
2022-05-18T04:07:04.9663533Z Ran 1 test in 4.847s
2022-05-18T04:07:04.9663703Z 
2022-05-18T04:07:04.9663803Z OK
2022-05-18T04:07:04.9663921Z 
2022-05-18T04:07:04.9664059Z Generating XML reports...
2022-05-18T04:07:04.9704345Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040700.xml
2022-05-18T04:07:06.3852319Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:06.3867183Z 
2022-05-18T04:07:06.3867604Z Running tests...
2022-05-18T04:07:06.3868152Z ----------------------------------------------------------------------
2022-05-18T04:07:06.3887748Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:07:06.3888709Z 
2022-05-18T04:07:06.3889037Z ----------------------------------------------------------------------
2022-05-18T04:07:06.3889371Z Ran 1 test in 0.002s
2022-05-18T04:07:06.3889540Z 
2022-05-18T04:07:06.3889657Z OK (skipped=1)
2022-05-18T04:07:06.3889817Z 
2022-05-18T04:07:06.3889945Z Generating XML reports...
2022-05-18T04:07:06.3925395Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040706.xml
2022-05-18T04:07:07.6173850Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:07.6187269Z 
2022-05-18T04:07:07.6187702Z Running tests...
2022-05-18T04:07:07.6188220Z ----------------------------------------------------------------------
2022-05-18T04:07:07.6206712Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:07:07.6207303Z 
2022-05-18T04:07:07.6207612Z ----------------------------------------------------------------------
2022-05-18T04:07:07.6207933Z Ran 1 test in 0.002s
2022-05-18T04:07:07.6208102Z 
2022-05-18T04:07:07.6208216Z OK (skipped=1)
2022-05-18T04:07:07.6208374Z 
2022-05-18T04:07:07.6208504Z Generating XML reports...
2022-05-18T04:07:07.6242298Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040707.xml
2022-05-18T04:07:08.8648112Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:08.8662372Z 
2022-05-18T04:07:08.8662827Z Running tests...
2022-05-18T04:07:08.8663344Z ----------------------------------------------------------------------
2022-05-18T04:07:10.4497346Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:07:10.4884955Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3183
2022-05-18T04:07:10.4985763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3184
2022-05-18T04:07:11.6399056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:07:11.6579631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:07:11.6580455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:07:11.6601601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:07:11.6608350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:11.7594314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:13.8072158Z ok (4.940s)
2022-05-18T04:07:13.8072415Z 
2022-05-18T04:07:13.8072810Z ----------------------------------------------------------------------
2022-05-18T04:07:13.8073160Z Ran 1 test in 4.941s
2022-05-18T04:07:13.8073330Z 
2022-05-18T04:07:13.8073432Z OK
2022-05-18T04:07:13.8073551Z 
2022-05-18T04:07:13.8073693Z Generating XML reports...
2022-05-18T04:07:13.8116626Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040708.xml
2022-05-18T04:07:15.2263765Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:15.2278235Z 
2022-05-18T04:07:15.2278676Z Running tests...
2022-05-18T04:07:15.2279466Z ----------------------------------------------------------------------
2022-05-18T04:07:16.7916899Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:07:16.8306575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3303
2022-05-18T04:07:16.8407532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3304
2022-05-18T04:07:17.9736363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:07:18.0003253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:07:18.0004053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:07:18.0040274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:07:18.0047100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:18.1017786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:20.2493798Z ok (5.021s)
2022-05-18T04:07:20.2494029Z 
2022-05-18T04:07:20.2494446Z ----------------------------------------------------------------------
2022-05-18T04:07:20.2494794Z Ran 1 test in 5.022s
2022-05-18T04:07:20.2494968Z 
2022-05-18T04:07:20.2495069Z OK
2022-05-18T04:07:20.2495188Z 
2022-05-18T04:07:20.2495324Z Generating XML reports...
2022-05-18T04:07:20.2535994Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040715.xml
2022-05-18T04:07:21.6802351Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:21.6816989Z 
2022-05-18T04:07:21.6817215Z Running tests...
2022-05-18T04:07:21.6817691Z ----------------------------------------------------------------------
2022-05-18T04:07:23.2781424Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:07:23.3168803Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3423
2022-05-18T04:07:23.3270037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3424
2022-05-18T04:07:24.4739144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:07:24.5235968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:07:24.5236797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:07:24.5244601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:07:24.5250474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:24.6250266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:27.2365149Z ok (5.554s)
2022-05-18T04:07:27.2365380Z 
2022-05-18T04:07:27.2365808Z ----------------------------------------------------------------------
2022-05-18T04:07:27.2366139Z Ran 1 test in 5.555s
2022-05-18T04:07:27.2366308Z 
2022-05-18T04:07:27.2366405Z OK
2022-05-18T04:07:27.2366547Z 
2022-05-18T04:07:27.2366686Z Generating XML reports...
2022-05-18T04:07:27.2408504Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040721.xml
2022-05-18T04:07:28.6490046Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:28.6504431Z 
2022-05-18T04:07:28.6504939Z Running tests...
2022-05-18T04:07:28.6505441Z ----------------------------------------------------------------------
2022-05-18T04:07:30.2095441Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:07:30.2474393Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3540
2022-05-18T04:07:30.2573900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3541
2022-05-18T04:07:31.3722858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:07:31.4372523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:07:31.4373343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:07:31.4433034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:07:31.4438994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:31.5384453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:31.5596161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:07:31.5596714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:07:31.5597417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:07:31.5598574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:07:34.8441900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:07:34.8442462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:07:34.8443253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:07:34.8443960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:07:34.8830980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:07:34.8831542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:07:34.8832299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:07:34.8833000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:07:35.3712440Z ok (6.720s)
2022-05-18T04:07:35.3712685Z 
2022-05-18T04:07:35.3713348Z ----------------------------------------------------------------------
2022-05-18T04:07:35.3713715Z Ran 1 test in 6.721s
2022-05-18T04:07:35.3713888Z 
2022-05-18T04:07:35.3714000Z OK
2022-05-18T04:07:35.3714146Z 
2022-05-18T04:07:35.3714287Z Generating XML reports...
2022-05-18T04:07:35.3756690Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040728.xml
2022-05-18T04:07:36.7900509Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:36.7915421Z 
2022-05-18T04:07:36.7915844Z Running tests...
2022-05-18T04:07:36.7916340Z ----------------------------------------------------------------------
2022-05-18T04:07:36.7940583Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:07:36.7940931Z 
2022-05-18T04:07:36.7941201Z ----------------------------------------------------------------------
2022-05-18T04:07:36.7941531Z Ran 1 test in 0.003s
2022-05-18T04:07:36.7941682Z 
2022-05-18T04:07:36.7941794Z OK (skipped=1)
2022-05-18T04:07:36.7942284Z 
2022-05-18T04:07:36.7942412Z Generating XML reports...
2022-05-18T04:07:36.7985571Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040736.xml
2022-05-18T04:07:38.0465715Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:38.0480992Z 
2022-05-18T04:07:38.0481384Z Running tests...
2022-05-18T04:07:38.0481903Z ----------------------------------------------------------------------
2022-05-18T04:07:38.0505900Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:07:38.0506419Z 
2022-05-18T04:07:38.0506818Z ----------------------------------------------------------------------
2022-05-18T04:07:38.0507157Z Ran 1 test in 0.003s
2022-05-18T04:07:38.0507332Z 
2022-05-18T04:07:38.0507445Z OK (skipped=1)
2022-05-18T04:07:38.0507601Z 
2022-05-18T04:07:38.0507741Z Generating XML reports...
2022-05-18T04:07:38.0550668Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040738.xml
2022-05-18T04:07:39.2985650Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:39.3001134Z 
2022-05-18T04:07:39.3001587Z Running tests...
2022-05-18T04:07:39.3002103Z ----------------------------------------------------------------------
2022-05-18T04:07:39.3025792Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:07:39.3026151Z 
2022-05-18T04:07:39.3026497Z ----------------------------------------------------------------------
2022-05-18T04:07:39.3027023Z Ran 1 test in 0.003s
2022-05-18T04:07:39.3027200Z 
2022-05-18T04:07:39.3027313Z OK (skipped=1)
2022-05-18T04:07:39.3027471Z 
2022-05-18T04:07:39.3027597Z Generating XML reports...
2022-05-18T04:07:39.3069609Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040739.xml
2022-05-18T04:07:40.5567779Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:40.5583018Z 
2022-05-18T04:07:40.5583276Z Running tests...
2022-05-18T04:07:40.5584011Z ----------------------------------------------------------------------
2022-05-18T04:07:40.5607942Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:07:40.5608548Z 
2022-05-18T04:07:40.5608845Z ----------------------------------------------------------------------
2022-05-18T04:07:40.5609180Z Ran 1 test in 0.003s
2022-05-18T04:07:40.5609344Z 
2022-05-18T04:07:40.5609457Z OK (skipped=1)
2022-05-18T04:07:40.5609596Z 
2022-05-18T04:07:40.5609721Z Generating XML reports...
2022-05-18T04:07:40.5652473Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040740.xml
2022-05-18T04:07:41.8275504Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:41.8290826Z 
2022-05-18T04:07:41.8291086Z Running tests...
2022-05-18T04:07:41.8291521Z ----------------------------------------------------------------------
2022-05-18T04:07:41.8316146Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:07:41.8316505Z 
2022-05-18T04:07:41.8316790Z ----------------------------------------------------------------------
2022-05-18T04:07:41.8317132Z Ran 1 test in 0.003s
2022-05-18T04:07:41.8317297Z 
2022-05-18T04:07:41.8317414Z OK (skipped=1)
2022-05-18T04:07:41.8317555Z 
2022-05-18T04:07:41.8317682Z Generating XML reports...
2022-05-18T04:07:41.8360678Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040741.xml
2022-05-18T04:07:43.0664935Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:43.0678473Z 
2022-05-18T04:07:43.0678715Z Running tests...
2022-05-18T04:07:43.0679163Z ----------------------------------------------------------------------
2022-05-18T04:07:43.0698894Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:07:43.0699239Z 
2022-05-18T04:07:43.0699513Z ----------------------------------------------------------------------
2022-05-18T04:07:43.0699826Z Ran 1 test in 0.002s
2022-05-18T04:07:43.0699990Z 
2022-05-18T04:07:43.0700102Z OK (skipped=1)
2022-05-18T04:07:43.0700260Z 
2022-05-18T04:07:43.0700385Z Generating XML reports...
2022-05-18T04:07:43.0734212Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040743.xml
2022-05-18T04:07:44.3145893Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:44.3161838Z 
2022-05-18T04:07:44.3162223Z Running tests...
2022-05-18T04:07:44.3162683Z ----------------------------------------------------------------------
2022-05-18T04:07:44.3186443Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:07:44.3186801Z 
2022-05-18T04:07:44.3187075Z ----------------------------------------------------------------------
2022-05-18T04:07:44.3187389Z Ran 1 test in 0.003s
2022-05-18T04:07:44.3187555Z 
2022-05-18T04:07:44.3187666Z OK (skipped=1)
2022-05-18T04:07:44.3187824Z 
2022-05-18T04:07:44.3187949Z Generating XML reports...
2022-05-18T04:07:44.3230505Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040744.xml
2022-05-18T04:07:45.5692264Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:45.5707292Z 
2022-05-18T04:07:45.5707829Z Running tests...
2022-05-18T04:07:45.5708312Z ----------------------------------------------------------------------
2022-05-18T04:07:45.5732617Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:07:45.5732967Z 
2022-05-18T04:07:45.5733515Z ----------------------------------------------------------------------
2022-05-18T04:07:45.5733890Z Ran 1 test in 0.003s
2022-05-18T04:07:45.5734066Z 
2022-05-18T04:07:45.5734162Z OK (skipped=1)
2022-05-18T04:07:45.5734327Z 
2022-05-18T04:07:45.5734458Z Generating XML reports...
2022-05-18T04:07:45.5776526Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040745.xml
2022-05-18T04:07:46.7957940Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:46.7974554Z 
2022-05-18T04:07:46.7974842Z Running tests...
2022-05-18T04:07:46.7975338Z ----------------------------------------------------------------------
2022-05-18T04:07:46.7996645Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:07:46.7998450Z 
2022-05-18T04:07:46.7999669Z ----------------------------------------------------------------------
2022-05-18T04:07:46.8000034Z Ran 1 test in 0.002s
2022-05-18T04:07:46.8000202Z 
2022-05-18T04:07:46.8000315Z OK (skipped=1)
2022-05-18T04:07:46.8000471Z 
2022-05-18T04:07:46.8000600Z Generating XML reports...
2022-05-18T04:07:46.8034716Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040746.xml
2022-05-18T04:07:48.0527571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:48.0541695Z 
2022-05-18T04:07:48.0542169Z Running tests...
2022-05-18T04:07:48.0542839Z ----------------------------------------------------------------------
2022-05-18T04:07:48.0563015Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:07:48.0563518Z 
2022-05-18T04:07:48.0564726Z ----------------------------------------------------------------------
2022-05-18T04:07:48.0565586Z Ran 1 test in 0.002s
2022-05-18T04:07:48.0565755Z 
2022-05-18T04:07:48.0565869Z OK (skipped=1)
2022-05-18T04:07:48.0566024Z 
2022-05-18T04:07:48.0566149Z Generating XML reports...
2022-05-18T04:07:48.0601728Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040748.xml
2022-05-18T04:07:49.2741011Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:49.2756338Z 
2022-05-18T04:07:49.2756485Z Running tests...
2022-05-18T04:07:49.2757125Z ----------------------------------------------------------------------
2022-05-18T04:07:49.2777970Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:07:49.2778334Z 
2022-05-18T04:07:49.2778881Z ----------------------------------------------------------------------
2022-05-18T04:07:49.2779254Z Ran 1 test in 0.002s
2022-05-18T04:07:49.2779422Z 
2022-05-18T04:07:49.2779550Z OK (skipped=1)
2022-05-18T04:07:49.2779717Z 
2022-05-18T04:07:49.2779831Z Generating XML reports...
2022-05-18T04:07:49.2815924Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040749.xml
2022-05-18T04:07:50.5202496Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:50.5217213Z 
2022-05-18T04:07:50.5217590Z Running tests...
2022-05-18T04:07:50.5218043Z ----------------------------------------------------------------------
2022-05-18T04:07:50.5237625Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:07:50.5238013Z 
2022-05-18T04:07:50.5238600Z ----------------------------------------------------------------------
2022-05-18T04:07:50.5238942Z Ran 1 test in 0.002s
2022-05-18T04:07:50.5239107Z 
2022-05-18T04:07:50.5239218Z OK (skipped=1)
2022-05-18T04:07:50.5239359Z 
2022-05-18T04:07:50.5239746Z Generating XML reports...
2022-05-18T04:07:50.5273089Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040750.xml
2022-05-18T04:07:51.7762666Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:51.7776682Z 
2022-05-18T04:07:51.7776811Z Running tests...
2022-05-18T04:07:51.7791425Z ----------------------------------------------------------------------
2022-05-18T04:07:51.7799278Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:07:51.7799805Z 
2022-05-18T04:07:51.7800234Z ----------------------------------------------------------------------
2022-05-18T04:07:51.7800560Z Ran 1 test in 0.002s
2022-05-18T04:07:51.7800737Z 
2022-05-18T04:07:51.7800852Z OK (skipped=1)
2022-05-18T04:07:51.7801012Z 
2022-05-18T04:07:51.7801141Z Generating XML reports...
2022-05-18T04:07:51.7837043Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040751.xml
2022-05-18T04:07:52.9977332Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:52.9991354Z 
2022-05-18T04:07:52.9991656Z Running tests...
2022-05-18T04:07:52.9992101Z ----------------------------------------------------------------------
2022-05-18T04:07:53.0015216Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:07:53.0017372Z 
2022-05-18T04:07:53.0017964Z ----------------------------------------------------------------------
2022-05-18T04:07:53.0018639Z Ran 1 test in 0.002s
2022-05-18T04:07:53.0018809Z 
2022-05-18T04:07:53.0018925Z OK (skipped=1)
2022-05-18T04:07:53.0019095Z 
2022-05-18T04:07:53.0019226Z Generating XML reports...
2022-05-18T04:07:53.0052872Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040752.xml
2022-05-18T04:07:54.2483996Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:54.2498132Z 
2022-05-18T04:07:54.2498369Z Running tests...
2022-05-18T04:07:54.2498827Z ----------------------------------------------------------------------
2022-05-18T04:07:54.2518680Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:07:54.2519518Z 
2022-05-18T04:07:54.2519810Z ----------------------------------------------------------------------
2022-05-18T04:07:54.2520153Z Ran 1 test in 0.002s
2022-05-18T04:07:54.2520344Z 
2022-05-18T04:07:54.2520460Z OK (skipped=1)
2022-05-18T04:07:54.2520620Z 
2022-05-18T04:07:54.2520728Z Generating XML reports...
2022-05-18T04:07:54.2555568Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040754.xml
2022-05-18T04:07:55.5079371Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:55.5094483Z 
2022-05-18T04:07:55.5094951Z Running tests...
2022-05-18T04:07:55.5095562Z ----------------------------------------------------------------------
2022-05-18T04:07:55.5114697Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:07:55.5115060Z 
2022-05-18T04:07:55.5115329Z ----------------------------------------------------------------------
2022-05-18T04:07:55.5115668Z Ran 1 test in 0.002s
2022-05-18T04:07:55.5115836Z 
2022-05-18T04:07:55.5115949Z OK (skipped=1)
2022-05-18T04:07:55.5116121Z 
2022-05-18T04:07:55.5116251Z Generating XML reports...
2022-05-18T04:07:55.5152056Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040755.xml
2022-05-18T04:07:56.7498892Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:56.7512772Z 
2022-05-18T04:07:56.7513453Z Running tests...
2022-05-18T04:07:56.7513973Z ----------------------------------------------------------------------
2022-05-18T04:07:56.7534101Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:07:56.7534623Z 
2022-05-18T04:07:56.7534925Z ----------------------------------------------------------------------
2022-05-18T04:07:56.7535266Z Ran 1 test in 0.002s
2022-05-18T04:07:56.7535525Z 
2022-05-18T04:07:56.7535719Z OK (skipped=1)
2022-05-18T04:07:56.7535947Z 
2022-05-18T04:07:56.7536076Z Generating XML reports...
2022-05-18T04:07:56.7569601Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040756.xml
2022-05-18T04:07:58.0119521Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:58.0133155Z 
2022-05-18T04:07:58.0133452Z Running tests...
2022-05-18T04:07:58.0134168Z ----------------------------------------------------------------------
2022-05-18T04:07:58.0154367Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:07:58.0154694Z 
2022-05-18T04:07:58.0154981Z ----------------------------------------------------------------------
2022-05-18T04:07:58.0155560Z Ran 1 test in 0.002s
2022-05-18T04:07:58.0155742Z 
2022-05-18T04:07:58.0155859Z OK (skipped=1)
2022-05-18T04:07:58.0156017Z 
2022-05-18T04:07:58.0156154Z Generating XML reports...
2022-05-18T04:07:58.0190121Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040758.xml
2022-05-18T04:07:59.2723942Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:07:59.2737985Z 
2022-05-18T04:07:59.2738144Z Running tests...
2022-05-18T04:07:59.2738905Z ----------------------------------------------------------------------
2022-05-18T04:07:59.2758461Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:07:59.2758782Z 
2022-05-18T04:07:59.2759336Z ----------------------------------------------------------------------
2022-05-18T04:07:59.2759685Z Ran 1 test in 0.002s
2022-05-18T04:07:59.2759853Z 
2022-05-18T04:07:59.2759949Z OK (skipped=1)
2022-05-18T04:07:59.2760109Z 
2022-05-18T04:07:59.2760238Z Generating XML reports...
2022-05-18T04:07:59.2795722Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040759.xml
2022-05-18T04:08:00.5325239Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:00.5339733Z 
2022-05-18T04:08:00.5340040Z Running tests...
2022-05-18T04:08:00.5340488Z ----------------------------------------------------------------------
2022-05-18T04:08:00.5361025Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:08:00.5361343Z 
2022-05-18T04:08:00.5361626Z ----------------------------------------------------------------------
2022-05-18T04:08:00.5361965Z Ran 1 test in 0.002s
2022-05-18T04:08:00.5362133Z 
2022-05-18T04:08:00.5362251Z OK (skipped=1)
2022-05-18T04:08:00.5362391Z 
2022-05-18T04:08:00.5362521Z Generating XML reports...
2022-05-18T04:08:00.5398461Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040800.xml
2022-05-18T04:08:01.7759430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:01.7774998Z 
2022-05-18T04:08:01.7775438Z Running tests...
2022-05-18T04:08:01.7775943Z ----------------------------------------------------------------------
2022-05-18T04:08:01.7797433Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:08:01.7798505Z 
2022-05-18T04:08:01.7798845Z ----------------------------------------------------------------------
2022-05-18T04:08:01.7799192Z Ran 1 test in 0.002s
2022-05-18T04:08:01.7799373Z 
2022-05-18T04:08:01.7799469Z OK (skipped=1)
2022-05-18T04:08:01.7799632Z 
2022-05-18T04:08:01.7799761Z Generating XML reports...
2022-05-18T04:08:01.7834183Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040801.xml
2022-05-18T04:08:03.0333271Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:03.0348379Z 
2022-05-18T04:08:03.0348862Z Running tests...
2022-05-18T04:08:03.0349380Z ----------------------------------------------------------------------
2022-05-18T04:08:03.0369929Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:08:03.0370250Z 
2022-05-18T04:08:03.0370554Z ----------------------------------------------------------------------
2022-05-18T04:08:03.0370878Z Ran 1 test in 0.002s
2022-05-18T04:08:03.0371051Z 
2022-05-18T04:08:03.0371165Z OK (skipped=1)
2022-05-18T04:08:03.0371328Z 
2022-05-18T04:08:03.0371459Z Generating XML reports...
2022-05-18T04:08:03.0408136Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040803.xml
2022-05-18T04:08:04.3044042Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:04.3058918Z 
2022-05-18T04:08:04.3059278Z Running tests...
2022-05-18T04:08:04.3059718Z ----------------------------------------------------------------------
2022-05-18T04:08:04.3078702Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:08:04.3079216Z 
2022-05-18T04:08:04.3079615Z ----------------------------------------------------------------------
2022-05-18T04:08:04.3079955Z Ran 1 test in 0.002s
2022-05-18T04:08:04.3080138Z 
2022-05-18T04:08:04.3080253Z OK (skipped=1)
2022-05-18T04:08:04.3080390Z 
2022-05-18T04:08:04.3080520Z Generating XML reports...
2022-05-18T04:08:04.3115886Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040804.xml
2022-05-18T04:08:05.5661540Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:05.5676313Z 
2022-05-18T04:08:05.5676847Z Running tests...
2022-05-18T04:08:05.5677355Z ----------------------------------------------------------------------
2022-05-18T04:08:05.5696678Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:08:05.5697012Z 
2022-05-18T04:08:05.5697518Z ----------------------------------------------------------------------
2022-05-18T04:08:05.5697910Z Ran 1 test in 0.002s
2022-05-18T04:08:05.5698080Z 
2022-05-18T04:08:05.5698203Z OK (skipped=1)
2022-05-18T04:08:05.5698362Z 
2022-05-18T04:08:05.5698501Z Generating XML reports...
2022-05-18T04:08:05.5733309Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040805.xml
2022-05-18T04:08:06.8378111Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:06.8392957Z 
2022-05-18T04:08:06.8393107Z Running tests...
2022-05-18T04:08:06.8394036Z ----------------------------------------------------------------------
2022-05-18T04:08:06.8416599Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-05-18T04:08:06.8416964Z 
2022-05-18T04:08:06.8417399Z ----------------------------------------------------------------------
2022-05-18T04:08:06.8417874Z Ran 1 test in 0.002s
2022-05-18T04:08:06.8418041Z 
2022-05-18T04:08:06.8418156Z OK (skipped=1)
2022-05-18T04:08:06.8418316Z 
2022-05-18T04:08:06.8418443Z Generating XML reports...
2022-05-18T04:08:06.8454205Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040806.xml
2022-05-18T04:08:08.0984084Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:08.0998516Z 
2022-05-18T04:08:08.0998827Z Running tests...
2022-05-18T04:08:08.0999258Z ----------------------------------------------------------------------
2022-05-18T04:08:08.1022481Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-05-18T04:08:08.1022996Z 
2022-05-18T04:08:08.1023378Z ----------------------------------------------------------------------
2022-05-18T04:08:08.1023739Z Ran 1 test in 0.002s
2022-05-18T04:08:08.1023905Z 
2022-05-18T04:08:08.1024000Z OK (skipped=1)
2022-05-18T04:08:08.1024159Z 
2022-05-18T04:08:08.1024288Z Generating XML reports...
2022-05-18T04:08:08.1059977Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040808.xml
2022-05-18T04:08:09.3633175Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:09.3648076Z 
2022-05-18T04:08:09.3648352Z Running tests...
2022-05-18T04:08:09.3649070Z ----------------------------------------------------------------------
2022-05-18T04:08:09.3670006Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:08:09.3670332Z 
2022-05-18T04:08:09.3670634Z ----------------------------------------------------------------------
2022-05-18T04:08:09.3671212Z Ran 1 test in 0.002s
2022-05-18T04:08:09.3671664Z 
2022-05-18T04:08:09.3671785Z OK (skipped=1)
2022-05-18T04:08:09.3671931Z 
2022-05-18T04:08:09.3672064Z Generating XML reports...
2022-05-18T04:08:09.3707680Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040809.xml
2022-05-18T04:08:10.6290134Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:10.6304928Z 
2022-05-18T04:08:10.6305198Z Running tests...
2022-05-18T04:08:10.6305627Z ----------------------------------------------------------------------
2022-05-18T04:08:12.2489418Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:08:12.2877983Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4588
2022-05-18T04:08:12.2979927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4589
2022-05-18T04:08:13.4113716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:08:13.4564269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:08:13.4565293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:13.4619622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:13.4625706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:13.5579209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:15.1056325Z ok (4.475s)
2022-05-18T04:08:15.1056588Z 
2022-05-18T04:08:15.1056982Z ----------------------------------------------------------------------
2022-05-18T04:08:15.1057331Z Ran 1 test in 4.475s
2022-05-18T04:08:15.1057500Z 
2022-05-18T04:08:15.1057623Z OK
2022-05-18T04:08:15.1057748Z 
2022-05-18T04:08:15.1057883Z Generating XML reports...
2022-05-18T04:08:15.1097733Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040810.xml
2022-05-18T04:08:16.4936036Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:16.4957032Z 
2022-05-18T04:08:16.4957335Z Running tests...
2022-05-18T04:08:16.4957854Z ----------------------------------------------------------------------
2022-05-18T04:08:16.4978187Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:08:16.4978762Z 
2022-05-18T04:08:16.4979080Z ----------------------------------------------------------------------
2022-05-18T04:08:16.4979426Z Ran 1 test in 0.002s
2022-05-18T04:08:16.4979582Z 
2022-05-18T04:08:16.4979698Z OK (skipped=1)
2022-05-18T04:08:16.4979856Z 
2022-05-18T04:08:16.4979988Z Generating XML reports...
2022-05-18T04:08:16.5016640Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040816.xml
2022-05-18T04:08:17.7295013Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:17.7310249Z 
2022-05-18T04:08:17.7310404Z Running tests...
2022-05-18T04:08:17.7310876Z ----------------------------------------------------------------------
2022-05-18T04:08:17.7332813Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:08:17.7333155Z 
2022-05-18T04:08:17.7333468Z ----------------------------------------------------------------------
2022-05-18T04:08:17.7333814Z Ran 1 test in 0.002s
2022-05-18T04:08:17.7333985Z 
2022-05-18T04:08:17.7334108Z OK (skipped=1)
2022-05-18T04:08:17.7334268Z 
2022-05-18T04:08:17.7334399Z Generating XML reports...
2022-05-18T04:08:17.7370948Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040817.xml
2022-05-18T04:08:18.9686350Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:18.9701813Z 
2022-05-18T04:08:18.9702406Z Running tests...
2022-05-18T04:08:18.9703022Z ----------------------------------------------------------------------
2022-05-18T04:08:18.9725106Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:08:18.9725547Z 
2022-05-18T04:08:18.9725897Z ----------------------------------------------------------------------
2022-05-18T04:08:18.9726253Z Ran 1 test in 0.002s
2022-05-18T04:08:18.9726430Z 
2022-05-18T04:08:18.9726527Z OK (skipped=1)
2022-05-18T04:08:18.9726686Z 
2022-05-18T04:08:18.9726817Z Generating XML reports...
2022-05-18T04:08:18.9763072Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040818.xml
2022-05-18T04:08:20.2444279Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:20.2459655Z 
2022-05-18T04:08:20.2459944Z Running tests...
2022-05-18T04:08:20.2460573Z ----------------------------------------------------------------------
2022-05-18T04:08:21.8391880Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:08:21.8780296Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4803
2022-05-18T04:08:21.8881915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4804
2022-05-18T04:08:23.0388620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:08:23.0683141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:08:23.0683960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:23.0692825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:23.0699058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:23.1696849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:25.0967228Z ok (4.850s)
2022-05-18T04:08:25.0967480Z 
2022-05-18T04:08:25.0967877Z ----------------------------------------------------------------------
2022-05-18T04:08:25.0968226Z Ran 1 test in 4.851s
2022-05-18T04:08:25.0968397Z 
2022-05-18T04:08:25.0968477Z OK
2022-05-18T04:08:25.0968612Z 
2022-05-18T04:08:25.0968747Z Generating XML reports...
2022-05-18T04:08:25.1010040Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040820.xml
2022-05-18T04:08:26.5086298Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:26.5100647Z 
2022-05-18T04:08:26.5100916Z Running tests...
2022-05-18T04:08:26.5101371Z ----------------------------------------------------------------------
2022-05-18T04:08:28.0455271Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:08:28.0832092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4923
2022-05-18T04:08:28.0933388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4924
2022-05-18T04:08:29.2472614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:08:29.2728213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:08:29.2729024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:29.2777180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:29.2783824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:29.3742194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:31.3020451Z ok (4.791s)
2022-05-18T04:08:31.3020685Z 
2022-05-18T04:08:31.3021101Z ----------------------------------------------------------------------
2022-05-18T04:08:31.3021426Z Ran 1 test in 4.792s
2022-05-18T04:08:31.3021603Z 
2022-05-18T04:08:31.3021702Z OK
2022-05-18T04:08:31.3021848Z 
2022-05-18T04:08:31.3021981Z Generating XML reports...
2022-05-18T04:08:31.3063415Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040826.xml
2022-05-18T04:08:32.7323270Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:32.7337434Z 
2022-05-18T04:08:32.7337833Z Running tests...
2022-05-18T04:08:32.7338610Z ----------------------------------------------------------------------
2022-05-18T04:08:34.3063132Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:08:34.3450502Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5043
2022-05-18T04:08:34.3551576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5044
2022-05-18T04:08:35.4947380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:08:35.5011675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:08:35.5012494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:35.5048717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:35.5055107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:35.6026521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:37.5635859Z ok (4.829s)
2022-05-18T04:08:37.5636118Z 
2022-05-18T04:08:37.5636797Z ----------------------------------------------------------------------
2022-05-18T04:08:37.5637169Z Ran 1 test in 4.830s
2022-05-18T04:08:37.5637338Z 
2022-05-18T04:08:37.5637444Z OK
2022-05-18T04:08:37.5637578Z 
2022-05-18T04:08:37.5637693Z Generating XML reports...
2022-05-18T04:08:37.5678002Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040832.xml
2022-05-18T04:08:38.9893514Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:38.9908201Z 
2022-05-18T04:08:38.9908570Z Running tests...
2022-05-18T04:08:38.9909074Z ----------------------------------------------------------------------
2022-05-18T04:08:38.9927726Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:08:38.9928057Z 
2022-05-18T04:08:38.9928379Z ----------------------------------------------------------------------
2022-05-18T04:08:38.9928736Z Ran 1 test in 0.002s
2022-05-18T04:08:38.9928905Z 
2022-05-18T04:08:38.9929017Z OK (skipped=1)
2022-05-18T04:08:38.9929175Z 
2022-05-18T04:08:38.9930742Z Generating XML reports...
2022-05-18T04:08:38.9965064Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040838.xml
2022-05-18T04:08:40.2548364Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:40.2562827Z 
2022-05-18T04:08:40.2563096Z Running tests...
2022-05-18T04:08:40.2563561Z ----------------------------------------------------------------------
2022-05-18T04:08:40.2582859Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:08:40.2583513Z 
2022-05-18T04:08:40.2583833Z ----------------------------------------------------------------------
2022-05-18T04:08:40.2584176Z Ran 1 test in 0.002s
2022-05-18T04:08:40.2584343Z 
2022-05-18T04:08:40.2584466Z OK (skipped=1)
2022-05-18T04:08:40.2584625Z 
2022-05-18T04:08:40.2584736Z Generating XML reports...
2022-05-18T04:08:40.2620183Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040840.xml
2022-05-18T04:08:41.5242202Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:41.5256827Z 
2022-05-18T04:08:41.5257086Z Running tests...
2022-05-18T04:08:41.5257539Z ----------------------------------------------------------------------
2022-05-18T04:08:43.1351596Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:08:43.1740724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5229
2022-05-18T04:08:43.1842385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5230
2022-05-18T04:08:44.3201704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:08:44.3430630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:08:44.3431475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:44.3505321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:44.3511503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:44.4445449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:45.9918714Z ok (4.466s)
2022-05-18T04:08:45.9919444Z 
2022-05-18T04:08:45.9919868Z ----------------------------------------------------------------------
2022-05-18T04:08:45.9920201Z Ran 1 test in 4.466s
2022-05-18T04:08:45.9920371Z 
2022-05-18T04:08:45.9920467Z OK
2022-05-18T04:08:45.9920607Z 
2022-05-18T04:08:45.9921005Z Generating XML reports...
2022-05-18T04:08:45.9960397Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040841.xml
2022-05-18T04:08:47.4099761Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:47.4114188Z 
2022-05-18T04:08:47.4114451Z Running tests...
2022-05-18T04:08:47.4114881Z ----------------------------------------------------------------------
2022-05-18T04:08:48.9947100Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:08:49.0335366Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5345
2022-05-18T04:08:49.0439011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5346
2022-05-18T04:08:50.1949778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:08:50.2239971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:08:50.2240810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:50.2253362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:50.2259540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:50.3255046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:51.9517053Z ok (4.540s)
2022-05-18T04:08:51.9517281Z 
2022-05-18T04:08:51.9518424Z ----------------------------------------------------------------------
2022-05-18T04:08:51.9518766Z Ran 1 test in 4.540s
2022-05-18T04:08:51.9518939Z 
2022-05-18T04:08:51.9519047Z OK
2022-05-18T04:08:51.9519184Z 
2022-05-18T04:08:51.9519322Z Generating XML reports...
2022-05-18T04:08:51.9558842Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040847.xml
2022-05-18T04:08:53.3582361Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:53.3596044Z 
2022-05-18T04:08:53.3596342Z Running tests...
2022-05-18T04:08:53.3597111Z ----------------------------------------------------------------------
2022-05-18T04:08:53.3615561Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:08:53.3616185Z 
2022-05-18T04:08:53.3616793Z ----------------------------------------------------------------------
2022-05-18T04:08:53.3617320Z Ran 1 test in 0.002s
2022-05-18T04:08:53.3617490Z 
2022-05-18T04:08:53.3617601Z OK (skipped=1)
2022-05-18T04:08:53.3617760Z 
2022-05-18T04:08:53.3617891Z Generating XML reports...
2022-05-18T04:08:53.3652628Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040853.xml
2022-05-18T04:08:54.5696777Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:08:54.5710629Z 
2022-05-18T04:08:54.5711031Z Running tests...
2022-05-18T04:08:54.5711997Z ----------------------------------------------------------------------
2022-05-18T04:08:56.1578100Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:08:56.1955089Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5494
2022-05-18T04:08:56.2055350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5495
2022-05-18T04:08:57.3507610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:08:57.3561407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:08:57.3562439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:57.3608598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:08:57.3614873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:57.3618659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:08:57.4571886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:57.4575667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:08:57.4576470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:08:57.4637816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:08:59.1133719Z ok (4.542s)
2022-05-18T04:08:59.1134196Z 
2022-05-18T04:08:59.1134611Z ----------------------------------------------------------------------
2022-05-18T04:08:59.1134940Z Ran 1 test in 4.542s
2022-05-18T04:08:59.1135114Z 
2022-05-18T04:08:59.1135212Z OK
2022-05-18T04:08:59.1135350Z 
2022-05-18T04:08:59.1135482Z Generating XML reports...
2022-05-18T04:08:59.1176255Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040854.xml
2022-05-18T04:09:00.5343797Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:00.5359229Z 
2022-05-18T04:09:00.5359918Z Running tests...
2022-05-18T04:09:00.5360428Z ----------------------------------------------------------------------
2022-05-18T04:09:00.5379766Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:09:00.5380082Z 
2022-05-18T04:09:00.5380386Z ----------------------------------------------------------------------
2022-05-18T04:09:00.5380723Z Ran 1 test in 0.002s
2022-05-18T04:09:00.5380889Z 
2022-05-18T04:09:00.5381008Z OK (skipped=1)
2022-05-18T04:09:00.5381171Z 
2022-05-18T04:09:00.5381280Z Generating XML reports...
2022-05-18T04:09:00.5417683Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040900.xml
2022-05-18T04:09:01.7965588Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:01.7980835Z 
2022-05-18T04:09:01.7981124Z Running tests...
2022-05-18T04:09:01.7981552Z ----------------------------------------------------------------------
2022-05-18T04:09:03.3901698Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:09:03.4293111Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5647
2022-05-18T04:09:03.4394028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5648
2022-05-18T04:09:04.5628487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:09:04.5652235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:09:04.5653057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:04.5729930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:04.5736206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:04.6666053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:04.8444325Z skip: Skipped due to small world size. (3.046s)
2022-05-18T04:09:04.8444593Z 
2022-05-18T04:09:04.8445193Z ----------------------------------------------------------------------
2022-05-18T04:09:04.8445564Z Ran 1 test in 3.046s
2022-05-18T04:09:04.8445731Z 
2022-05-18T04:09:04.8445843Z OK (skipped=1)
2022-05-18T04:09:04.8446000Z 
2022-05-18T04:09:04.8446134Z Generating XML reports...
2022-05-18T04:09:04.8486992Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040901.xml
2022-05-18T04:09:06.2600597Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:06.2615626Z 
2022-05-18T04:09:06.2615962Z Running tests...
2022-05-18T04:09:06.2616410Z ----------------------------------------------------------------------
2022-05-18T04:09:06.2635246Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:09:06.2635582Z 
2022-05-18T04:09:06.2635875Z ----------------------------------------------------------------------
2022-05-18T04:09:06.2636191Z Ran 1 test in 0.002s
2022-05-18T04:09:06.2636370Z 
2022-05-18T04:09:06.2636485Z OK (skipped=1)
2022-05-18T04:09:06.2636643Z 
2022-05-18T04:09:06.2636771Z Generating XML reports...
2022-05-18T04:09:06.2672674Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040906.xml
2022-05-18T04:09:07.4837593Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:07.4853091Z 
2022-05-18T04:09:07.4853473Z Running tests...
2022-05-18T04:09:07.4853973Z ----------------------------------------------------------------------
2022-05-18T04:09:07.4873260Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:09:07.4873918Z 
2022-05-18T04:09:07.4874200Z ----------------------------------------------------------------------
2022-05-18T04:09:07.4874549Z Ran 1 test in 0.002s
2022-05-18T04:09:07.4874717Z 
2022-05-18T04:09:07.4874829Z OK (skipped=1)
2022-05-18T04:09:07.4874995Z 
2022-05-18T04:09:07.4875126Z Generating XML reports...
2022-05-18T04:09:07.4911484Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040907.xml
2022-05-18T04:09:08.7497938Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:08.7513084Z 
2022-05-18T04:09:08.7513459Z Running tests...
2022-05-18T04:09:08.7513974Z ----------------------------------------------------------------------
2022-05-18T04:09:10.3449856Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:09:10.3840177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5820
2022-05-18T04:09:10.3941763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5821
2022-05-18T04:09:11.5344619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:09:11.5733053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:09:11.5733862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:11.5749402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:11.5755968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:11.6748261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:13.7028212Z ok (4.951s)
2022-05-18T04:09:13.7028568Z 
2022-05-18T04:09:13.7029008Z ----------------------------------------------------------------------
2022-05-18T04:09:13.7029341Z Ran 1 test in 4.951s
2022-05-18T04:09:13.7029511Z 
2022-05-18T04:09:13.7029606Z OK
2022-05-18T04:09:13.7029744Z 
2022-05-18T04:09:13.7030155Z Generating XML reports...
2022-05-18T04:09:13.7071067Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040908.xml
2022-05-18T04:09:15.0961022Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:15.0976067Z 
2022-05-18T04:09:15.0976364Z Running tests...
2022-05-18T04:09:16.6325389Z ----------------------------------------------------------------------
2022-05-18T04:09:16.6326008Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:09:16.6707923Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5940
2022-05-18T04:09:16.6812333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5941
2022-05-18T04:09:17.8226586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:09:17.8629928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:09:17.8630748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:17.8631700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:17.8638465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:17.8639510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:19.8911230Z ok (4.793s)
2022-05-18T04:09:19.8911923Z 
2022-05-18T04:09:19.8912365Z ----------------------------------------------------------------------
2022-05-18T04:09:19.8912720Z Ran 1 test in 4.793s
2022-05-18T04:09:19.8912890Z 
2022-05-18T04:09:19.8912967Z OK
2022-05-18T04:09:19.8913102Z 
2022-05-18T04:09:19.8913238Z Generating XML reports...
2022-05-18T04:09:19.8953489Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040915.xml
2022-05-18T04:09:21.3128325Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:21.3143462Z 
2022-05-18T04:09:21.3144019Z Running tests...
2022-05-18T04:09:21.3144616Z ----------------------------------------------------------------------
2022-05-18T04:09:21.3164030Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:09:21.3164495Z 
2022-05-18T04:09:21.3164792Z ----------------------------------------------------------------------
2022-05-18T04:09:21.3165192Z Ran 1 test in 0.002s
2022-05-18T04:09:21.3165479Z 
2022-05-18T04:09:21.3165654Z OK (skipped=1)
2022-05-18T04:09:21.3165816Z 
2022-05-18T04:09:21.3165944Z Generating XML reports...
2022-05-18T04:09:21.3200504Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040921.xml
2022-05-18T04:09:22.5713124Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:22.5728054Z 
2022-05-18T04:09:22.5728455Z Running tests...
2022-05-18T04:09:22.5728946Z ----------------------------------------------------------------------
2022-05-18T04:09:24.1348736Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:09:24.1730522Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6093
2022-05-18T04:09:24.1832724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6094
2022-05-18T04:09:25.3294823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:09:25.3529326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:09:25.3531007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:25.3599624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:25.3607194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:25.3609942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:09:25.4539932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:25.4544236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:09:25.4544986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:09:25.4629129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:09:27.4931598Z ok (4.920s)
2022-05-18T04:09:27.4931810Z 
2022-05-18T04:09:27.4932234Z ----------------------------------------------------------------------
2022-05-18T04:09:27.4932582Z Ran 1 test in 4.920s
2022-05-18T04:09:27.4932756Z 
2022-05-18T04:09:27.4932834Z OK
2022-05-18T04:09:27.4932975Z 
2022-05-18T04:09:27.4933115Z Generating XML reports...
2022-05-18T04:09:27.4973631Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040922.xml
2022-05-18T04:09:28.9083859Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:28.9099173Z 
2022-05-18T04:09:28.9099493Z Running tests...
2022-05-18T04:09:28.9099929Z ----------------------------------------------------------------------
2022-05-18T04:09:28.9119419Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:09:28.9119758Z 
2022-05-18T04:09:28.9120659Z ----------------------------------------------------------------------
2022-05-18T04:09:28.9121045Z Ran 1 test in 0.002s
2022-05-18T04:09:28.9121217Z 
2022-05-18T04:09:28.9121333Z OK (skipped=1)
2022-05-18T04:09:28.9121476Z 
2022-05-18T04:09:28.9121620Z Generating XML reports...
2022-05-18T04:09:28.9157590Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040928.xml
2022-05-18T04:09:30.1444312Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:30.1457941Z 
2022-05-18T04:09:30.1458438Z Running tests...
2022-05-18T04:09:30.1458940Z ----------------------------------------------------------------------
2022-05-18T04:09:31.7080183Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:09:31.7464677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6250
2022-05-18T04:09:31.7565581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6251
2022-05-18T04:09:32.9053339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:09:32.9310046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:09:32.9311138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:32.9357410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:32.9363595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:33.0324732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:33.1615463Z skip: Skipped due to small world size. (3.015s)
2022-05-18T04:09:33.1615940Z 
2022-05-18T04:09:33.1616346Z ----------------------------------------------------------------------
2022-05-18T04:09:33.1616689Z Ran 1 test in 3.016s
2022-05-18T04:09:33.1616875Z 
2022-05-18T04:09:33.1617048Z OK (skipped=1)
2022-05-18T04:09:33.1617324Z 
2022-05-18T04:09:33.1617456Z Generating XML reports...
2022-05-18T04:09:33.1658465Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040930.xml
2022-05-18T04:09:34.5351341Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:34.5365961Z 
2022-05-18T04:09:34.5366288Z Running tests...
2022-05-18T04:09:34.5366725Z ----------------------------------------------------------------------
2022-05-18T04:09:34.5385283Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:09:34.5385624Z 
2022-05-18T04:09:34.5385919Z ----------------------------------------------------------------------
2022-05-18T04:09:34.5386259Z Ran 1 test in 0.002s
2022-05-18T04:09:34.5386435Z 
2022-05-18T04:09:34.5386533Z OK (skipped=1)
2022-05-18T04:09:34.5386692Z 
2022-05-18T04:09:34.5386821Z Generating XML reports...
2022-05-18T04:09:34.5421761Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040934.xml
2022-05-18T04:09:35.7541056Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:35.7555700Z 
2022-05-18T04:09:35.7556152Z Running tests...
2022-05-18T04:09:35.7556647Z ----------------------------------------------------------------------
2022-05-18T04:09:35.7575990Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:09:35.7576346Z 
2022-05-18T04:09:35.7576657Z ----------------------------------------------------------------------
2022-05-18T04:09:35.7576999Z Ran 1 test in 0.002s
2022-05-18T04:09:35.7577148Z 
2022-05-18T04:09:35.7577261Z OK (skipped=1)
2022-05-18T04:09:35.7577424Z 
2022-05-18T04:09:35.7577558Z Generating XML reports...
2022-05-18T04:09:35.7613902Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040935.xml
2022-05-18T04:09:37.0276927Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:37.0292078Z 
2022-05-18T04:09:37.0292296Z Running tests...
2022-05-18T04:09:37.0292757Z ----------------------------------------------------------------------
2022-05-18T04:09:38.6252498Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:09:38.6641890Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6423
2022-05-18T04:09:38.6743428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6424
2022-05-18T04:09:39.8164743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:09:39.8198905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:09:39.8199754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:39.8265905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:39.8271745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:39.9213864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:41.4820683Z ok (4.452s)
2022-05-18T04:09:41.4821064Z 
2022-05-18T04:09:41.4821721Z ----------------------------------------------------------------------
2022-05-18T04:09:41.4822626Z Ran 1 test in 4.453s
2022-05-18T04:09:41.4822979Z 
2022-05-18T04:09:41.4823140Z OK
2022-05-18T04:09:41.4823379Z 
2022-05-18T04:09:41.4823625Z Generating XML reports...
2022-05-18T04:09:41.4865179Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040937.xml
2022-05-18T04:09:42.9102104Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:42.9116548Z 
2022-05-18T04:09:42.9116941Z Running tests...
2022-05-18T04:09:42.9117451Z ----------------------------------------------------------------------
2022-05-18T04:09:44.5021893Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:09:44.5405808Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6539
2022-05-18T04:09:44.5508620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6540
2022-05-18T04:09:45.6782220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:09:45.6928328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:09:45.6929133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:45.6985441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:45.6991181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:45.7943347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:47.3584194Z ok (4.446s)
2022-05-18T04:09:47.3584560Z 
2022-05-18T04:09:47.3585213Z ----------------------------------------------------------------------
2022-05-18T04:09:47.3585845Z Ran 1 test in 4.447s
2022-05-18T04:09:47.3586161Z 
2022-05-18T04:09:47.3586354Z OK
2022-05-18T04:09:47.3586611Z 
2022-05-18T04:09:47.3586853Z Generating XML reports...
2022-05-18T04:09:47.3627875Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040942.xml
2022-05-18T04:09:48.7851724Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:48.7865999Z 
2022-05-18T04:09:48.7866262Z Running tests...
2022-05-18T04:09:48.7866694Z ----------------------------------------------------------------------
2022-05-18T04:09:48.7885593Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:09:48.7885962Z 
2022-05-18T04:09:48.7886640Z ----------------------------------------------------------------------
2022-05-18T04:09:48.7887004Z Ran 1 test in 0.002s
2022-05-18T04:09:48.7887178Z 
2022-05-18T04:09:48.7887295Z OK (skipped=1)
2022-05-18T04:09:48.7887456Z 
2022-05-18T04:09:48.7887605Z Generating XML reports...
2022-05-18T04:09:48.7923108Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040948.xml
2022-05-18T04:09:50.0515966Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:50.0530636Z 
2022-05-18T04:09:50.0531043Z Running tests...
2022-05-18T04:09:50.0531541Z ----------------------------------------------------------------------
2022-05-18T04:09:51.6419260Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:09:51.6799943Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6688
2022-05-18T04:09:51.6901581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6689
2022-05-18T04:09:52.8867155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:09:52.9003588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:09:52.9004427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:52.9069585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:09:52.9075913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:52.9079179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:09:53.0014501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:53.0018740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:09:53.0019479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:09:53.0094253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:09:54.6981363Z ok (4.645s)
2022-05-18T04:09:54.6981776Z 
2022-05-18T04:09:54.6982424Z ----------------------------------------------------------------------
2022-05-18T04:09:54.6983036Z Ran 1 test in 4.645s
2022-05-18T04:09:54.6983351Z 
2022-05-18T04:09:54.6983521Z OK
2022-05-18T04:09:54.6983761Z 
2022-05-18T04:09:54.6984010Z Generating XML reports...
2022-05-18T04:09:54.7025051Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040950.xml
2022-05-18T04:09:56.1104202Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:56.1118592Z 
2022-05-18T04:09:56.1119049Z Running tests...
2022-05-18T04:09:56.1119538Z ----------------------------------------------------------------------
2022-05-18T04:09:56.1138209Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:09:56.1138659Z 
2022-05-18T04:09:56.1139012Z ----------------------------------------------------------------------
2022-05-18T04:09:56.1139368Z Ran 1 test in 0.002s
2022-05-18T04:09:56.1139517Z 
2022-05-18T04:09:56.1139639Z OK (skipped=1)
2022-05-18T04:09:56.1139796Z 
2022-05-18T04:09:56.1139924Z Generating XML reports...
2022-05-18T04:09:56.1175242Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040956.xml
2022-05-18T04:09:57.3651460Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:09:57.3665795Z 
2022-05-18T04:09:57.3666055Z Running tests...
2022-05-18T04:09:57.3666502Z ----------------------------------------------------------------------
2022-05-18T04:09:58.9425047Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:09:58.9815285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6841
2022-05-18T04:09:58.9917352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6842
2022-05-18T04:10:00.1591193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:00.1785910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:00.1786723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:00.1793477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:00.1800083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:00.2800966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:00.4970389Z skip: Skipped due to small world size. (3.130s)
2022-05-18T04:10:00.4970654Z 
2022-05-18T04:10:00.4971052Z ----------------------------------------------------------------------
2022-05-18T04:10:00.4971393Z Ran 1 test in 3.130s
2022-05-18T04:10:00.4971558Z 
2022-05-18T04:10:00.4971654Z OK (skipped=1)
2022-05-18T04:10:00.4971822Z 
2022-05-18T04:10:00.4971954Z Generating XML reports...
2022-05-18T04:10:00.5013121Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040957.xml
2022-05-18T04:10:01.9111640Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:01.9126557Z 
2022-05-18T04:10:01.9126893Z Running tests...
2022-05-18T04:10:01.9127327Z ----------------------------------------------------------------------
2022-05-18T04:10:03.5000887Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:03.5391769Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6948
2022-05-18T04:10:03.5493233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6949
2022-05-18T04:10:04.6813595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:04.6854956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:04.6855772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:04.6914943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:04.6921362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:04.7870079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:07.1204152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:10:07.1205132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:10:07.1205921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:10:07.1307186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:10:07.4589848Z ok (5.546s)
2022-05-18T04:10:07.4590091Z 
2022-05-18T04:10:07.4590477Z ----------------------------------------------------------------------
2022-05-18T04:10:07.4590798Z Ran 1 test in 5.546s
2022-05-18T04:10:07.4590971Z 
2022-05-18T04:10:07.4591066Z OK
2022-05-18T04:10:07.4591203Z 
2022-05-18T04:10:07.4591338Z Generating XML reports...
2022-05-18T04:10:07.4631032Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041001.xml
2022-05-18T04:10:08.8804226Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:08.8818846Z 
2022-05-18T04:10:08.8819220Z Running tests...
2022-05-18T04:10:08.8819887Z ----------------------------------------------------------------------
2022-05-18T04:10:10.4695937Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:10.5085251Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7076
2022-05-18T04:10:10.5187408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7077
2022-05-18T04:10:11.6394676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:11.6478434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:11.6479819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:11.6496000Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:11.6502438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:11.7493379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:11.9237757Z skip: Need at least 3 CUDA devices (3.041s)
2022-05-18T04:10:11.9238387Z 
2022-05-18T04:10:11.9238854Z ----------------------------------------------------------------------
2022-05-18T04:10:11.9239204Z Ran 1 test in 3.042s
2022-05-18T04:10:11.9239374Z 
2022-05-18T04:10:11.9239578Z OK (skipped=1)
2022-05-18T04:10:11.9239869Z 
2022-05-18T04:10:11.9240015Z Generating XML reports...
2022-05-18T04:10:11.9280067Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041008.xml
2022-05-18T04:10:13.3369269Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:13.3383099Z 
2022-05-18T04:10:13.3383585Z Running tests...
2022-05-18T04:10:13.3384111Z ----------------------------------------------------------------------
2022-05-18T04:10:13.3403890Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-05-18T04:10:13.3404192Z 
2022-05-18T04:10:13.3404492Z ----------------------------------------------------------------------
2022-05-18T04:10:13.3405180Z Ran 1 test in 0.002s
2022-05-18T04:10:13.3405344Z 
2022-05-18T04:10:13.3405463Z OK (skipped=1)
2022-05-18T04:10:13.3405617Z 
2022-05-18T04:10:13.3405743Z Generating XML reports...
2022-05-18T04:10:13.3441165Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041013.xml
2022-05-18T04:10:14.5614821Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:14.5628592Z 
2022-05-18T04:10:14.5628995Z Running tests...
2022-05-18T04:10:14.5629483Z ----------------------------------------------------------------------
2022-05-18T04:10:14.5648559Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-05-18T04:10:14.5648882Z 
2022-05-18T04:10:14.5649153Z ----------------------------------------------------------------------
2022-05-18T04:10:14.5649485Z Ran 1 test in 0.002s
2022-05-18T04:10:14.5649653Z 
2022-05-18T04:10:14.5649762Z OK (skipped=1)
2022-05-18T04:10:14.5649932Z 
2022-05-18T04:10:14.5650066Z Generating XML reports...
2022-05-18T04:10:14.5686514Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041014.xml
2022-05-18T04:10:15.7900839Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:15.7915946Z 
2022-05-18T04:10:15.7916223Z Running tests...
2022-05-18T04:10:15.7916675Z ----------------------------------------------------------------------
2022-05-18T04:10:17.3462897Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:17.3851016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7249
2022-05-18T04:10:17.3950606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7250
2022-05-18T04:10:18.5279573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:18.5353578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:18.5354402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:18.5380840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:18.5386586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:18.6369045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:21.0038410Z ok (5.212s)
2022-05-18T04:10:21.0038870Z 
2022-05-18T04:10:21.0039274Z ----------------------------------------------------------------------
2022-05-18T04:10:21.0039614Z Ran 1 test in 5.212s
2022-05-18T04:10:21.0039781Z 
2022-05-18T04:10:21.0039877Z OK
2022-05-18T04:10:21.0040012Z 
2022-05-18T04:10:21.0040146Z Generating XML reports...
2022-05-18T04:10:21.0080394Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041015.xml
2022-05-18T04:10:22.3955780Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:22.3969614Z 
2022-05-18T04:10:22.3969782Z Running tests...
2022-05-18T04:10:22.3970558Z ----------------------------------------------------------------------
2022-05-18T04:10:22.3988879Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-05-18T04:10:22.3989217Z 
2022-05-18T04:10:22.3989534Z ----------------------------------------------------------------------
2022-05-18T04:10:22.3989868Z Ran 1 test in 0.002s
2022-05-18T04:10:22.3990032Z 
2022-05-18T04:10:22.3990127Z OK (skipped=1)
2022-05-18T04:10:22.3990285Z 
2022-05-18T04:10:22.3990412Z Generating XML reports...
2022-05-18T04:10:22.4025039Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041022.xml
2022-05-18T04:10:23.6472671Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:23.6490942Z 
2022-05-18T04:10:23.6491098Z Running tests...
2022-05-18T04:10:23.6491952Z ----------------------------------------------------------------------
2022-05-18T04:10:25.2188134Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:25.2572306Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7398
2022-05-18T04:10:25.2673978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7399
2022-05-18T04:10:26.4004142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:26.4104584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:26.4105932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:26.4106656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:26.4112991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:26.4114087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:26.5723570Z skip: Skipped due to small world size. (2.923s)
2022-05-18T04:10:26.5724012Z 
2022-05-18T04:10:26.5724589Z ----------------------------------------------------------------------
2022-05-18T04:10:26.5724923Z Ran 1 test in 2.923s
2022-05-18T04:10:26.5725088Z 
2022-05-18T04:10:26.5725200Z OK (skipped=1)
2022-05-18T04:10:26.5725354Z 
2022-05-18T04:10:26.5725481Z Generating XML reports...
2022-05-18T04:10:26.5766240Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041023.xml
2022-05-18T04:10:27.9694579Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:27.9708997Z 
2022-05-18T04:10:27.9709366Z Running tests...
2022-05-18T04:10:27.9709792Z ----------------------------------------------------------------------
2022-05-18T04:10:27.9729844Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-05-18T04:10:27.9730176Z 
2022-05-18T04:10:27.9730466Z ----------------------------------------------------------------------
2022-05-18T04:10:27.9730795Z Ran 1 test in 0.002s
2022-05-18T04:10:27.9730959Z 
2022-05-18T04:10:27.9731053Z OK (skipped=1)
2022-05-18T04:10:27.9731208Z 
2022-05-18T04:10:27.9731337Z Generating XML reports...
2022-05-18T04:10:27.9767772Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041027.xml
2022-05-18T04:10:29.2217711Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:29.2233055Z 
2022-05-18T04:10:29.2233356Z Running tests...
2022-05-18T04:10:29.2235030Z ----------------------------------------------------------------------
2022-05-18T04:10:30.8089674Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:30.8484960Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7538
2022-05-18T04:10:30.8587585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7539
2022-05-18T04:10:32.0255426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:32.0309079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:32.0309918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:32.0356782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:32.0363785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:32.1324091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:32.2637629Z skip: Skipped due to small world size. (3.040s)
2022-05-18T04:10:32.2637919Z 
2022-05-18T04:10:32.2638999Z ----------------------------------------------------------------------
2022-05-18T04:10:32.2639352Z Ran 1 test in 3.040s
2022-05-18T04:10:32.2639518Z 
2022-05-18T04:10:32.2639629Z OK (skipped=1)
2022-05-18T04:10:32.2639768Z 
2022-05-18T04:10:32.2639896Z Generating XML reports...
2022-05-18T04:10:32.2682454Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041029.xml
2022-05-18T04:10:33.6745963Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:33.6760644Z 
2022-05-18T04:10:33.6760968Z Running tests...
2022-05-18T04:10:33.6761430Z ----------------------------------------------------------------------
2022-05-18T04:10:33.6781871Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-05-18T04:10:33.6782202Z 
2022-05-18T04:10:33.6782502Z ----------------------------------------------------------------------
2022-05-18T04:10:33.6782840Z Ran 1 test in 0.002s
2022-05-18T04:10:33.6783009Z 
2022-05-18T04:10:33.6783122Z OK (skipped=1)
2022-05-18T04:10:33.6783262Z 
2022-05-18T04:10:33.6783391Z Generating XML reports...
2022-05-18T04:10:33.6819416Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041033.xml
2022-05-18T04:10:34.9415487Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:34.9429860Z 
2022-05-18T04:10:34.9430111Z Running tests...
2022-05-18T04:10:34.9430562Z ----------------------------------------------------------------------
2022-05-18T04:10:34.9455036Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-05-18T04:10:34.9455369Z 
2022-05-18T04:10:34.9455930Z ----------------------------------------------------------------------
2022-05-18T04:10:34.9456296Z Ran 1 test in 0.002s
2022-05-18T04:10:34.9456466Z 
2022-05-18T04:10:34.9456580Z OK (skipped=1)
2022-05-18T04:10:34.9456743Z 
2022-05-18T04:10:34.9456852Z Generating XML reports...
2022-05-18T04:10:34.9492267Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041034.xml
2022-05-18T04:10:36.1870621Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:36.1884842Z 
2022-05-18T04:10:36.1885158Z Running tests...
2022-05-18T04:10:36.1885610Z ----------------------------------------------------------------------
2022-05-18T04:10:36.1905112Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-05-18T04:10:36.1905449Z 
2022-05-18T04:10:36.1905746Z ----------------------------------------------------------------------
2022-05-18T04:10:36.1906073Z Ran 1 test in 0.002s
2022-05-18T04:10:36.1906240Z 
2022-05-18T04:10:36.1906355Z OK (skipped=1)
2022-05-18T04:10:36.1906514Z 
2022-05-18T04:10:36.1906641Z Generating XML reports...
2022-05-18T04:10:36.1941215Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041036.xml
2022-05-18T04:10:37.4117314Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:37.4132424Z 
2022-05-18T04:10:37.4132755Z Running tests...
2022-05-18T04:10:37.4133221Z ----------------------------------------------------------------------
2022-05-18T04:10:37.4157547Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-05-18T04:10:37.4158052Z 
2022-05-18T04:10:37.4158778Z ----------------------------------------------------------------------
2022-05-18T04:10:37.4159140Z Ran 1 test in 0.003s
2022-05-18T04:10:37.4159322Z 
2022-05-18T04:10:37.4159459Z OK (skipped=1)
2022-05-18T04:10:37.4159625Z 
2022-05-18T04:10:37.4159758Z Generating XML reports...
2022-05-18T04:10:37.4194617Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041037.xml
2022-05-18T04:10:38.6718863Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:38.6733560Z 
2022-05-18T04:10:38.6733892Z Running tests...
2022-05-18T04:10:38.6734344Z ----------------------------------------------------------------------
2022-05-18T04:10:38.6759936Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-05-18T04:10:38.6760313Z 
2022-05-18T04:10:38.6760580Z ----------------------------------------------------------------------
2022-05-18T04:10:38.6760941Z Ran 1 test in 0.003s
2022-05-18T04:10:38.6761108Z 
2022-05-18T04:10:38.6761220Z OK (skipped=1)
2022-05-18T04:10:38.6761382Z 
2022-05-18T04:10:38.6761518Z Generating XML reports...
2022-05-18T04:10:38.6796690Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041038.xml
2022-05-18T04:10:39.9507081Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:39.9521908Z 
2022-05-18T04:10:39.9522453Z Running tests...
2022-05-18T04:10:39.9522972Z ----------------------------------------------------------------------
2022-05-18T04:10:41.5426328Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:41.5818156Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7810
2022-05-18T04:10:41.5920678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7811
2022-05-18T04:10:42.7478209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:42.7721074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:42.7721908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:42.7782056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:42.7788809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:42.8732552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:42.8916585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:10:42.8917120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:10:42.8917838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:10:42.8918842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:10:42.8919869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:10:42.8920679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:10:42.8921457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:10:42.8922800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:10:43.1974938Z ok (3.245s)
2022-05-18T04:10:43.1975262Z 
2022-05-18T04:10:43.1975652Z ----------------------------------------------------------------------
2022-05-18T04:10:43.1975978Z Ran 1 test in 3.245s
2022-05-18T04:10:43.1976144Z 
2022-05-18T04:10:43.1976243Z OK
2022-05-18T04:10:43.1976381Z 
2022-05-18T04:10:43.1976526Z Generating XML reports...
2022-05-18T04:10:43.2017597Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041039.xml
2022-05-18T04:10:44.6184644Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:44.6199157Z 
2022-05-18T04:10:44.6199526Z Running tests...
2022-05-18T04:10:44.6199987Z ----------------------------------------------------------------------
2022-05-18T04:10:46.1999749Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:46.2391630Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7927
2022-05-18T04:10:46.2494706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7928
2022-05-18T04:10:47.3810797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:47.3932398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:47.3933245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:47.4013677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:47.4019783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:47.4943460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:49.0571761Z ok (4.437s)
2022-05-18T04:10:49.0572170Z 
2022-05-18T04:10:49.0572575Z ----------------------------------------------------------------------
2022-05-18T04:10:49.0572923Z Ran 1 test in 4.437s
2022-05-18T04:10:49.0573200Z 
2022-05-18T04:10:49.0573377Z OK
2022-05-18T04:10:49.0573633Z 
2022-05-18T04:10:49.0573755Z Generating XML reports...
2022-05-18T04:10:49.0614336Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041044.xml
2022-05-18T04:10:50.4763261Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:50.4778351Z 
2022-05-18T04:10:50.4778790Z Running tests...
2022-05-18T04:10:50.4779268Z ----------------------------------------------------------------------
2022-05-18T04:10:52.0714790Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:52.1100475Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8043
2022-05-18T04:10:52.1203298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8044
2022-05-18T04:10:53.2280775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:53.2565763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:53.2566589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:53.2585922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:53.2591887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:53.3580616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:53.5254189Z skip: Skipped due to small world size. (3.047s)
2022-05-18T04:10:53.5254742Z 
2022-05-18T04:10:53.5255142Z ----------------------------------------------------------------------
2022-05-18T04:10:53.5255491Z Ran 1 test in 3.048s
2022-05-18T04:10:53.5255661Z 
2022-05-18T04:10:53.5255774Z OK (skipped=1)
2022-05-18T04:10:53.5255930Z 
2022-05-18T04:10:53.5256041Z Generating XML reports...
2022-05-18T04:10:53.5296679Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041050.xml
2022-05-18T04:10:54.9261900Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:10:54.9276407Z 
2022-05-18T04:10:54.9276685Z Running tests...
2022-05-18T04:10:54.9277142Z ----------------------------------------------------------------------
2022-05-18T04:10:56.4986777Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:56.5380953Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8150
2022-05-18T04:10:56.5484693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8151
2022-05-18T04:10:57.6723729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:57.6728260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:57.6729115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:57.6825699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:10:57.6832538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:57.7740390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:59.2559871Z ok (4.328s)
2022-05-18T04:10:59.2560065Z 
2022-05-18T04:10:59.2560479Z ----------------------------------------------------------------------
2022-05-18T04:10:59.2560857Z Ran 1 test in 4.328s
2022-05-18T04:10:59.2561030Z 
2022-05-18T04:10:59.2561129Z OK
2022-05-18T04:10:59.2561269Z 
2022-05-18T04:10:59.2561408Z Generating XML reports...
2022-05-18T04:10:59.2604233Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041054.xml
2022-05-18T04:11:00.6927079Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:00.6942144Z 
2022-05-18T04:11:00.6942479Z Running tests...
2022-05-18T04:11:00.6942943Z ----------------------------------------------------------------------
2022-05-18T04:11:02.2786703Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:02.3172670Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8258
2022-05-18T04:11:02.3276333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8259
2022-05-18T04:11:03.4568970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:03.4690992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:03.4691840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:03.4772309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:03.4778828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:03.5703917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:03.8330504Z ok (3.138s)
2022-05-18T04:11:03.8330877Z 
2022-05-18T04:11:03.8331552Z ----------------------------------------------------------------------
2022-05-18T04:11:03.8332575Z Ran 1 test in 3.139s
2022-05-18T04:11:03.8332849Z 
2022-05-18T04:11:03.8333048Z OK
2022-05-18T04:11:03.8333309Z 
2022-05-18T04:11:03.8333564Z Generating XML reports...
2022-05-18T04:11:03.8374387Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041100.xml
2022-05-18T04:11:05.1943257Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:05.1957767Z 
2022-05-18T04:11:05.1958152Z Running tests...
2022-05-18T04:11:05.1958880Z ----------------------------------------------------------------------
2022-05-18T04:11:06.7347481Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:06.7730051Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8365
2022-05-18T04:11:06.7834027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8366
2022-05-18T04:11:07.9399249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:07.9706219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:07.9707057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:07.9804202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:07.9810607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:08.0718419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:09.6909659Z ok (4.495s)
2022-05-18T04:11:09.6909884Z 
2022-05-18T04:11:09.6910278Z ----------------------------------------------------------------------
2022-05-18T04:11:09.6910602Z Ran 1 test in 4.495s
2022-05-18T04:11:09.6910788Z 
2022-05-18T04:11:09.6910885Z OK
2022-05-18T04:11:09.6911022Z 
2022-05-18T04:11:09.6911158Z Generating XML reports...
2022-05-18T04:11:09.6953904Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041105.xml
2022-05-18T04:11:11.1153190Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:11.1168155Z 
2022-05-18T04:11:11.1168496Z Running tests...
2022-05-18T04:11:11.1168951Z ----------------------------------------------------------------------
2022-05-18T04:11:12.6978288Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:12.7367726Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8481
2022-05-18T04:11:12.7468936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8482
2022-05-18T04:11:13.8898987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:13.9206885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:13.9207694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:13.9306354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:13.9313048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:14.0219665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:15.7550469Z ok (4.638s)
2022-05-18T04:11:15.7550696Z 
2022-05-18T04:11:15.7551083Z ----------------------------------------------------------------------
2022-05-18T04:11:15.7551408Z Ran 1 test in 4.638s
2022-05-18T04:11:15.7551573Z 
2022-05-18T04:11:15.7551679Z OK
2022-05-18T04:11:15.7552092Z 
2022-05-18T04:11:15.7552231Z Generating XML reports...
2022-05-18T04:11:15.7593026Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041111.xml
2022-05-18T04:11:17.1695753Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:17.1709988Z 
2022-05-18T04:11:17.1710190Z Running tests...
2022-05-18T04:11:17.1710629Z ----------------------------------------------------------------------
2022-05-18T04:11:18.7570706Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:18.7961241Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8597
2022-05-18T04:11:18.8064190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8598
2022-05-18T04:11:19.9483549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:19.9884374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:19.9885187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:19.9888725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:19.9894920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:20.0895295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:20.4119559Z ok (3.240s)
2022-05-18T04:11:20.4119780Z 
2022-05-18T04:11:20.4120146Z ----------------------------------------------------------------------
2022-05-18T04:11:20.4120493Z Ran 1 test in 3.241s
2022-05-18T04:11:20.4120662Z 
2022-05-18T04:11:20.4120765Z OK
2022-05-18T04:11:20.4120902Z 
2022-05-18T04:11:20.4121039Z Generating XML reports...
2022-05-18T04:11:20.4164062Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041117.xml
2022-05-18T04:11:21.8382517Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:21.8397157Z 
2022-05-18T04:11:21.8397620Z Running tests...
2022-05-18T04:11:21.8398688Z ----------------------------------------------------------------------
2022-05-18T04:11:21.8417428Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:11:21.8417734Z 
2022-05-18T04:11:21.8418008Z ----------------------------------------------------------------------
2022-05-18T04:11:21.8418342Z Ran 1 test in 0.002s
2022-05-18T04:11:21.8418513Z 
2022-05-18T04:11:21.8418625Z OK (skipped=1)
2022-05-18T04:11:21.8418781Z 
2022-05-18T04:11:21.8418907Z Generating XML reports...
2022-05-18T04:11:21.8455757Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041121.xml
2022-05-18T04:11:23.0647556Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:23.0661891Z 
2022-05-18T04:11:23.0662320Z Running tests...
2022-05-18T04:11:23.0662813Z ----------------------------------------------------------------------
2022-05-18T04:11:24.6435220Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:24.6825216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8737
2022-05-18T04:11:24.6926824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8738
2022-05-18T04:11:25.8301845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:25.8433091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:25.8433892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:25.8504528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:25.8510455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:25.9449016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:28.1016547Z ok (5.035s)
2022-05-18T04:11:28.1016932Z 
2022-05-18T04:11:28.1017416Z ----------------------------------------------------------------------
2022-05-18T04:11:28.1017817Z Ran 1 test in 5.035s
2022-05-18T04:11:28.1017969Z 
2022-05-18T04:11:28.1018064Z OK
2022-05-18T04:11:28.1018201Z 
2022-05-18T04:11:28.1018333Z Generating XML reports...
2022-05-18T04:11:28.1058564Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041123.xml
2022-05-18T04:11:29.5273295Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:29.5288855Z 
2022-05-18T04:11:29.5289120Z Running tests...
2022-05-18T04:11:29.5289557Z ----------------------------------------------------------------------
2022-05-18T04:11:29.5308650Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:11:29.5309530Z 
2022-05-18T04:11:29.5309913Z ----------------------------------------------------------------------
2022-05-18T04:11:29.5310282Z Ran 1 test in 0.002s
2022-05-18T04:11:29.5310431Z 
2022-05-18T04:11:29.5310550Z OK (skipped=1)
2022-05-18T04:11:29.5310714Z 
2022-05-18T04:11:29.5310845Z Generating XML reports...
2022-05-18T04:11:29.5346950Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041129.xml
2022-05-18T04:11:30.7686065Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:30.7700581Z 
2022-05-18T04:11:30.7700838Z Running tests...
2022-05-18T04:11:30.7701300Z ----------------------------------------------------------------------
2022-05-18T04:11:30.7721965Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:11:30.7722278Z 
2022-05-18T04:11:30.7722840Z ----------------------------------------------------------------------
2022-05-18T04:11:30.7723181Z Ran 1 test in 0.002s
2022-05-18T04:11:30.7723350Z 
2022-05-18T04:11:30.7723472Z OK (skipped=1)
2022-05-18T04:11:30.7723633Z 
2022-05-18T04:11:30.7723763Z Generating XML reports...
2022-05-18T04:11:30.7759063Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041130.xml
2022-05-18T04:11:32.0113557Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:32.0127701Z 
2022-05-18T04:11:32.0127913Z Running tests...
2022-05-18T04:11:32.0128396Z ----------------------------------------------------------------------
2022-05-18T04:11:32.0147289Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-05-18T04:11:32.0147614Z 
2022-05-18T04:11:32.0147908Z ----------------------------------------------------------------------
2022-05-18T04:11:32.0148239Z Ran 1 test in 0.002s
2022-05-18T04:11:32.0148410Z 
2022-05-18T04:11:32.0148518Z OK (skipped=1)
2022-05-18T04:11:32.0148674Z 
2022-05-18T04:11:32.0148800Z Generating XML reports...
2022-05-18T04:11:32.0183117Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041132.xml
2022-05-18T04:11:33.2720079Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:33.2735984Z 
2022-05-18T04:11:33.2736404Z Running tests...
2022-05-18T04:11:33.2736926Z ----------------------------------------------------------------------
2022-05-18T04:11:34.8510522Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:34.8893449Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8956
2022-05-18T04:11:34.8993383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8957
2022-05-18T04:11:36.0693270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:36.0844045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:36.0844873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:36.0894372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:36.0900469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:36.1859344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:38.8092679Z ok (5.535s)
2022-05-18T04:11:38.8092882Z 
2022-05-18T04:11:38.8093306Z ----------------------------------------------------------------------
2022-05-18T04:11:38.8093650Z Ran 1 test in 5.536s
2022-05-18T04:11:38.8093835Z 
2022-05-18T04:11:38.8093941Z OK
2022-05-18T04:11:38.8094077Z 
2022-05-18T04:11:38.8094198Z Generating XML reports...
2022-05-18T04:11:38.8135515Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041133.xml
2022-05-18T04:11:40.2325683Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:40.2339956Z 
2022-05-18T04:11:40.2340314Z Running tests...
2022-05-18T04:11:40.2340767Z ----------------------------------------------------------------------
2022-05-18T04:11:41.8102950Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:41.8495590Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9073
2022-05-18T04:11:41.8597561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9074
2022-05-18T04:11:43.0226054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:43.0269913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:43.0270708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:43.0326939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:43.0333562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:43.1282528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:43.1392785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:11:43.1393305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:11:43.1394018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:11:43.1394717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:11:43.1396795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:11:43.1397307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:11:43.1397974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:11:43.1399122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:11:44.4152962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjirkjn8y
2022-05-18T04:11:44.4154108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjirkjn8y/_remote_module_non_scriptable.py
2022-05-18T04:11:44.4636218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5spqhm0
2022-05-18T04:11:44.4638191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5spqhm0/_remote_module_non_scriptable.py
2022-05-18T04:11:44.8678098Z ok (4.633s)
2022-05-18T04:11:44.8678611Z 
2022-05-18T04:11:44.8679035Z ----------------------------------------------------------------------
2022-05-18T04:11:44.8679387Z Ran 1 test in 4.634s
2022-05-18T04:11:44.8679555Z 
2022-05-18T04:11:44.8679634Z OK
2022-05-18T04:11:44.8679773Z 
2022-05-18T04:11:44.8679927Z Generating XML reports...
2022-05-18T04:11:44.8720596Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041140.xml
2022-05-18T04:11:46.2939433Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:46.2954800Z 
2022-05-18T04:11:46.2955147Z Running tests...
2022-05-18T04:11:46.2955574Z ----------------------------------------------------------------------
2022-05-18T04:11:47.8610097Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:47.9014384Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9197
2022-05-18T04:11:47.9114956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9198
2022-05-18T04:11:49.0608319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:49.0922420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:49.0923283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:49.1013560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:49.1019852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:49.1933893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:49.2044996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:11:49.2045521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:11:49.2046253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:11:49.2046948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:11:49.2049035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:11:49.2049556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:11:49.2050235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:11:49.2050917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:11:50.4939859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4e_05te
2022-05-18T04:11:50.4940723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4e_05te/_remote_module_non_scriptable.py
2022-05-18T04:11:50.5132500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vmvjao7
2022-05-18T04:11:50.5135387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vmvjao7/_remote_module_non_scriptable.py
2022-05-18T04:11:50.9195373Z ok (4.624s)
2022-05-18T04:11:50.9195584Z 
2022-05-18T04:11:50.9196023Z ----------------------------------------------------------------------
2022-05-18T04:11:50.9196372Z Ran 1 test in 4.624s
2022-05-18T04:11:50.9196539Z 
2022-05-18T04:11:50.9196618Z OK
2022-05-18T04:11:50.9196754Z 
2022-05-18T04:11:50.9196890Z Generating XML reports...
2022-05-18T04:11:50.9237823Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041146.xml
2022-05-18T04:11:52.3012096Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:52.3027029Z 
2022-05-18T04:11:52.3027426Z Running tests...
2022-05-18T04:11:52.3027863Z ----------------------------------------------------------------------
2022-05-18T04:11:53.8862148Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:53.9272613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9321
2022-05-18T04:11:53.9374350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9322
2022-05-18T04:11:55.0799547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:55.1015668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:55.1016483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:55.1104976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:55.1111687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:55.2030814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:56.3746447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpih14d96b
2022-05-18T04:11:56.3747310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpih14d96b/_remote_module_non_scriptable.py
2022-05-18T04:11:56.4384905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1xbxyob
2022-05-18T04:11:56.4386178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1xbxyob/_remote_module_non_scriptable.py
2022-05-18T04:11:56.6867153Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.6867714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.9455479Z ok (4.642s)
2022-05-18T04:11:56.9455701Z 
2022-05-18T04:11:56.9456219Z ----------------------------------------------------------------------
2022-05-18T04:11:56.9456759Z Ran 1 test in 4.643s
2022-05-18T04:11:56.9456929Z 
2022-05-18T04:11:56.9457027Z OK
2022-05-18T04:11:56.9459082Z 
2022-05-18T04:11:56.9459520Z Generating XML reports...
2022-05-18T04:11:56.9498358Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041152.xml
2022-05-18T04:11:58.3679582Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:11:58.3693870Z 
2022-05-18T04:11:58.3694451Z Running tests...
2022-05-18T04:11:58.3694957Z ----------------------------------------------------------------------
2022-05-18T04:11:59.9416729Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:59.9801344Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9441
2022-05-18T04:11:59.9903408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9442
2022-05-18T04:12:01.1541755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:12:01.1686840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:12:01.1687710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:01.1743986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:01.1750010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:01.2700843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:02.4247059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_2mlnmy
2022-05-18T04:12:02.4247675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_2mlnmy/_remote_module_non_scriptable.py
2022-05-18T04:12:02.5479891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1l1u78ck
2022-05-18T04:12:02.5481077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1l1u78ck/_remote_module_non_scriptable.py
2022-05-18T04:12:02.8021395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.8022458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.8027454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.8031364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:03.0998880Z ok (4.730s)
2022-05-18T04:12:03.0999270Z 
2022-05-18T04:12:03.0999795Z ----------------------------------------------------------------------
2022-05-18T04:12:03.1000124Z Ran 1 test in 4.730s
2022-05-18T04:12:03.1000311Z 
2022-05-18T04:12:03.1000408Z OK
2022-05-18T04:12:03.1000545Z 
2022-05-18T04:12:03.1000678Z Generating XML reports...
2022-05-18T04:12:03.1040033Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041158.xml
2022-05-18T04:12:04.5316262Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:12:04.5331313Z 
2022-05-18T04:12:04.5331590Z Running tests...
2022-05-18T04:12:04.5332043Z ----------------------------------------------------------------------
2022-05-18T04:12:06.1089808Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:06.1481951Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9561
2022-05-18T04:12:06.1584132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9562
2022-05-18T04:12:07.3385854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:12:07.3475383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:12:07.3476203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:07.3487117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:07.3493197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:07.4490089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:08.6371685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoqtvt70r
2022-05-18T04:12:08.6372328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoqtvt70r/_remote_module_non_scriptable.py
2022-05-18T04:12:08.6995824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8oxqedeu
2022-05-18T04:12:08.6997380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8oxqedeu/_remote_module_non_scriptable.py
2022-05-18T04:12:08.9661100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:08.9661655Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:08.9669732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:08.9670210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:08.9793308Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:08.9793788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:08.9802827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:08.9803305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:09.2665386Z ok (4.733s)
2022-05-18T04:12:09.2665764Z 
2022-05-18T04:12:09.2666329Z ----------------------------------------------------------------------
2022-05-18T04:12:09.2666685Z Ran 1 test in 4.733s
2022-05-18T04:12:09.2666864Z 
2022-05-18T04:12:09.2666972Z OK
2022-05-18T04:12:09.2667109Z 
2022-05-18T04:12:09.2667242Z Generating XML reports...
2022-05-18T04:12:09.2707774Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041204.xml
2022-05-18T04:12:10.6954385Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:12:10.6969528Z 
2022-05-18T04:12:10.6969842Z Running tests...
2022-05-18T04:12:10.6970283Z ----------------------------------------------------------------------
2022-05-18T04:12:12.2670533Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:12.2826274Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.585s)
2022-05-18T04:12:12.2826983Z 
2022-05-18T04:12:12.2827281Z ----------------------------------------------------------------------
2022-05-18T04:12:12.2827633Z Ran 1 test in 1.586s
2022-05-18T04:12:12.2827789Z 
2022-05-18T04:12:12.2827907Z OK (skipped=1)
2022-05-18T04:12:12.2828073Z 
2022-05-18T04:12:12.2830227Z Generating XML reports...
2022-05-18T04:12:12.2861395Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041210.xml
2022-05-18T04:12:13.6629367Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:12:13.6644144Z 
2022-05-18T04:12:13.6644561Z Running tests...
2022-05-18T04:12:13.6645519Z ----------------------------------------------------------------------
2022-05-18T04:12:15.2433459Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:15.2827486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9715
2022-05-18T04:12:15.2930495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9716
2022-05-18T04:12:16.4281574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:12:16.4342010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:12:16.4342831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:16.4383007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:16.4389534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:16.5356242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:17.7079878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpht8ue25e
2022-05-18T04:12:17.7085599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpht8ue25e/_remote_module_non_scriptable.py
2022-05-18T04:12:17.7909710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqti2x8i0
2022-05-18T04:12:17.7910817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqti2x8i0/_remote_module_non_scriptable.py
2022-05-18T04:12:17.8725087Z <class 'torch.nn.parameter.Parameter'>
2022-05-18T04:12:18.2010344Z ok (4.536s)
2022-05-18T04:12:18.2010591Z 
2022-05-18T04:12:18.2010998Z ----------------------------------------------------------------------
2022-05-18T04:12:18.2011372Z Ran 1 test in 4.537s
2022-05-18T04:12:18.2011546Z 
2022-05-18T04:12:18.2011624Z OK
2022-05-18T04:12:18.2011777Z 
2022-05-18T04:12:18.2011912Z Generating XML reports...
2022-05-18T04:12:18.2054938Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041213.xml
2022-05-18T04:12:19.5994398Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:12:19.6008333Z 
2022-05-18T04:12:19.6008584Z Running tests...
2022-05-18T04:12:19.6009041Z ----------------------------------------------------------------------
2022-05-18T04:12:21.1443957Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:21.1830649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9831
2022-05-18T04:12:21.1931716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9832
2022-05-18T04:12:22.3248816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:12:22.3570528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:12:22.3571589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:22.3655645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:22.3661960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:22.4584855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:23.6212386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkd9h_msg
2022-05-18T04:12:23.6212980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkd9h_msg/_remote_module_non_scriptable.py
2022-05-18T04:12:23.7021325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbu6g3qi
2022-05-18T04:12:23.7022182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbu6g3qi/_remote_module_non_scriptable.py
2022-05-18T04:12:24.1012191Z ok (4.500s)
2022-05-18T04:12:24.1012519Z 
2022-05-18T04:12:24.1012931Z ----------------------------------------------------------------------
2022-05-18T04:12:24.1013320Z Ran 1 test in 4.500s
2022-05-18T04:12:24.1013638Z 
2022-05-18T04:12:24.1013784Z OK
2022-05-18T04:12:24.1013949Z 
2022-05-18T04:12:24.1014087Z Generating XML reports...
2022-05-18T04:12:24.1056122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041219.xml
2022-05-18T04:12:25.5131879Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:12:25.5145918Z 
2022-05-18T04:12:25.5146243Z Running tests...
2022-05-18T04:12:25.5147034Z ----------------------------------------------------------------------
2022-05-18T04:12:27.1034211Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:27.1432918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9947
2022-05-18T04:12:27.1537634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9948
2022-05-18T04:12:28.3078026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:12:28.3287422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:12:28.3288270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:28.3381978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:28.3388402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:28.4301557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:29.6150490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxw1k4bx
2022-05-18T04:12:29.6151105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxw1k4bx/_remote_module_non_scriptable.py
2022-05-18T04:12:29.6877806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjr6fdsbw
2022-05-18T04:12:29.6879028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjr6fdsbw/_remote_module_non_scriptable.py
2022-05-18T04:12:29.9462034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:29.9462560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:30.2619795Z ok (4.747s)
2022-05-18T04:12:30.2619992Z 
2022-05-18T04:12:30.2620451Z ----------------------------------------------------------------------
2022-05-18T04:12:30.2620792Z Ran 1 test in 4.747s
2022-05-18T04:12:30.2620959Z 
2022-05-18T04:12:30.2621054Z OK
2022-05-18T04:12:30.2621196Z 
2022-05-18T04:12:30.2621313Z Generating XML reports...
2022-05-18T04:12:30.2661624Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041225.xml
2022-05-18T04:12:31.6892625Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:12:31.6906961Z 
2022-05-18T04:12:31.6907263Z Running tests...
2022-05-18T04:12:31.6907706Z ----------------------------------------------------------------------
2022-05-18T04:12:33.2559429Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:33.2947525Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10067
2022-05-18T04:12:33.3048518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10068
2022-05-18T04:12:34.4334802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:12:34.4578327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:12:34.4579160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:34.4638519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:34.4645412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:34.5592449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:35.7046036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ya3mj0a
2022-05-18T04:12:35.7046985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ya3mj0a/_remote_module_non_scriptable.py
2022-05-18T04:12:35.8079519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpow8j82r2
2022-05-18T04:12:35.8080878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpow8j82r2/_remote_module_non_scriptable.py
2022-05-18T04:12:36.0553491Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:12:36.4141516Z ok (4.723s)
2022-05-18T04:12:36.4141761Z 
2022-05-18T04:12:36.4142158Z ----------------------------------------------------------------------
2022-05-18T04:12:36.4142500Z Ran 1 test in 4.723s
2022-05-18T04:12:36.4142665Z 
2022-05-18T04:12:36.4142743Z OK
2022-05-18T04:12:36.4142877Z 
2022-05-18T04:12:36.4143009Z Generating XML reports...
2022-05-18T04:12:36.4183277Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041231.xml
2022-05-18T04:12:37.8105100Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:12:37.8119392Z 
2022-05-18T04:12:37.8119778Z Running tests...
2022-05-18T04:12:37.8120225Z ----------------------------------------------------------------------
2022-05-18T04:12:39.3702831Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:39.4088818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10187
2022-05-18T04:12:39.4189421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10188
2022-05-18T04:12:40.5980485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:12:40.5982490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:12:40.5983335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:40.6081695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:40.6088026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:40.6996601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:41.8902116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeavnvrf2
2022-05-18T04:12:41.8902759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeavnvrf2/_remote_module_non_scriptable.py
2022-05-18T04:12:41.9391650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpng9p3kwh
2022-05-18T04:12:41.9393684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpng9p3kwh/_remote_module_non_scriptable.py
2022-05-18T04:12:42.1878820Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:12:42.1887265Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:12:42.5268732Z ok (4.714s)
2022-05-18T04:12:42.5268954Z 
2022-05-18T04:12:42.5269342Z ----------------------------------------------------------------------
2022-05-18T04:12:42.5269692Z Ran 1 test in 4.715s
2022-05-18T04:12:42.5269847Z 
2022-05-18T04:12:42.5269954Z OK
2022-05-18T04:12:42.5270090Z 
2022-05-18T04:12:42.5270226Z Generating XML reports...
2022-05-18T04:12:42.5310533Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041237.xml
2022-05-18T04:12:43.9166500Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:12:43.9179856Z 
2022-05-18T04:12:43.9180125Z Running tests...
2022-05-18T04:12:43.9180552Z ----------------------------------------------------------------------
2022-05-18T04:12:43.9204381Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2022-05-18T04:12:43.9204762Z 
2022-05-18T04:12:43.9205234Z ----------------------------------------------------------------------
2022-05-18T04:12:43.9205583Z Ran 1 test in 0.002s
2022-05-18T04:12:43.9205749Z 
2022-05-18T04:12:43.9205842Z OK (skipped=1)
2022-05-18T04:12:43.9205999Z 
2022-05-18T04:12:43.9206130Z Generating XML reports...
2022-05-18T04:12:43.9239731Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041243.xml
2022-05-18T04:12:45.1568493Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:12:45.1582770Z 
2022-05-18T04:12:45.1583071Z Running tests...
2022-05-18T04:12:45.1583517Z ----------------------------------------------------------------------
2022-05-18T04:12:46.7536822Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:46.7919498Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10340
2022-05-18T04:12:46.8021605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10341
2022-05-18T04:12:47.9405407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:12:47.9626311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:12:47.9627125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:47.9709078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:47.9715764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:48.0640365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:49.2469608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpefab22vw
2022-05-18T04:12:49.2470222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpefab22vw/_remote_module_non_scriptable.py
2022-05-18T04:12:49.3159115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpec1f8gf9
2022-05-18T04:12:49.3159725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpec1f8gf9/_remote_module_non_scriptable.py
2022-05-18T04:12:49.5643498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:49.5644077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:49.9105415Z ok (4.752s)
2022-05-18T04:12:49.9105624Z 
2022-05-18T04:12:49.9106152Z ----------------------------------------------------------------------
2022-05-18T04:12:49.9106500Z Ran 1 test in 4.752s
2022-05-18T04:12:49.9106654Z 
2022-05-18T04:12:49.9106760Z OK
2022-05-18T04:12:49.9106898Z 
2022-05-18T04:12:49.9107047Z Generating XML reports...
2022-05-18T04:12:49.9147915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041245.xml
2022-05-18T04:12:51.3294554Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:12:51.3308761Z 
2022-05-18T04:12:51.3309156Z Running tests...
2022-05-18T04:12:51.3309657Z ----------------------------------------------------------------------
2022-05-18T04:12:52.9132856Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:52.9515557Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10460
2022-05-18T04:12:52.9618117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10461
2022-05-18T04:12:54.0974915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:12:54.1363609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:12:54.1364434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:54.1379355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:12:54.1385234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:54.2377787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:55.3797724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppeap72b2
2022-05-18T04:12:55.3798697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppeap72b2/_remote_module_non_scriptable.py
2022-05-18T04:12:55.4897934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg20oli5c
2022-05-18T04:12:55.4899323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg20oli5c/_remote_module_non_scriptable.py
2022-05-18T04:12:55.5648451Z /opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T04:12:55.5650920Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T04:12:55.5653463Z /opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T04:12:55.5655327Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T04:12:56.0700256Z ok (4.739s)
2022-05-18T04:12:56.0700486Z 
2022-05-18T04:12:56.0700882Z ----------------------------------------------------------------------
2022-05-18T04:12:56.0701240Z Ran 1 test in 4.739s
2022-05-18T04:12:56.0701408Z 
2022-05-18T04:12:56.0701510Z OK
2022-05-18T04:12:56.0701892Z 
2022-05-18T04:12:56.0702173Z Generating XML reports...
2022-05-18T04:12:56.0742253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041251.xml
2022-05-18T04:12:57.4929101Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:12:57.4945012Z 
2022-05-18T04:12:57.4945386Z Running tests...
2022-05-18T04:12:57.4946280Z ----------------------------------------------------------------------
2022-05-18T04:12:59.0825522Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:59.1220076Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10610
2022-05-18T04:12:59.1324594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10611
2022-05-18T04:13:00.2804576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:13:00.2815234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:13:00.2816068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:00.2907843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:00.2913176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:00.3828711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:01.5495293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjq99a8hu
2022-05-18T04:13:01.5495922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjq99a8hu/_remote_module_non_scriptable.py
2022-05-18T04:13:01.6108347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppa506023
2022-05-18T04:13:01.6109733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppa506023/_remote_module_non_scriptable.py
2022-05-18T04:13:01.8702672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:01.8703240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:01.8876469Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmppa506023'>
2022-05-18T04:13:01.8877563Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:13:01.8879562Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpjq99a8hu'>
2022-05-18T04:13:01.8880583Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:13:02.2406572Z ok (4.746s)
2022-05-18T04:13:02.2406799Z 
2022-05-18T04:13:02.2407193Z ----------------------------------------------------------------------
2022-05-18T04:13:02.2407534Z Ran 1 test in 4.746s
2022-05-18T04:13:02.2407700Z 
2022-05-18T04:13:02.2407797Z OK
2022-05-18T04:13:02.2407918Z 
2022-05-18T04:13:02.2408054Z Generating XML reports...
2022-05-18T04:13:02.2448710Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041257.xml
2022-05-18T04:13:03.6574800Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:03.6588863Z 
2022-05-18T04:13:03.6589280Z Running tests...
2022-05-18T04:13:03.6589760Z ----------------------------------------------------------------------
2022-05-18T04:13:05.2360286Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:05.2510856Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.592s)
2022-05-18T04:13:05.2511430Z 
2022-05-18T04:13:05.2511710Z ----------------------------------------------------------------------
2022-05-18T04:13:05.2512026Z Ran 1 test in 1.592s
2022-05-18T04:13:05.2512193Z 
2022-05-18T04:13:05.2512303Z OK (skipped=1)
2022-05-18T04:13:05.2512758Z 
2022-05-18T04:13:05.2512889Z Generating XML reports...
2022-05-18T04:13:05.2545353Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041303.xml
2022-05-18T04:13:06.6316707Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:06.6331725Z 
2022-05-18T04:13:06.6332114Z Running tests...
2022-05-18T04:13:06.6332773Z ----------------------------------------------------------------------
2022-05-18T04:13:08.2100970Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:08.2497521Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10764
2022-05-18T04:13:08.2603634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10765
2022-05-18T04:13:09.3974246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:13:09.4143191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:13:09.4144017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:09.4176846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:09.4183222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:09.4186701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:13:09.5155210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:09.5158626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:13:09.5159337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:13:09.5206214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:13:10.8062343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgyr3yxzr
2022-05-18T04:13:10.8063244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgyr3yxzr/_remote_module_non_scriptable.py
2022-05-18T04:13:10.8207713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrp0zbw_
2022-05-18T04:13:10.8210826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrp0zbw_/_remote_module_non_scriptable.py
2022-05-18T04:13:11.0693807Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:11.0694329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:11.0714979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:11.0715510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:11.5690444Z ok (4.935s)
2022-05-18T04:13:11.5690672Z 
2022-05-18T04:13:11.5691251Z ----------------------------------------------------------------------
2022-05-18T04:13:11.5691700Z Ran 1 test in 4.936s
2022-05-18T04:13:11.5691871Z 
2022-05-18T04:13:11.5691969Z OK
2022-05-18T04:13:11.5692113Z 
2022-05-18T04:13:11.5692249Z Generating XML reports...
2022-05-18T04:13:11.5732159Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041306.xml
2022-05-18T04:13:12.9948269Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:12.9963544Z 
2022-05-18T04:13:12.9963887Z Running tests...
2022-05-18T04:13:12.9964345Z ----------------------------------------------------------------------
2022-05-18T04:13:14.5703821Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:14.6090602Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10888
2022-05-18T04:13:14.6192728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10889
2022-05-18T04:13:15.7681301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:13:15.8015383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:13:15.8016220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:15.8086544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:15.8092654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:15.8094925Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:13:15.9030909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:15.9031942Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:13:17.0469868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddk16nbv
2022-05-18T04:13:17.0470488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddk16nbv/_remote_module_non_scriptable.py
2022-05-18T04:13:17.1495378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vw22i2o
2022-05-18T04:13:17.1497082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vw22i2o/_remote_module_non_scriptable.py
2022-05-18T04:13:17.4042342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:17.4046255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:17.4065707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:17.4069848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:17.4269284Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:13:17.4272802Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:13:17.6303952Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:13:17.6306582Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:13:17.6364562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:17.6367499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:17.6375748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:17.6379314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:17.6576877Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:13:17.6580444Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:13:17.7853565Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T04:13:17.7862396Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T04:13:17.7918491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:17.7921544Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:17.7929685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:17.7933326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:18.3283391Z ok (5.332s)
2022-05-18T04:13:18.3283603Z 
2022-05-18T04:13:18.3284127Z ----------------------------------------------------------------------
2022-05-18T04:13:18.3284622Z Ran 1 test in 5.332s
2022-05-18T04:13:18.3284792Z 
2022-05-18T04:13:18.3284897Z OK
2022-05-18T04:13:18.3285039Z 
2022-05-18T04:13:18.3285177Z Generating XML reports...
2022-05-18T04:13:18.3326175Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041312.xml
2022-05-18T04:13:19.7500820Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:19.7515553Z 
2022-05-18T04:13:19.7515801Z Running tests...
2022-05-18T04:13:19.7516358Z ----------------------------------------------------------------------
2022-05-18T04:13:21.3273872Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:21.3668454Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11008
2022-05-18T04:13:21.3772405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11009
2022-05-18T04:13:22.5538657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:13:22.5539228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:13:22.5540026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:22.5540712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:22.5546345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:22.5547129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:22.5548418Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:13:22.5550086Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:13:23.8452790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvr7fmh5
2022-05-18T04:13:23.8453898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvr7fmh5/_remote_module_non_scriptable.py
2022-05-18T04:13:23.8655133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwzn2a_g
2022-05-18T04:13:23.8657826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwzn2a_g/_remote_module_non_scriptable.py
2022-05-18T04:13:24.1273181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:24.1275873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:24.1295810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:24.1299537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:24.1302752Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:13:24.1306607Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:13:24.1329879Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:13:24.1332417Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:13:24.1333174Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:13:24.1333786Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T04:13:24.1335317Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:13:24.1336415Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T04:13:24.4137695Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:13:24.4141160Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:13:24.4201790Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:24.4203047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:24.4212650Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:24.4215277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:24.4218682Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:13:24.4221261Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:13:24.4244573Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:13:24.4246307Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:13:24.4246959Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:13:24.4249088Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:13:24.9860628Z ok (5.234s)
2022-05-18T04:13:24.9860985Z 
2022-05-18T04:13:24.9861756Z ----------------------------------------------------------------------
2022-05-18T04:13:24.9862395Z Ran 1 test in 5.234s
2022-05-18T04:13:24.9862571Z 
2022-05-18T04:13:24.9862650Z OK
2022-05-18T04:13:24.9862796Z 
2022-05-18T04:13:24.9862935Z Generating XML reports...
2022-05-18T04:13:24.9903279Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041319.xml
2022-05-18T04:13:26.4013499Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:26.4028260Z 
2022-05-18T04:13:26.4028501Z Running tests...
2022-05-18T04:13:26.4029370Z ----------------------------------------------------------------------
2022-05-18T04:13:26.4051746Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:13:26.4052730Z 
2022-05-18T04:13:26.4053140Z ----------------------------------------------------------------------
2022-05-18T04:13:26.4053482Z Ran 1 test in 0.002s
2022-05-18T04:13:26.4053648Z 
2022-05-18T04:13:26.4053742Z OK (skipped=1)
2022-05-18T04:13:26.4053900Z 
2022-05-18T04:13:26.4054030Z Generating XML reports...
2022-05-18T04:13:26.4089536Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041326.xml
2022-05-18T04:13:27.6512833Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:27.6527999Z 
2022-05-18T04:13:27.6528538Z Running tests...
2022-05-18T04:13:27.6529043Z ----------------------------------------------------------------------
2022-05-18T04:13:27.6550100Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:13:27.6550535Z 
2022-05-18T04:13:27.6550821Z ----------------------------------------------------------------------
2022-05-18T04:13:27.6551159Z Ran 1 test in 0.002s
2022-05-18T04:13:27.6551305Z 
2022-05-18T04:13:27.6551421Z OK (skipped=1)
2022-05-18T04:13:27.6551578Z 
2022-05-18T04:13:27.6551706Z Generating XML reports...
2022-05-18T04:13:27.6587427Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041327.xml
2022-05-18T04:13:28.8910939Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:28.8926078Z 
2022-05-18T04:13:28.8926371Z Running tests...
2022-05-18T04:13:28.8926827Z ----------------------------------------------------------------------
2022-05-18T04:13:28.8950476Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:13:28.8951011Z 
2022-05-18T04:13:28.8951304Z ----------------------------------------------------------------------
2022-05-18T04:13:28.8951645Z Ran 1 test in 0.002s
2022-05-18T04:13:28.8951791Z 
2022-05-18T04:13:28.8951902Z OK (skipped=1)
2022-05-18T04:13:28.8952058Z 
2022-05-18T04:13:28.8952185Z Generating XML reports...
2022-05-18T04:13:28.8988162Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041328.xml
2022-05-18T04:13:30.1557108Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:30.1572641Z 
2022-05-18T04:13:30.1572975Z Running tests...
2022-05-18T04:13:30.1573434Z ----------------------------------------------------------------------
2022-05-18T04:13:30.1597503Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:13:30.1598017Z 
2022-05-18T04:13:30.1598555Z ----------------------------------------------------------------------
2022-05-18T04:13:30.1598902Z Ran 1 test in 0.003s
2022-05-18T04:13:30.1599073Z 
2022-05-18T04:13:30.1599187Z OK (skipped=1)
2022-05-18T04:13:30.1599345Z 
2022-05-18T04:13:30.1599474Z Generating XML reports...
2022-05-18T04:13:30.1636153Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041330.xml
2022-05-18T04:13:31.3920803Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:31.3934876Z 
2022-05-18T04:13:31.3935189Z Running tests...
2022-05-18T04:13:31.3935906Z ----------------------------------------------------------------------
2022-05-18T04:13:31.3959186Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:13:31.3959821Z 
2022-05-18T04:13:31.3960251Z ----------------------------------------------------------------------
2022-05-18T04:13:31.3960598Z Ran 1 test in 0.002s
2022-05-18T04:13:31.3960765Z 
2022-05-18T04:13:31.3960891Z OK (skipped=1)
2022-05-18T04:13:31.3961127Z 
2022-05-18T04:13:31.3961369Z Generating XML reports...
2022-05-18T04:13:31.3995606Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041331.xml
2022-05-18T04:13:32.6204865Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:32.6218168Z 
2022-05-18T04:13:32.6218430Z Running tests...
2022-05-18T04:13:32.6218867Z ----------------------------------------------------------------------
2022-05-18T04:13:32.6241613Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:13:32.6242150Z 
2022-05-18T04:13:32.6242478Z ----------------------------------------------------------------------
2022-05-18T04:13:32.6242823Z Ran 1 test in 0.002s
2022-05-18T04:13:32.6242992Z 
2022-05-18T04:13:32.6243086Z OK (skipped=1)
2022-05-18T04:13:32.6243246Z 
2022-05-18T04:13:32.6243380Z Generating XML reports...
2022-05-18T04:13:32.6276508Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041332.xml
2022-05-18T04:13:33.8717833Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:33.8733828Z 
2022-05-18T04:13:33.8734337Z Running tests...
2022-05-18T04:13:33.8735133Z ----------------------------------------------------------------------
2022-05-18T04:13:33.8757754Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:13:33.8758614Z 
2022-05-18T04:13:33.8758926Z ----------------------------------------------------------------------
2022-05-18T04:13:33.8759286Z Ran 1 test in 0.002s
2022-05-18T04:13:33.8759433Z 
2022-05-18T04:13:33.8759548Z OK (skipped=1)
2022-05-18T04:13:33.8759707Z 
2022-05-18T04:13:33.8759836Z Generating XML reports...
2022-05-18T04:13:33.8795725Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041333.xml
2022-05-18T04:13:35.1367428Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:35.1381845Z 
2022-05-18T04:13:35.1382118Z Running tests...
2022-05-18T04:13:35.1382594Z ----------------------------------------------------------------------
2022-05-18T04:13:35.1406200Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:13:35.1406747Z 
2022-05-18T04:13:35.1407262Z ----------------------------------------------------------------------
2022-05-18T04:13:35.1407591Z Ran 1 test in 0.002s
2022-05-18T04:13:35.1407758Z 
2022-05-18T04:13:35.1407869Z OK (skipped=1)
2022-05-18T04:13:35.1408025Z 
2022-05-18T04:13:35.1408237Z Generating XML reports...
2022-05-18T04:13:35.1443708Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041335.xml
2022-05-18T04:13:36.3520682Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:36.3536353Z 
2022-05-18T04:13:36.3536669Z Running tests...
2022-05-18T04:13:36.3537120Z ----------------------------------------------------------------------
2022-05-18T04:13:36.3562828Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:13:36.3563335Z 
2022-05-18T04:13:36.3563624Z ----------------------------------------------------------------------
2022-05-18T04:13:36.3563962Z Ran 1 test in 0.003s
2022-05-18T04:13:36.3564128Z 
2022-05-18T04:13:36.3564245Z OK (skipped=1)
2022-05-18T04:13:36.3564398Z 
2022-05-18T04:13:36.3566732Z Generating XML reports...
2022-05-18T04:13:36.3600033Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041336.xml
2022-05-18T04:13:37.6135897Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:37.6150689Z 
2022-05-18T04:13:37.6151129Z Running tests...
2022-05-18T04:13:37.6151625Z ----------------------------------------------------------------------
2022-05-18T04:13:37.6175126Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:13:37.6175969Z 
2022-05-18T04:13:37.6176289Z ----------------------------------------------------------------------
2022-05-18T04:13:37.6176630Z Ran 1 test in 0.002s
2022-05-18T04:13:37.6176799Z 
2022-05-18T04:13:37.6176908Z OK (skipped=1)
2022-05-18T04:13:37.6177068Z 
2022-05-18T04:13:37.6177196Z Generating XML reports...
2022-05-18T04:13:37.6213346Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041337.xml
2022-05-18T04:13:38.8721465Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:38.8736935Z 
2022-05-18T04:13:38.8737213Z Running tests...
2022-05-18T04:13:38.8737641Z ----------------------------------------------------------------------
2022-05-18T04:13:38.8760296Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:13:38.8760721Z 
2022-05-18T04:13:38.8761004Z ----------------------------------------------------------------------
2022-05-18T04:13:38.8761344Z Ran 1 test in 0.002s
2022-05-18T04:13:38.8761515Z 
2022-05-18T04:13:38.8761646Z OK (skipped=1)
2022-05-18T04:13:38.8761788Z 
2022-05-18T04:13:38.8761916Z Generating XML reports...
2022-05-18T04:13:38.8796875Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041338.xml
2022-05-18T04:13:40.1244464Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:40.1259075Z 
2022-05-18T04:13:40.1259357Z Running tests...
2022-05-18T04:13:40.1259811Z ----------------------------------------------------------------------
2022-05-18T04:13:40.1281936Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:13:40.1282395Z 
2022-05-18T04:13:40.1282893Z ----------------------------------------------------------------------
2022-05-18T04:13:40.1283258Z Ran 1 test in 0.002s
2022-05-18T04:13:40.1283424Z 
2022-05-18T04:13:40.1283828Z OK (skipped=1)
2022-05-18T04:13:40.1283987Z 
2022-05-18T04:13:40.1284116Z Generating XML reports...
2022-05-18T04:13:40.1319175Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041340.xml
2022-05-18T04:13:41.3826755Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:41.3841675Z 
2022-05-18T04:13:41.3841960Z Running tests...
2022-05-18T04:13:41.3842400Z ----------------------------------------------------------------------
2022-05-18T04:13:42.9384787Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:42.9533640Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.569s)
2022-05-18T04:13:42.9534238Z 
2022-05-18T04:13:42.9534503Z ----------------------------------------------------------------------
2022-05-18T04:13:42.9534847Z Ran 1 test in 1.569s
2022-05-18T04:13:42.9535013Z 
2022-05-18T04:13:42.9535122Z OK (skipped=1)
2022-05-18T04:13:42.9535279Z 
2022-05-18T04:13:42.9535407Z Generating XML reports...
2022-05-18T04:13:42.9566343Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041341.xml
2022-05-18T04:13:44.3257551Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:44.3272107Z 
2022-05-18T04:13:44.3272383Z Running tests...
2022-05-18T04:13:44.3272832Z ----------------------------------------------------------------------
2022-05-18T04:13:45.9040356Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:45.9433678Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11558
2022-05-18T04:13:45.9537057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11559
2022-05-18T04:13:47.1136756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:13:47.1203459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:13:47.1204343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:47.1238024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:47.1244123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:47.2217995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:48.3795235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp705rruqq
2022-05-18T04:13:48.3795869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp705rruqq/_remote_module_non_scriptable.py
2022-05-18T04:13:48.4655047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvsr3ky6m
2022-05-18T04:13:48.4656824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvsr3ky6m/_remote_module_non_scriptable.py
2022-05-18T04:13:49.6629040Z ok (5.335s)
2022-05-18T04:13:49.6629262Z 
2022-05-18T04:13:49.6629650Z ----------------------------------------------------------------------
2022-05-18T04:13:49.6630000Z Ran 1 test in 5.336s
2022-05-18T04:13:49.6630148Z 
2022-05-18T04:13:49.6630249Z OK
2022-05-18T04:13:49.6630387Z 
2022-05-18T04:13:49.6630523Z Generating XML reports...
2022-05-18T04:13:49.6671407Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041344.xml
2022-05-18T04:13:51.0834006Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:51.0849359Z 
2022-05-18T04:13:51.0849799Z Running tests...
2022-05-18T04:13:51.0850297Z ----------------------------------------------------------------------
2022-05-18T04:13:52.6459679Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:52.6844066Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11674
2022-05-18T04:13:52.6946249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11675
2022-05-18T04:13:53.8545360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:13:53.8830234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:13:53.8831322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:53.8849331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:53.8855702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:53.9846904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:55.2930886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57qnw5_t
2022-05-18T04:13:55.2931489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57qnw5_t/_remote_module_non_scriptable.py
2022-05-18T04:13:55.4382093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5y5ukgtg
2022-05-18T04:13:55.4383292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5y5ukgtg/_remote_module_non_scriptable.py
2022-05-18T04:13:55.5182781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:55.5183302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:55.5351603Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp57qnw5_t'>
2022-05-18T04:13:55.5352081Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:13:55.5352890Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp5y5ukgtg'>
2022-05-18T04:13:55.5353351Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:13:55.8025523Z ok (4.717s)
2022-05-18T04:13:55.8026015Z 
2022-05-18T04:13:55.8026386Z ----------------------------------------------------------------------
2022-05-18T04:13:55.8026746Z Ran 1 test in 4.718s
2022-05-18T04:13:55.8026917Z 
2022-05-18T04:13:55.8027011Z OK
2022-05-18T04:13:55.8027150Z 
2022-05-18T04:13:55.8027283Z Generating XML reports...
2022-05-18T04:13:55.8068527Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041351.xml
2022-05-18T04:13:57.2186476Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:57.2201313Z 
2022-05-18T04:13:57.2201630Z Running tests...
2022-05-18T04:13:57.2202333Z ----------------------------------------------------------------------
2022-05-18T04:13:57.2270244Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.007s)
2022-05-18T04:13:57.2270987Z 
2022-05-18T04:13:57.2271540Z ----------------------------------------------------------------------
2022-05-18T04:13:57.2271916Z Ran 1 test in 0.007s
2022-05-18T04:13:57.2272089Z 
2022-05-18T04:13:57.2272216Z OK (skipped=1)
2022-05-18T04:13:57.2272361Z 
2022-05-18T04:13:57.2272493Z Generating XML reports...
2022-05-18T04:13:57.2307621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041357.xml
2022-05-18T04:13:58.4758694Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:13:58.4774254Z 
2022-05-18T04:13:58.4774524Z Running tests...
2022-05-18T04:13:58.4775275Z ----------------------------------------------------------------------
2022-05-18T04:14:00.0671850Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:00.1065505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11827
2022-05-18T04:14:00.1169023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11828
2022-05-18T04:14:01.2533041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:14:01.2618963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:14:01.2619992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:01.2634218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:01.2640644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:01.3634903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:02.5265596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0vtog1r
2022-05-18T04:14:02.5266609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0vtog1r/_remote_module_non_scriptable.py
2022-05-18T04:14:02.6449020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigagns38
2022-05-18T04:14:02.6450541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigagns38/_remote_module_non_scriptable.py
2022-05-18T04:14:02.8947593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:02.8965650Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:03.2252907Z ok (4.747s)
2022-05-18T04:14:03.2253127Z 
2022-05-18T04:14:03.2253529Z ----------------------------------------------------------------------
2022-05-18T04:14:03.2253887Z Ran 1 test in 4.748s
2022-05-18T04:14:03.2254037Z 
2022-05-18T04:14:03.2254136Z OK
2022-05-18T04:14:03.2254537Z 
2022-05-18T04:14:03.2254690Z Generating XML reports...
2022-05-18T04:14:03.2295795Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041358.xml
2022-05-18T04:14:04.6434660Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:14:04.6449898Z 
2022-05-18T04:14:04.6450361Z Running tests...
2022-05-18T04:14:04.6450962Z ----------------------------------------------------------------------
2022-05-18T04:14:06.2382459Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:06.2774749Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11947
2022-05-18T04:14:06.2876450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11948
2022-05-18T04:14:07.4346226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:14:07.4598163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:14:07.4599211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:07.4649897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:07.4656391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:07.5609234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:07.5773590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:14:07.5774477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:14:07.5775281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:14:07.5775966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:14:07.5777086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:14:07.5777586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:14:07.5778324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:14:07.5779006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:14:08.8518512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39wwnznw
2022-05-18T04:14:08.8519165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39wwnznw/_remote_module_non_scriptable.py
2022-05-18T04:14:08.8821942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpah3ymga9
2022-05-18T04:14:08.8824329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpah3ymga9/_remote_module_non_scriptable.py
2022-05-18T04:14:09.2957776Z ok (4.650s)
2022-05-18T04:14:09.2957971Z 
2022-05-18T04:14:09.2958650Z ----------------------------------------------------------------------
2022-05-18T04:14:09.2959005Z Ran 1 test in 4.651s
2022-05-18T04:14:09.2959153Z 
2022-05-18T04:14:09.2959261Z OK
2022-05-18T04:14:09.2959398Z 
2022-05-18T04:14:09.2959532Z Generating XML reports...
2022-05-18T04:14:09.3000583Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041404.xml
2022-05-18T04:14:10.7175430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:14:10.7189406Z 
2022-05-18T04:14:10.7189569Z Running tests...
2022-05-18T04:14:10.7191147Z ----------------------------------------------------------------------
2022-05-18T04:14:12.3087160Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:12.3476834Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12071
2022-05-18T04:14:12.3580767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12072
2022-05-18T04:14:13.4884360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:14:13.4935152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:14:13.4936474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:13.4986147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:13.4991496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:13.5946609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:13.6157285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:14:13.6158720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:14:13.6159530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:14:13.6160212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:14:13.6161592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:14:13.6262324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:14:13.6263807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:14:13.6264507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:14:14.9213651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3arccog4
2022-05-18T04:14:14.9214765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3arccog4/_remote_module_non_scriptable.py
2022-05-18T04:14:14.9402474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28e0bymr
2022-05-18T04:14:14.9405217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28e0bymr/_remote_module_non_scriptable.py
2022-05-18T04:14:25.6334304Z [W ProcessGroupNCCL.cpp:865] [Rank 1] Found key in store: NCCLABORTEDCOMM:20a8fac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-05-18T04:14:25.9844532Z ok (15.265s)
2022-05-18T04:14:25.9844750Z 
2022-05-18T04:14:25.9845169Z ----------------------------------------------------------------------
2022-05-18T04:14:25.9845645Z Ran 1 test in 15.265s
2022-05-18T04:14:25.9845937Z 
2022-05-18T04:14:25.9846103Z OK
2022-05-18T04:14:25.9846355Z 
2022-05-18T04:14:25.9846569Z Generating XML reports...
2022-05-18T04:14:25.9887517Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041410.xml
2022-05-18T04:14:27.4004470Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:14:27.4019214Z 
2022-05-18T04:14:27.4019365Z Running tests...
2022-05-18T04:14:27.4020327Z ----------------------------------------------------------------------
2022-05-18T04:14:28.9700517Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:29.0093946Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12195
2022-05-18T04:14:29.0196917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12196
2022-05-18T04:14:30.1433742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:14:30.1959528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:14:30.1960367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:30.2041990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:30.2048762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:30.2974530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:31.4261720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkecqblgp
2022-05-18T04:14:31.4262830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkecqblgp/_remote_module_non_scriptable.py
2022-05-18T04:14:31.5487919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadbdsw58
2022-05-18T04:14:31.5489119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadbdsw58/_remote_module_non_scriptable.py
2022-05-18T04:14:32.7293508Z ok (5.327s)
2022-05-18T04:14:32.7293723Z 
2022-05-18T04:14:32.7294120Z ----------------------------------------------------------------------
2022-05-18T04:14:32.7294444Z Ran 1 test in 5.327s
2022-05-18T04:14:32.7294610Z 
2022-05-18T04:14:32.7294710Z OK
2022-05-18T04:14:32.7294845Z 
2022-05-18T04:14:32.7294979Z Generating XML reports...
2022-05-18T04:14:32.7337668Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041427.xml
2022-05-18T04:14:34.1615024Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:14:34.1629479Z 
2022-05-18T04:14:34.1629985Z Running tests...
2022-05-18T04:14:34.1630496Z ----------------------------------------------------------------------
2022-05-18T04:14:35.7390367Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:35.7786632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12315
2022-05-18T04:14:35.7891003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12316
2022-05-18T04:14:36.9081664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:14:36.9690914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:14:36.9691709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:36.9790448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:36.9796358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:37.0705354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:38.1961491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfguu8uqn
2022-05-18T04:14:38.1962408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfguu8uqn/_remote_module_non_scriptable.py
2022-05-18T04:14:38.3156312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7p_ujy__
2022-05-18T04:14:38.3157975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7p_ujy__/_remote_module_non_scriptable.py
2022-05-18T04:14:39.4984284Z ok (5.335s)
2022-05-18T04:14:39.4984493Z 
2022-05-18T04:14:39.4984889Z ----------------------------------------------------------------------
2022-05-18T04:14:39.4985217Z Ran 1 test in 5.335s
2022-05-18T04:14:39.4985383Z 
2022-05-18T04:14:39.4985487Z OK
2022-05-18T04:14:39.4985623Z 
2022-05-18T04:14:39.4985759Z Generating XML reports...
2022-05-18T04:14:39.5029211Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041434.xml
2022-05-18T04:14:40.9027276Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:14:40.9041510Z 
2022-05-18T04:14:40.9041754Z Running tests...
2022-05-18T04:14:40.9042373Z ----------------------------------------------------------------------
2022-05-18T04:14:42.4330393Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:42.4713707Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12435
2022-05-18T04:14:42.4816657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12436
2022-05-18T04:14:43.6697294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:14:43.6697863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:14:43.6698672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:43.6699351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:43.6704796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:43.6705521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:44.9495342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu1x2wh33
2022-05-18T04:14:44.9495995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu1x2wh33/_remote_module_non_scriptable.py
2022-05-18T04:14:44.9612035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74bm4feq
2022-05-18T04:14:44.9614810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74bm4feq/_remote_module_non_scriptable.py
2022-05-18T04:14:45.5897352Z ok (4.685s)
2022-05-18T04:14:45.5897579Z 
2022-05-18T04:14:45.5897994Z ----------------------------------------------------------------------
2022-05-18T04:14:45.5898341Z Ran 1 test in 4.686s
2022-05-18T04:14:45.5898510Z 
2022-05-18T04:14:45.5898615Z OK
2022-05-18T04:14:45.5898750Z 
2022-05-18T04:14:45.5898887Z Generating XML reports...
2022-05-18T04:14:45.5939687Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041440.xml
2022-05-18T04:14:47.0073105Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:14:47.0087832Z 
2022-05-18T04:14:47.0087963Z Running tests...
2022-05-18T04:14:47.0088819Z ----------------------------------------------------------------------
2022-05-18T04:14:48.5861709Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:48.6244848Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12551
2022-05-18T04:14:48.6346575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12552
2022-05-18T04:14:49.7943924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:14:49.8131975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:14:49.8133036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:49.8146442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:49.8152641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:49.9146353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:51.0299253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3ccisyq
2022-05-18T04:14:51.0299888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3ccisyq/_remote_module_non_scriptable.py
2022-05-18T04:14:51.1636625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppcm08msj
2022-05-18T04:14:51.1637780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppcm08msj/_remote_module_non_scriptable.py
2022-05-18T04:14:51.4052304Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:14:51.4057300Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:14:51.4218684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:51.4219216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:51.7428781Z ok (4.734s)
2022-05-18T04:14:51.7429056Z 
2022-05-18T04:14:51.7429443Z ----------------------------------------------------------------------
2022-05-18T04:14:51.7429788Z Ran 1 test in 4.734s
2022-05-18T04:14:51.7429958Z 
2022-05-18T04:14:51.7430055Z OK
2022-05-18T04:14:51.7430190Z 
2022-05-18T04:14:51.7430326Z Generating XML reports...
2022-05-18T04:14:51.7470785Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041447.xml
2022-05-18T04:14:53.1422042Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:14:53.1436139Z 
2022-05-18T04:14:53.1436535Z Running tests...
2022-05-18T04:14:53.1437027Z ----------------------------------------------------------------------
2022-05-18T04:14:54.6789324Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:54.7183145Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12671
2022-05-18T04:14:54.7290900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12672
2022-05-18T04:14:55.8633756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:14:55.8699674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:14:55.8700546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:55.8735210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:55.8741694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:55.9714000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:57.1069445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfql9eyyc
2022-05-18T04:14:57.1070056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfql9eyyc/_remote_module_non_scriptable.py
2022-05-18T04:14:57.2222531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppebnnuoj
2022-05-18T04:14:57.2223857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppebnnuoj/_remote_module_non_scriptable.py
2022-05-18T04:14:57.2981096Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:14:57.2981881Z   warnings.warn(
2022-05-18T04:14:57.2982931Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:14:57.2983659Z   warnings.warn(
2022-05-18T04:14:57.8374818Z ok (4.693s)
2022-05-18T04:14:57.8375012Z 
2022-05-18T04:14:57.8375612Z ----------------------------------------------------------------------
2022-05-18T04:14:57.8375945Z Ran 1 test in 4.694s
2022-05-18T04:14:57.8376115Z 
2022-05-18T04:14:57.8376211Z OK
2022-05-18T04:14:57.8376348Z 
2022-05-18T04:14:57.8376480Z Generating XML reports...
2022-05-18T04:14:57.8419834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041453.xml
2022-05-18T04:14:59.2680394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:14:59.2695507Z 
2022-05-18T04:14:59.2695848Z Running tests...
2022-05-18T04:14:59.2696323Z ----------------------------------------------------------------------
2022-05-18T04:15:00.8385292Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:00.8535711Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.584s)
2022-05-18T04:15:00.8536302Z 
2022-05-18T04:15:00.8536579Z ----------------------------------------------------------------------
2022-05-18T04:15:00.8536909Z Ran 1 test in 1.584s
2022-05-18T04:15:00.8537074Z 
2022-05-18T04:15:00.8537184Z OK (skipped=1)
2022-05-18T04:15:00.8537338Z 
2022-05-18T04:15:00.8537464Z Generating XML reports...
2022-05-18T04:15:00.8568943Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041459.xml
2022-05-18T04:15:02.2350508Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:15:02.2365996Z 
2022-05-18T04:15:02.2366262Z Running tests...
2022-05-18T04:15:02.2366698Z ----------------------------------------------------------------------
2022-05-18T04:15:03.8239138Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:03.8625271Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12825
2022-05-18T04:15:03.8726386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12826
2022-05-18T04:15:05.0388508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:15:05.0570308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:15:05.0571159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:05.0591213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:05.0597448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:05.1585102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:06.3158987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplq5c15oj
2022-05-18T04:15:06.3159918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplq5c15oj/_remote_module_non_scriptable.py
2022-05-18T04:15:06.3988561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjgdk8m5
2022-05-18T04:15:06.3990542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjgdk8m5/_remote_module_non_scriptable.py
2022-05-18T04:15:08.1299769Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:08.1300315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:08.6973796Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:15:08.6987801Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:15:09.8862717Z ok (7.649s)
2022-05-18T04:15:09.8862944Z 
2022-05-18T04:15:09.8863341Z ----------------------------------------------------------------------
2022-05-18T04:15:09.8863716Z Ran 1 test in 7.650s
2022-05-18T04:15:09.8863887Z 
2022-05-18T04:15:09.8863984Z OK
2022-05-18T04:15:09.8864120Z 
2022-05-18T04:15:09.8864237Z Generating XML reports...
2022-05-18T04:15:09.8905350Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041502.xml
2022-05-18T04:15:11.3261779Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:15:11.3276414Z 
2022-05-18T04:15:11.3276827Z Running tests...
2022-05-18T04:15:11.3277309Z ----------------------------------------------------------------------
2022-05-18T04:15:12.9222178Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:12.9615348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12949
2022-05-18T04:15:12.9717065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12950
2022-05-18T04:15:14.0807447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:15:14.1416712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:15:14.1417744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:14.1516536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:14.1522814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:14.2431139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:15.4097834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx70_4kph
2022-05-18T04:15:15.4098761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx70_4kph/_remote_module_non_scriptable.py
2022-05-18T04:15:15.4872118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgsxr6mp
2022-05-18T04:15:15.4873460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgsxr6mp/_remote_module_non_scriptable.py
2022-05-18T04:15:15.8794582Z ok (4.551s)
2022-05-18T04:15:15.8794893Z 
2022-05-18T04:15:15.8795573Z ----------------------------------------------------------------------
2022-05-18T04:15:15.8796319Z Ran 1 test in 4.552s
2022-05-18T04:15:15.8796665Z 
2022-05-18T04:15:15.8796834Z OK
2022-05-18T04:15:15.8796978Z 
2022-05-18T04:15:15.8797092Z Generating XML reports...
2022-05-18T04:15:15.8837616Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041511.xml
2022-05-18T04:15:17.2747490Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:15:17.2762074Z 
2022-05-18T04:15:17.2762611Z Running tests...
2022-05-18T04:15:17.2763119Z ----------------------------------------------------------------------
2022-05-18T04:15:18.8360154Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:18.8753877Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13065
2022-05-18T04:15:18.8856820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13066
2022-05-18T04:15:20.0194419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:15:20.0298060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:15:20.0298858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:20.0396956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:20.0403033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:20.1312486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:21.2819665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfr0gde01
2022-05-18T04:15:21.2820310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfr0gde01/_remote_module_non_scriptable.py
2022-05-18T04:15:21.3924009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4wjjx_o
2022-05-18T04:15:21.3925096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4wjjx_o/_remote_module_non_scriptable.py
2022-05-18T04:15:21.4738043Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:15:21.4738858Z   warnings.warn(
2022-05-18T04:15:21.4740214Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:15:21.4740954Z   warnings.warn(
2022-05-18T04:15:21.6632577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:21.6633132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:21.6687337Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:15:21.6688924Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:15:21.6793716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:21.6794507Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:21.6856978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:21.6857762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:21.9937200Z ok (4.717s)
2022-05-18T04:15:21.9937426Z 
2022-05-18T04:15:21.9937830Z ----------------------------------------------------------------------
2022-05-18T04:15:21.9938157Z Ran 1 test in 4.717s
2022-05-18T04:15:21.9938326Z 
2022-05-18T04:15:21.9939725Z OK
2022-05-18T04:15:21.9940788Z 
2022-05-18T04:15:21.9941200Z Generating XML reports...
2022-05-18T04:15:21.9979193Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041517.xml
2022-05-18T04:15:23.4177331Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:15:23.4191968Z 
2022-05-18T04:15:23.4192121Z Running tests...
2022-05-18T04:15:23.4192891Z ----------------------------------------------------------------------
2022-05-18T04:15:25.0076514Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:25.0472943Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13185
2022-05-18T04:15:25.0579075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13186
2022-05-18T04:15:26.1972625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:15:26.2224999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:15:26.2226057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:26.2276769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:26.2282846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:26.3239392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:27.4793841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxglipj3
2022-05-18T04:15:27.4794499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxglipj3/_remote_module_non_scriptable.py
2022-05-18T04:15:27.5676155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84hzr_8n
2022-05-18T04:15:27.5677215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84hzr_8n/_remote_module_non_scriptable.py
2022-05-18T04:15:27.6385563Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:15:27.6386381Z   warnings.warn(
2022-05-18T04:15:27.6387450Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:15:27.6388164Z   warnings.warn(
2022-05-18T04:15:27.8264923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:27.8265465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:28.1660135Z ok (4.746s)
2022-05-18T04:15:28.1660458Z 
2022-05-18T04:15:28.1660848Z ----------------------------------------------------------------------
2022-05-18T04:15:28.1661192Z Ran 1 test in 4.747s
2022-05-18T04:15:28.1661357Z 
2022-05-18T04:15:28.1661451Z OK
2022-05-18T04:15:28.1661588Z 
2022-05-18T04:15:28.1661722Z Generating XML reports...
2022-05-18T04:15:28.1702768Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041523.xml
2022-05-18T04:15:29.5936057Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:15:29.5950613Z 
2022-05-18T04:15:29.5950924Z Running tests...
2022-05-18T04:15:29.5951364Z ----------------------------------------------------------------------
2022-05-18T04:15:31.1786315Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:31.1942080Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.599s)
2022-05-18T04:15:31.1942670Z 
2022-05-18T04:15:31.1942967Z ----------------------------------------------------------------------
2022-05-18T04:15:31.1943287Z Ran 1 test in 1.599s
2022-05-18T04:15:31.1943451Z 
2022-05-18T04:15:31.1943560Z OK (skipped=1)
2022-05-18T04:15:31.1943715Z 
2022-05-18T04:15:31.1943840Z Generating XML reports...
2022-05-18T04:15:31.1976636Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041529.xml
2022-05-18T04:15:32.5856001Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:15:32.5870638Z 
2022-05-18T04:15:32.5870904Z Running tests...
2022-05-18T04:15:32.5871352Z ----------------------------------------------------------------------
2022-05-18T04:15:34.1869527Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:34.2266990Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13339
2022-05-18T04:15:34.2371060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13340
2022-05-18T04:15:35.3884603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:15:35.4219823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:15:35.4220647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:35.4291841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:35.4297876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:35.5235212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:36.7017692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprc9kta1s
2022-05-18T04:15:36.7018311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprc9kta1s/_remote_module_non_scriptable.py
2022-05-18T04:15:36.7861199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph61ffue6
2022-05-18T04:15:36.7863349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph61ffue6/_remote_module_non_scriptable.py
2022-05-18T04:15:37.1595405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:37.1595947Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:38.2489685Z ok (5.661s)
2022-05-18T04:15:38.2489906Z 
2022-05-18T04:15:38.2490305Z ----------------------------------------------------------------------
2022-05-18T04:15:38.2490927Z Ran 1 test in 5.662s
2022-05-18T04:15:38.2491084Z 
2022-05-18T04:15:38.2491186Z OK
2022-05-18T04:15:38.2491321Z 
2022-05-18T04:15:38.2491454Z Generating XML reports...
2022-05-18T04:15:38.2533021Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041532.xml
2022-05-18T04:15:39.6948233Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:15:39.6962898Z 
2022-05-18T04:15:39.6963208Z Running tests...
2022-05-18T04:15:39.6963923Z ----------------------------------------------------------------------
2022-05-18T04:15:41.2837512Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:41.3235010Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13463
2022-05-18T04:15:41.3339003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13464
2022-05-18T04:15:42.5307037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:15:42.5380984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:15:42.5381827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:42.5408121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:42.5414343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:42.6396237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:43.7889981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1uh0emo
2022-05-18T04:15:43.7890602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1uh0emo/_remote_module_non_scriptable.py
2022-05-18T04:15:43.9213297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4u7ybafu
2022-05-18T04:15:43.9216062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4u7ybafu/_remote_module_non_scriptable.py
2022-05-18T04:15:44.3419991Z ok (4.645s)
2022-05-18T04:15:44.3420330Z 
2022-05-18T04:15:44.3421411Z ----------------------------------------------------------------------
2022-05-18T04:15:44.3422077Z Ran 1 test in 4.646s
2022-05-18T04:15:44.3422398Z 
2022-05-18T04:15:44.3422559Z OK
2022-05-18T04:15:44.3422800Z 
2022-05-18T04:15:44.3423041Z Generating XML reports...
2022-05-18T04:15:44.3465713Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041539.xml
2022-05-18T04:15:45.7577948Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:15:45.7591660Z 
2022-05-18T04:15:45.7592118Z Running tests...
2022-05-18T04:15:45.7592633Z ----------------------------------------------------------------------
2022-05-18T04:15:47.3071269Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:47.3461988Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13579
2022-05-18T04:15:47.3564389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13580
2022-05-18T04:15:48.5032743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:15:48.5033301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:15:48.5034110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:48.5034838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:48.5041177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:48.5041819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:49.7962074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8fzv_ga
2022-05-18T04:15:49.7962665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8fzv_ga/_remote_module_non_scriptable.py
2022-05-18T04:15:49.8098747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1x4aevc
2022-05-18T04:15:49.8101483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1x4aevc/_remote_module_non_scriptable.py
2022-05-18T04:15:50.1640544Z ok (4.404s)
2022-05-18T04:15:50.1640913Z 
2022-05-18T04:15:50.1641570Z ----------------------------------------------------------------------
2022-05-18T04:15:50.1642179Z Ran 1 test in 4.405s
2022-05-18T04:15:50.1642470Z 
2022-05-18T04:15:50.1642661Z OK
2022-05-18T04:15:50.1642918Z 
2022-05-18T04:15:50.1643151Z Generating XML reports...
2022-05-18T04:15:50.1684927Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041545.xml
2022-05-18T04:15:51.5912647Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:15:51.5927557Z 
2022-05-18T04:15:51.5928099Z Running tests...
2022-05-18T04:15:51.5928603Z ----------------------------------------------------------------------
2022-05-18T04:15:53.1860439Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:53.2257196Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13695
2022-05-18T04:15:53.2360889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13696
2022-05-18T04:15:54.3423212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:15:54.3698358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:15:54.3699168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:54.3727310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:15:54.3732846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:54.4713507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:55.6176627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9mh_bax
2022-05-18T04:15:55.6177246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9mh_bax/_remote_module_non_scriptable.py
2022-05-18T04:15:55.7048180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe55gclho
2022-05-18T04:15:55.7048805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe55gclho/_remote_module_non_scriptable.py
2022-05-18T04:15:55.9552473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:55.9553048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:15:56.2440066Z ok (4.651s)
2022-05-18T04:15:56.2440278Z 
2022-05-18T04:15:56.2440677Z ----------------------------------------------------------------------
2022-05-18T04:15:56.2441026Z Ran 1 test in 4.651s
2022-05-18T04:15:56.2441192Z 
2022-05-18T04:15:56.2441271Z OK
2022-05-18T04:15:56.2441410Z 
2022-05-18T04:15:56.2441541Z Generating XML reports...
2022-05-18T04:15:56.2481545Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041551.xml
2022-05-18T04:15:57.6693672Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:15:57.6708517Z 
2022-05-18T04:15:57.6708677Z Running tests...
2022-05-18T04:15:57.6709458Z ----------------------------------------------------------------------
2022-05-18T04:15:59.2589431Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:59.2738653Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.603s)
2022-05-18T04:15:59.2739329Z 
2022-05-18T04:15:59.2739613Z ----------------------------------------------------------------------
2022-05-18T04:15:59.2739955Z Ran 1 test in 1.603s
2022-05-18T04:15:59.2740118Z 
2022-05-18T04:15:59.2740227Z OK (skipped=1)
2022-05-18T04:15:59.2740382Z 
2022-05-18T04:15:59.2740493Z Generating XML reports...
2022-05-18T04:15:59.2772036Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041557.xml
2022-05-18T04:16:00.6792383Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:00.6808413Z 
2022-05-18T04:16:00.6808574Z Running tests...
2022-05-18T04:16:00.6809411Z ----------------------------------------------------------------------
2022-05-18T04:16:02.2687055Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:02.3079922Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13849
2022-05-18T04:16:02.3184873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13850
2022-05-18T04:16:03.4468842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:16:03.4544530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:16:03.4545663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:03.4570306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:03.4577344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:03.5560193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:04.6960912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjrupr0b
2022-05-18T04:16:04.6961534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjrupr0b/_remote_module_non_scriptable.py
2022-05-18T04:16:04.8305264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpimh6aowf
2022-05-18T04:16:04.8306703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpimh6aowf/_remote_module_non_scriptable.py
2022-05-18T04:16:05.0842085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:16:05.0842608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:16:05.1065582Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:16:05.1066078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:16:05.1161839Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpimh6aowf'>
2022-05-18T04:16:05.1162305Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:16:05.4266670Z ok (4.745s)
2022-05-18T04:16:05.4266891Z 
2022-05-18T04:16:05.4267274Z ----------------------------------------------------------------------
2022-05-18T04:16:05.4267618Z Ran 1 test in 4.746s
2022-05-18T04:16:05.4268028Z 
2022-05-18T04:16:05.4268131Z OK
2022-05-18T04:16:05.4268275Z 
2022-05-18T04:16:05.4268409Z Generating XML reports...
2022-05-18T04:16:05.4308968Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041600.xml
2022-05-18T04:16:06.8642660Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:06.8658210Z 
2022-05-18T04:16:06.8658654Z Running tests...
2022-05-18T04:16:06.8659147Z ----------------------------------------------------------------------
2022-05-18T04:16:08.4381296Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:08.4774902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13969
2022-05-18T04:16:08.4877335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13970
2022-05-18T04:16:09.6426152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:16:09.6697582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:16:09.6698626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:09.6729496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:09.6735718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:09.7712185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:10.9431179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbp6ahvm0
2022-05-18T04:16:10.9431801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbp6ahvm0/_remote_module_non_scriptable.py
2022-05-18T04:16:11.0185992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ek69sxm
2022-05-18T04:16:11.0187289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ek69sxm/_remote_module_non_scriptable.py
2022-05-18T04:16:11.5961120Z ok (4.730s)
2022-05-18T04:16:11.5961719Z 
2022-05-18T04:16:11.5962378Z ----------------------------------------------------------------------
2022-05-18T04:16:11.5962784Z Ran 1 test in 4.730s
2022-05-18T04:16:11.5962944Z 
2022-05-18T04:16:11.5963041Z OK
2022-05-18T04:16:11.5963183Z 
2022-05-18T04:16:11.5963321Z Generating XML reports...
2022-05-18T04:16:11.6003794Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041606.xml
2022-05-18T04:16:13.0132353Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:13.0146764Z 
2022-05-18T04:16:13.0147183Z Running tests...
2022-05-18T04:16:13.0147677Z ----------------------------------------------------------------------
2022-05-18T04:16:14.5877327Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:14.6270004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14089
2022-05-18T04:16:14.6373204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14090
2022-05-18T04:16:15.7732159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:16:15.7817680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:16:15.7818494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:15.7833187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:15.7839196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:15.7842882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:16:15.8828817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:15.8831994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:16:15.8832701Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:16:15.8861699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:16:16.1424247Z ok (3.127s)
2022-05-18T04:16:16.1424469Z 
2022-05-18T04:16:16.1424842Z ----------------------------------------------------------------------
2022-05-18T04:16:16.1425185Z Ran 1 test in 3.128s
2022-05-18T04:16:16.1425352Z 
2022-05-18T04:16:16.1426671Z OK
2022-05-18T04:16:16.1426862Z 
2022-05-18T04:16:16.1427284Z Generating XML reports...
2022-05-18T04:16:16.1467253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041613.xml
2022-05-18T04:16:17.5382843Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:17.5396069Z 
2022-05-18T04:16:17.5396319Z Running tests...
2022-05-18T04:16:17.5396765Z ----------------------------------------------------------------------
2022-05-18T04:16:19.1008986Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:19.1396438Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14200
2022-05-18T04:16:19.1500450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14201
2022-05-18T04:16:20.2727062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:16:20.3270968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:16:20.3271801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:20.3334726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:20.3340946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:20.3344463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:16:20.4281891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:20.4286894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:16:20.4287685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:16:20.4361525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:16:20.7558457Z ok (3.216s)
2022-05-18T04:16:20.7559046Z 
2022-05-18T04:16:20.7559445Z ----------------------------------------------------------------------
2022-05-18T04:16:20.7559813Z Ran 1 test in 3.216s
2022-05-18T04:16:20.7559973Z 
2022-05-18T04:16:20.7560074Z OK
2022-05-18T04:16:20.7560213Z 
2022-05-18T04:16:20.7560348Z Generating XML reports...
2022-05-18T04:16:20.7601748Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041617.xml
2022-05-18T04:16:22.1588170Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:22.1601916Z 
2022-05-18T04:16:22.1602222Z Running tests...
2022-05-18T04:16:22.1602657Z ----------------------------------------------------------------------
2022-05-18T04:16:23.7080759Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:23.7470507Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14311
2022-05-18T04:16:23.7572306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14312
2022-05-18T04:16:24.9010103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:16:24.9503854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:16:24.9504663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:24.9516294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:24.9522806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:25.0517817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:26.2241900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprho9nddh
2022-05-18T04:16:26.2242545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprho9nddh/_remote_module_non_scriptable.py
2022-05-18T04:16:26.2939349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcqorruov
2022-05-18T04:16:26.2940522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcqorruov/_remote_module_non_scriptable.py
2022-05-18T04:16:26.5620692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:16:26.5621224Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:16:26.5682559Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:16:26.5684171Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:16:26.8656000Z ok (4.705s)
2022-05-18T04:16:26.8656327Z 
2022-05-18T04:16:26.8657142Z ----------------------------------------------------------------------
2022-05-18T04:16:26.8657810Z Ran 1 test in 4.705s
2022-05-18T04:16:26.8658130Z 
2022-05-18T04:16:26.8658283Z OK
2022-05-18T04:16:26.8658526Z 
2022-05-18T04:16:26.8658766Z Generating XML reports...
2022-05-18T04:16:26.8700625Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041622.xml
2022-05-18T04:16:28.2785527Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:28.2799704Z 
2022-05-18T04:16:28.2799957Z Running tests...
2022-05-18T04:16:28.2800408Z ----------------------------------------------------------------------
2022-05-18T04:16:29.8385842Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:29.8780653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14431
2022-05-18T04:16:29.8886304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14432
2022-05-18T04:16:31.0374718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:16:31.0491357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:16:31.0492181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:31.0578392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:31.0584638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:31.1505390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:32.3077223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuq3oexoj
2022-05-18T04:16:32.3077827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuq3oexoj/_remote_module_non_scriptable.py
2022-05-18T04:16:32.3973641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_feyd4r8
2022-05-18T04:16:32.3974524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_feyd4r8/_remote_module_non_scriptable.py
2022-05-18T04:16:32.6283453Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:16:32.6539084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:16:32.6539620Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:16:32.9969597Z ok (4.717s)
2022-05-18T04:16:32.9969827Z 
2022-05-18T04:16:32.9970234Z ----------------------------------------------------------------------
2022-05-18T04:16:32.9970831Z Ran 1 test in 4.717s
2022-05-18T04:16:32.9971258Z 
2022-05-18T04:16:32.9971370Z OK
2022-05-18T04:16:32.9971503Z 
2022-05-18T04:16:32.9971647Z Generating XML reports...
2022-05-18T04:16:33.0012236Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041628.xml
2022-05-18T04:16:34.4183292Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:34.4197155Z 
2022-05-18T04:16:34.4197582Z Running tests...
2022-05-18T04:16:34.4198391Z ----------------------------------------------------------------------
2022-05-18T04:16:36.0024384Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:36.0418826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14551
2022-05-18T04:16:36.0524632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14552
2022-05-18T04:16:37.1842943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:16:37.1926240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:16:37.1927042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:37.1944215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:37.1950472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:37.2936447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:37.4575539Z ok (3.037s)
2022-05-18T04:16:37.4575753Z 
2022-05-18T04:16:37.4576563Z ----------------------------------------------------------------------
2022-05-18T04:16:37.4576946Z Ran 1 test in 3.038s
2022-05-18T04:16:37.4577119Z 
2022-05-18T04:16:37.4577223Z OK
2022-05-18T04:16:37.4577348Z 
2022-05-18T04:16:37.4577506Z Generating XML reports...
2022-05-18T04:16:37.4617810Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041634.xml
2022-05-18T04:16:38.8559299Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:38.8574693Z 
2022-05-18T04:16:38.8575274Z Running tests...
2022-05-18T04:16:38.8575902Z ----------------------------------------------------------------------
2022-05-18T04:16:38.8597952Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:16:38.8598744Z 
2022-05-18T04:16:38.8599057Z ----------------------------------------------------------------------
2022-05-18T04:16:38.8599404Z Ran 1 test in 0.002s
2022-05-18T04:16:38.8599574Z 
2022-05-18T04:16:38.8599690Z OK (skipped=1)
2022-05-18T04:16:38.8599829Z 
2022-05-18T04:16:38.8599958Z Generating XML reports...
2022-05-18T04:16:38.8643012Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041638.xml
2022-05-18T04:16:40.0822150Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:40.0836436Z 
2022-05-18T04:16:40.0836712Z Running tests...
2022-05-18T04:16:40.0837164Z ----------------------------------------------------------------------
2022-05-18T04:16:40.0863221Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:16:40.0863520Z 
2022-05-18T04:16:40.0863814Z ----------------------------------------------------------------------
2022-05-18T04:16:40.0864179Z Ran 1 test in 0.003s
2022-05-18T04:16:40.0864350Z 
2022-05-18T04:16:40.0864465Z OK (skipped=1)
2022-05-18T04:16:40.0864633Z 
2022-05-18T04:16:40.0864742Z Generating XML reports...
2022-05-18T04:16:40.0901131Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041640.xml
2022-05-18T04:16:41.3417758Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:41.3431798Z 
2022-05-18T04:16:41.3432186Z Running tests...
2022-05-18T04:16:41.3432697Z ----------------------------------------------------------------------
2022-05-18T04:16:42.9174130Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:42.9567646Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14724
2022-05-18T04:16:42.9669952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14725
2022-05-18T04:16:44.0750680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:16:44.1108544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:16:44.1109376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:44.1155859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:44.1162831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:44.2123233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:46.1753402Z ok (4.832s)
2022-05-18T04:16:46.1753671Z 
2022-05-18T04:16:46.1754087Z ----------------------------------------------------------------------
2022-05-18T04:16:46.1754443Z Ran 1 test in 4.832s
2022-05-18T04:16:46.1754948Z 
2022-05-18T04:16:46.1755049Z OK
2022-05-18T04:16:46.1755173Z 
2022-05-18T04:16:46.1755323Z Generating XML reports...
2022-05-18T04:16:46.1795129Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041641.xml
2022-05-18T04:16:47.5906250Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:47.5920353Z 
2022-05-18T04:16:47.5920873Z Running tests...
2022-05-18T04:16:47.5921391Z ----------------------------------------------------------------------
2022-05-18T04:16:47.5940182Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:16:47.5940775Z 
2022-05-18T04:16:47.5941331Z ----------------------------------------------------------------------
2022-05-18T04:16:47.5941692Z Ran 1 test in 0.002s
2022-05-18T04:16:47.5941862Z 
2022-05-18T04:16:47.5941974Z OK (skipped=1)
2022-05-18T04:16:47.5942135Z 
2022-05-18T04:16:47.5942984Z Generating XML reports...
2022-05-18T04:16:47.5977236Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041647.xml
2022-05-18T04:16:48.8569517Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:48.8584270Z 
2022-05-18T04:16:48.8584536Z Running tests...
2022-05-18T04:16:48.8585229Z ----------------------------------------------------------------------
2022-05-18T04:16:48.8604290Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:16:48.8604741Z 
2022-05-18T04:16:48.8605157Z ----------------------------------------------------------------------
2022-05-18T04:16:48.8605479Z Ran 1 test in 0.002s
2022-05-18T04:16:48.8605645Z 
2022-05-18T04:16:48.8605761Z OK (skipped=1)
2022-05-18T04:16:48.8605919Z 
2022-05-18T04:16:48.8606046Z Generating XML reports...
2022-05-18T04:16:48.8642010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041648.xml
2022-05-18T04:16:50.1200872Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:50.1215887Z 
2022-05-18T04:16:50.1216181Z Running tests...
2022-05-18T04:16:50.1216907Z ----------------------------------------------------------------------
2022-05-18T04:16:51.6956475Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:51.7350725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14910
2022-05-18T04:16:51.7454097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14911
2022-05-18T04:16:52.9012448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:16:52.9351654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:16:52.9352466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:52.9417507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:52.9423748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:53.0366693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:55.7567538Z ok (5.635s)
2022-05-18T04:16:55.7567759Z 
2022-05-18T04:16:55.7568129Z ----------------------------------------------------------------------
2022-05-18T04:16:55.7568480Z Ran 1 test in 5.635s
2022-05-18T04:16:55.7568765Z 
2022-05-18T04:16:55.7568940Z OK
2022-05-18T04:16:55.7569191Z 
2022-05-18T04:16:55.7569345Z Generating XML reports...
2022-05-18T04:16:55.7609484Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041650.xml
2022-05-18T04:16:57.1769867Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:16:57.1783972Z 
2022-05-18T04:16:57.1784392Z Running tests...
2022-05-18T04:16:57.1784866Z ----------------------------------------------------------------------
2022-05-18T04:16:58.7346888Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:58.7734102Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15028
2022-05-18T04:16:58.7835531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15029
2022-05-18T04:16:59.9139398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:16:59.9197071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:16:59.9197881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:59.9240886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:16:59.9247131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:00.0207764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:00.0418024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:17:00.0418551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:17:00.0419253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:17:00.0419933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:17:02.4574342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:17:02.4575050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:17:02.4576123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:17:02.4576900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:17:02.4983123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:17:02.4983904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:17:02.4984671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:17:02.4985363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:17:02.8935269Z ok (5.715s)
2022-05-18T04:17:02.8935488Z 
2022-05-18T04:17:02.8936187Z ----------------------------------------------------------------------
2022-05-18T04:17:02.8936927Z Ran 1 test in 5.715s
2022-05-18T04:17:02.8937168Z 
2022-05-18T04:17:02.8937283Z OK
2022-05-18T04:17:02.8937425Z 
2022-05-18T04:17:02.8937557Z Generating XML reports...
2022-05-18T04:17:02.8978479Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041657.xml
2022-05-18T04:17:04.3115533Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:04.3129643Z 
2022-05-18T04:17:04.3130023Z Running tests...
2022-05-18T04:17:04.3130474Z ----------------------------------------------------------------------
2022-05-18T04:17:05.8646389Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:05.9039104Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15185
2022-05-18T04:17:05.9143418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15186
2022-05-18T04:17:07.0437995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:17:07.0494260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:17:07.0495061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:07.0539740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:07.0546181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:07.0549344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:17:07.1505347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:07.1508871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:17:07.1509598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:17:07.1564205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:17:07.4197390Z ok (3.106s)
2022-05-18T04:17:07.4197585Z 
2022-05-18T04:17:07.4197994Z ----------------------------------------------------------------------
2022-05-18T04:17:07.4198585Z Ran 1 test in 3.107s
2022-05-18T04:17:07.4198755Z 
2022-05-18T04:17:07.4198858Z OK
2022-05-18T04:17:07.4198997Z 
2022-05-18T04:17:07.4199117Z Generating XML reports...
2022-05-18T04:17:07.4239203Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041704.xml
2022-05-18T04:17:08.8356220Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:08.8371063Z 
2022-05-18T04:17:08.8371530Z Running tests...
2022-05-18T04:17:08.8372365Z ----------------------------------------------------------------------
2022-05-18T04:17:10.4099208Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:10.4496073Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15296
2022-05-18T04:17:10.4602384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15297
2022-05-18T04:17:11.5810966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:17:11.5995361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:17:11.5996480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:11.6013478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:11.6019592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:11.7009558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:13.2679176Z ok (4.430s)
2022-05-18T04:17:13.2679411Z 
2022-05-18T04:17:13.2679792Z ----------------------------------------------------------------------
2022-05-18T04:17:13.2680147Z Ran 1 test in 4.431s
2022-05-18T04:17:13.2680314Z 
2022-05-18T04:17:13.2680409Z OK
2022-05-18T04:17:13.2680546Z 
2022-05-18T04:17:13.2680681Z Generating XML reports...
2022-05-18T04:17:13.2722476Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041708.xml
2022-05-18T04:17:14.7088107Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:14.7092718Z 
2022-05-18T04:17:14.7093088Z Running tests...
2022-05-18T04:17:14.7093588Z ----------------------------------------------------------------------
2022-05-18T04:17:16.2903415Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:16.3291243Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15412
2022-05-18T04:17:16.3395673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15413
2022-05-18T04:17:17.4757380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:17:17.4838915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:17:17.4839724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:17.4858448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:17.4864850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:17.5849717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:17.9448703Z ok (3.235s)
2022-05-18T04:17:17.9448923Z 
2022-05-18T04:17:17.9449320Z ----------------------------------------------------------------------
2022-05-18T04:17:17.9449650Z Ran 1 test in 3.236s
2022-05-18T04:17:17.9449817Z 
2022-05-18T04:17:17.9449920Z OK
2022-05-18T04:17:17.9450059Z 
2022-05-18T04:17:17.9450198Z Generating XML reports...
2022-05-18T04:17:17.9491076Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041714.xml
2022-05-18T04:17:19.3728467Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:19.3743663Z 
2022-05-18T04:17:19.3744090Z Running tests...
2022-05-18T04:17:19.3744598Z ----------------------------------------------------------------------
2022-05-18T04:17:20.9501529Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:20.9897497Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15519
2022-05-18T04:17:21.0000739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15520
2022-05-18T04:17:22.1346410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:17:22.1841849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:17:22.1842629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:22.1852485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:22.1859108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:22.1862079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:17:22.2852779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:22.2856212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:17:22.2856931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:17:22.2880912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:17:22.6059941Z ok (3.231s)
2022-05-18T04:17:22.6060140Z 
2022-05-18T04:17:22.6060525Z ----------------------------------------------------------------------
2022-05-18T04:17:22.6061150Z Ran 1 test in 3.232s
2022-05-18T04:17:22.6061330Z 
2022-05-18T04:17:22.6061426Z OK
2022-05-18T04:17:22.6061562Z 
2022-05-18T04:17:22.6061677Z Generating XML reports...
2022-05-18T04:17:22.6102895Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041719.xml
2022-05-18T04:17:24.0335622Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:24.0349921Z 
2022-05-18T04:17:24.0350357Z Running tests...
2022-05-18T04:17:24.0350852Z ----------------------------------------------------------------------
2022-05-18T04:17:25.6099636Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:25.6484080Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15630
2022-05-18T04:17:25.6587049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15631
2022-05-18T04:17:26.7853536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:17:26.8051479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:17:26.8052539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:26.8055667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:26.8061748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:26.8064698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:17:26.9064074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:26.9067733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:17:26.9068750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:17:26.9084368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:17:27.1639444Z ok (3.129s)
2022-05-18T04:17:27.1639664Z 
2022-05-18T04:17:27.1640058Z ----------------------------------------------------------------------
2022-05-18T04:17:27.1640651Z Ran 1 test in 3.129s
2022-05-18T04:17:27.1640835Z 
2022-05-18T04:17:27.1640931Z OK
2022-05-18T04:17:27.1641067Z 
2022-05-18T04:17:27.1641203Z Generating XML reports...
2022-05-18T04:17:27.1682992Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041724.xml
2022-05-18T04:17:28.5590906Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:28.5605637Z 
2022-05-18T04:17:28.5606314Z Running tests...
2022-05-18T04:17:28.5606831Z ----------------------------------------------------------------------
2022-05-18T04:17:30.1291551Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:30.1675690Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15741
2022-05-18T04:17:30.1780880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15742
2022-05-18T04:17:31.3164851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:17:31.3235322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:17:31.3236405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:31.3266337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:31.3272533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:31.4250946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:32.5640679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf4w5ujg1
2022-05-18T04:17:32.5641654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf4w5ujg1/_remote_module_non_scriptable.py
2022-05-18T04:17:32.6690912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3h7ybuuc
2022-05-18T04:17:32.6692116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3h7ybuuc/_remote_module_non_scriptable.py
2022-05-18T04:17:33.2863399Z ok (4.725s)
2022-05-18T04:17:33.2863633Z 
2022-05-18T04:17:33.2864046Z ----------------------------------------------------------------------
2022-05-18T04:17:33.2864372Z Ran 1 test in 4.726s
2022-05-18T04:17:33.2864558Z 
2022-05-18T04:17:33.2864653Z OK
2022-05-18T04:17:33.2864788Z 
2022-05-18T04:17:33.2864927Z Generating XML reports...
2022-05-18T04:17:33.2905431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041728.xml
2022-05-18T04:17:34.7110133Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:34.7125123Z 
2022-05-18T04:17:34.7125573Z Running tests...
2022-05-18T04:17:34.7126067Z ----------------------------------------------------------------------
2022-05-18T04:17:34.7151313Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2022-05-18T04:17:34.7151828Z 
2022-05-18T04:17:34.7152281Z ----------------------------------------------------------------------
2022-05-18T04:17:34.7152878Z Ran 1 test in 0.003s
2022-05-18T04:17:34.7153060Z 
2022-05-18T04:17:34.7153179Z OK (skipped=1)
2022-05-18T04:17:34.7153341Z 
2022-05-18T04:17:34.7153465Z Generating XML reports...
2022-05-18T04:17:34.7188563Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041734.xml
2022-05-18T04:17:35.9330274Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:35.9344892Z 
2022-05-18T04:17:35.9345335Z Running tests...
2022-05-18T04:17:35.9345838Z ----------------------------------------------------------------------
2022-05-18T04:17:35.9363953Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-05-18T04:17:35.9364245Z 
2022-05-18T04:17:35.9364518Z ----------------------------------------------------------------------
2022-05-18T04:17:35.9364861Z Ran 1 test in 0.002s
2022-05-18T04:17:35.9365027Z 
2022-05-18T04:17:35.9365140Z OK (skipped=1)
2022-05-18T04:17:35.9365302Z 
2022-05-18T04:17:35.9365431Z Generating XML reports...
2022-05-18T04:17:35.9402642Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041735.xml
2022-05-18T04:17:37.1918864Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:37.1933837Z 
2022-05-18T04:17:37.1934264Z Running tests...
2022-05-18T04:17:37.1934787Z ----------------------------------------------------------------------
2022-05-18T04:17:37.1952723Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-05-18T04:17:37.1953043Z 
2022-05-18T04:17:37.1953334Z ----------------------------------------------------------------------
2022-05-18T04:17:37.1953669Z Ran 1 test in 0.002s
2022-05-18T04:17:37.1953838Z 
2022-05-18T04:17:37.1953953Z OK (skipped=1)
2022-05-18T04:17:37.1954116Z 
2022-05-18T04:17:37.1954247Z Generating XML reports...
2022-05-18T04:17:37.1989914Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041737.xml
2022-05-18T04:17:38.4075592Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:38.4090143Z 
2022-05-18T04:17:38.4090569Z Running tests...
2022-05-18T04:17:38.4091074Z ----------------------------------------------------------------------
2022-05-18T04:17:38.4110300Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-05-18T04:17:38.4110615Z 
2022-05-18T04:17:38.4110909Z ----------------------------------------------------------------------
2022-05-18T04:17:38.4111237Z Ran 1 test in 0.002s
2022-05-18T04:17:38.4111406Z 
2022-05-18T04:17:38.4111521Z OK (skipped=1)
2022-05-18T04:17:38.4111685Z 
2022-05-18T04:17:38.4111814Z Generating XML reports...
2022-05-18T04:17:38.4147680Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041738.xml
2022-05-18T04:17:39.6640021Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:39.6655943Z 
2022-05-18T04:17:39.6656415Z Running tests...
2022-05-18T04:17:39.6656911Z ----------------------------------------------------------------------
2022-05-18T04:17:39.6677353Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T04:17:39.6677836Z 
2022-05-18T04:17:39.6678761Z ----------------------------------------------------------------------
2022-05-18T04:17:39.6679390Z Ran 1 test in 0.002s
2022-05-18T04:17:39.6679583Z 
2022-05-18T04:17:39.6679680Z OK (skipped=1)
2022-05-18T04:17:39.6679840Z 
2022-05-18T04:17:39.6679967Z Generating XML reports...
2022-05-18T04:17:39.6715570Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041739.xml
2022-05-18T04:17:40.9152964Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:40.9167890Z 
2022-05-18T04:17:40.9168188Z Running tests...
2022-05-18T04:17:40.9168643Z ----------------------------------------------------------------------
2022-05-18T04:17:40.9188152Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T04:17:40.9188841Z 
2022-05-18T04:17:40.9189126Z ----------------------------------------------------------------------
2022-05-18T04:17:40.9189468Z Ran 1 test in 0.002s
2022-05-18T04:17:40.9189634Z 
2022-05-18T04:17:40.9189753Z OK (skipped=1)
2022-05-18T04:17:40.9189911Z 
2022-05-18T04:17:40.9190039Z Generating XML reports...
2022-05-18T04:17:40.9225084Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041740.xml
2022-05-18T04:17:42.1795282Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:42.1810266Z 
2022-05-18T04:17:42.1810539Z Running tests...
2022-05-18T04:17:42.1810984Z ----------------------------------------------------------------------
2022-05-18T04:17:42.1834178Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:17:42.1834528Z 
2022-05-18T04:17:42.1834811Z ----------------------------------------------------------------------
2022-05-18T04:17:42.1835148Z Ran 1 test in 0.002s
2022-05-18T04:17:42.1835315Z 
2022-05-18T04:17:42.1835428Z OK (skipped=1)
2022-05-18T04:17:42.1835585Z 
2022-05-18T04:17:42.1835693Z Generating XML reports...
2022-05-18T04:17:42.1871201Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041742.xml
2022-05-18T04:17:43.4390601Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:43.4405552Z 
2022-05-18T04:17:43.4405910Z Running tests...
2022-05-18T04:17:43.4406612Z ----------------------------------------------------------------------
2022-05-18T04:17:43.4434027Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.003s)
2022-05-18T04:17:43.4434626Z 
2022-05-18T04:17:43.4434933Z ----------------------------------------------------------------------
2022-05-18T04:17:43.4435286Z Ran 1 test in 0.003s
2022-05-18T04:17:43.4435453Z 
2022-05-18T04:17:43.4435547Z OK (skipped=1)
2022-05-18T04:17:43.4435704Z 
2022-05-18T04:17:43.4435832Z Generating XML reports...
2022-05-18T04:17:43.4471232Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041743.xml
2022-05-18T04:17:44.6675470Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:44.6690619Z 
2022-05-18T04:17:44.6691053Z Running tests...
2022-05-18T04:17:44.6691530Z ----------------------------------------------------------------------
2022-05-18T04:17:44.6712137Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:17:44.6712689Z 
2022-05-18T04:17:44.6713064Z ----------------------------------------------------------------------
2022-05-18T04:17:44.6713404Z Ran 1 test in 0.002s
2022-05-18T04:17:44.6713587Z 
2022-05-18T04:17:44.6713681Z OK (skipped=1)
2022-05-18T04:17:44.6713840Z 
2022-05-18T04:17:44.6713967Z Generating XML reports...
2022-05-18T04:17:44.6749183Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041744.xml
2022-05-18T04:17:45.9269870Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:45.9285135Z 
2022-05-18T04:17:45.9285369Z Running tests...
2022-05-18T04:17:45.9285819Z ----------------------------------------------------------------------
2022-05-18T04:17:45.9307255Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:17:45.9307734Z 
2022-05-18T04:17:45.9308189Z ----------------------------------------------------------------------
2022-05-18T04:17:45.9308515Z Ran 1 test in 0.002s
2022-05-18T04:17:45.9308683Z 
2022-05-18T04:17:45.9308795Z OK (skipped=1)
2022-05-18T04:17:45.9309226Z 
2022-05-18T04:17:45.9309372Z Generating XML reports...
2022-05-18T04:17:45.9344388Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041745.xml
2022-05-18T04:17:47.1828238Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:47.1842901Z 
2022-05-18T04:17:47.1843202Z Running tests...
2022-05-18T04:17:47.1843626Z ----------------------------------------------------------------------
2022-05-18T04:17:47.1865049Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:17:47.1865429Z 
2022-05-18T04:17:47.1865710Z ----------------------------------------------------------------------
2022-05-18T04:17:47.1866059Z Ran 1 test in 0.002s
2022-05-18T04:17:47.1866209Z 
2022-05-18T04:17:47.1866328Z OK (skipped=1)
2022-05-18T04:17:47.1866489Z 
2022-05-18T04:17:47.1866619Z Generating XML reports...
2022-05-18T04:17:47.1901595Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041747.xml
2022-05-18T04:17:48.4394885Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:48.4409366Z 
2022-05-18T04:17:48.4409799Z Running tests...
2022-05-18T04:17:48.4410293Z ----------------------------------------------------------------------
2022-05-18T04:17:50.0189382Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:50.0580833Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16224
2022-05-18T04:17:50.0684149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16225
2022-05-18T04:17:51.1967682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:17:51.2264266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:17:51.2265068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:51.2271199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:51.2277394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:51.3279362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:52.8760327Z ok (4.435s)
2022-05-18T04:17:52.8760564Z 
2022-05-18T04:17:52.8760989Z ----------------------------------------------------------------------
2022-05-18T04:17:52.8761336Z Ran 1 test in 4.435s
2022-05-18T04:17:52.8761512Z 
2022-05-18T04:17:52.8761610Z OK
2022-05-18T04:17:52.8761730Z 
2022-05-18T04:17:52.8761869Z Generating XML reports...
2022-05-18T04:17:52.8802932Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041748.xml
2022-05-18T04:17:54.2715501Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:17:54.2729516Z 
2022-05-18T04:17:54.2729778Z Running tests...
2022-05-18T04:17:54.2730205Z ----------------------------------------------------------------------
2022-05-18T04:17:55.8030355Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:55.8414495Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16340
2022-05-18T04:17:55.8517800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16341
2022-05-18T04:17:57.0050918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:17:57.0349900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:17:57.0350933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:57.0354587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:17:57.0360565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:57.1364312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:58.7595860Z ok (4.486s)
2022-05-18T04:17:58.7596081Z 
2022-05-18T04:17:58.7596485Z ----------------------------------------------------------------------
2022-05-18T04:17:58.7596840Z Ran 1 test in 4.487s
2022-05-18T04:17:58.7597013Z 
2022-05-18T04:17:58.7597113Z OK
2022-05-18T04:17:58.7597259Z 
2022-05-18T04:17:58.7597397Z Generating XML reports...
2022-05-18T04:17:58.7638648Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041754.xml
2022-05-18T04:18:00.1772564Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:00.1787153Z 
2022-05-18T04:18:00.1787426Z Running tests...
2022-05-18T04:18:00.1787868Z ----------------------------------------------------------------------
2022-05-18T04:18:01.7666071Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:01.8065952Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16456
2022-05-18T04:18:01.8171858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16457
2022-05-18T04:18:02.9674614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:18:02.9873016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:18:02.9874557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:02.9876343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:02.9883733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:03.0889070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:04.6248362Z ok (4.446s)
2022-05-18T04:18:04.6248725Z 
2022-05-18T04:18:04.6249497Z ----------------------------------------------------------------------
2022-05-18T04:18:04.6250070Z Ran 1 test in 4.446s
2022-05-18T04:18:04.6250242Z 
2022-05-18T04:18:04.6250342Z OK
2022-05-18T04:18:04.6250477Z 
2022-05-18T04:18:04.6250611Z Generating XML reports...
2022-05-18T04:18:04.6291463Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041800.xml
2022-05-18T04:18:06.0456244Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:06.0469884Z 
2022-05-18T04:18:06.0470090Z Running tests...
2022-05-18T04:18:06.0471018Z ----------------------------------------------------------------------
2022-05-18T04:18:07.6181959Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:07.6568853Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16572
2022-05-18T04:18:07.6671150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16573
2022-05-18T04:18:08.7915782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:18:08.7977847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:18:08.7978648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:08.8017583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:08.8023899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:08.8991645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:10.5750769Z ok (4.528s)
2022-05-18T04:18:10.5751156Z 
2022-05-18T04:18:10.5751577Z ----------------------------------------------------------------------
2022-05-18T04:18:10.5751933Z Ran 1 test in 4.528s
2022-05-18T04:18:10.5752101Z 
2022-05-18T04:18:10.5752220Z OK
2022-05-18T04:18:10.5752359Z 
2022-05-18T04:18:10.5752478Z Generating XML reports...
2022-05-18T04:18:10.5793597Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041806.xml
2022-05-18T04:18:11.9979281Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:11.9993695Z 
2022-05-18T04:18:11.9994149Z Running tests...
2022-05-18T04:18:11.9994646Z ----------------------------------------------------------------------
2022-05-18T04:18:13.5782577Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:13.6176666Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16688
2022-05-18T04:18:13.6283612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16689
2022-05-18T04:18:14.7844141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:18:14.7932149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:18:14.7932974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:14.7946140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:14.7952432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:14.8943403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:18.0385824Z ok (6.039s)
2022-05-18T04:18:18.0386055Z 
2022-05-18T04:18:18.0386463Z ----------------------------------------------------------------------
2022-05-18T04:18:18.0386794Z Ran 1 test in 6.039s
2022-05-18T04:18:18.0386964Z 
2022-05-18T04:18:18.0387061Z OK
2022-05-18T04:18:18.0387204Z 
2022-05-18T04:18:18.0387338Z Generating XML reports...
2022-05-18T04:18:18.0427659Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041811.xml
2022-05-18T04:18:19.4634560Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:19.4649364Z 
2022-05-18T04:18:19.4649867Z Running tests...
2022-05-18T04:18:19.4650377Z ----------------------------------------------------------------------
2022-05-18T04:18:19.4672774Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:18:19.4673097Z 
2022-05-18T04:18:19.4673424Z ----------------------------------------------------------------------
2022-05-18T04:18:19.4673748Z Ran 1 test in 0.002s
2022-05-18T04:18:19.4673917Z 
2022-05-18T04:18:19.4674038Z OK (skipped=1)
2022-05-18T04:18:19.4674197Z 
2022-05-18T04:18:19.4674328Z Generating XML reports...
2022-05-18T04:18:19.4710056Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041819.xml
2022-05-18T04:18:20.7343797Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:20.7359157Z 
2022-05-18T04:18:20.7359770Z Running tests...
2022-05-18T04:18:20.7360627Z ----------------------------------------------------------------------
2022-05-18T04:18:20.7386469Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:18:20.7386797Z 
2022-05-18T04:18:20.7387095Z ----------------------------------------------------------------------
2022-05-18T04:18:20.7387434Z Ran 1 test in 0.003s
2022-05-18T04:18:20.7387602Z 
2022-05-18T04:18:20.7387703Z OK (skipped=1)
2022-05-18T04:18:20.7387862Z 
2022-05-18T04:18:20.7387988Z Generating XML reports...
2022-05-18T04:18:20.7424202Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041820.xml
2022-05-18T04:18:21.9735371Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:21.9750005Z 
2022-05-18T04:18:21.9750482Z Running tests...
2022-05-18T04:18:21.9750973Z ----------------------------------------------------------------------
2022-05-18T04:18:21.9777613Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:18:21.9777980Z 
2022-05-18T04:18:21.9778275Z ----------------------------------------------------------------------
2022-05-18T04:18:21.9778609Z Ran 1 test in 0.003s
2022-05-18T04:18:21.9778774Z 
2022-05-18T04:18:21.9778884Z OK (skipped=1)
2022-05-18T04:18:21.9779025Z 
2022-05-18T04:18:21.9779157Z Generating XML reports...
2022-05-18T04:18:21.9820599Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041821.xml
2022-05-18T04:18:23.1870932Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:23.1886020Z 
2022-05-18T04:18:23.1886344Z Running tests...
2022-05-18T04:18:23.1886799Z ----------------------------------------------------------------------
2022-05-18T04:18:24.7666097Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:24.8058234Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16914
2022-05-18T04:18:24.8162120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16915
2022-05-18T04:18:25.9503419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:18:25.9687117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:18:25.9687925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:25.9706251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:25.9712661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:26.0698082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:26.2211761Z ok (3.032s)
2022-05-18T04:18:26.2211999Z 
2022-05-18T04:18:26.2212385Z ----------------------------------------------------------------------
2022-05-18T04:18:26.2212728Z Ran 1 test in 3.033s
2022-05-18T04:18:26.2212895Z 
2022-05-18T04:18:26.2212997Z OK
2022-05-18T04:18:26.2213114Z 
2022-05-18T04:18:26.2213254Z Generating XML reports...
2022-05-18T04:18:26.2253627Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041823.xml
2022-05-18T04:18:27.6206139Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:27.6220437Z 
2022-05-18T04:18:27.6220885Z Running tests...
2022-05-18T04:18:27.6221391Z ----------------------------------------------------------------------
2022-05-18T04:18:29.2182447Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:29.2584312Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17021
2022-05-18T04:18:29.2692854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17022
2022-05-18T04:18:30.4421449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:18:30.4550111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:18:30.4551633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:30.4623212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:30.4629121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:30.5562370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:30.7745414Z ok (3.152s)
2022-05-18T04:18:30.7745638Z 
2022-05-18T04:18:30.7746014Z ----------------------------------------------------------------------
2022-05-18T04:18:30.7746364Z Ran 1 test in 3.152s
2022-05-18T04:18:30.7746531Z 
2022-05-18T04:18:30.7746634Z OK
2022-05-18T04:18:30.7746773Z 
2022-05-18T04:18:30.7746911Z Generating XML reports...
2022-05-18T04:18:30.7787701Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041827.xml
2022-05-18T04:18:32.1879545Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:32.1894872Z 
2022-05-18T04:18:32.1895270Z Running tests...
2022-05-18T04:18:32.1895784Z ----------------------------------------------------------------------
2022-05-18T04:18:32.1916253Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:18:32.1916598Z 
2022-05-18T04:18:32.1916903Z ----------------------------------------------------------------------
2022-05-18T04:18:32.1917238Z Ran 1 test in 0.002s
2022-05-18T04:18:32.1917405Z 
2022-05-18T04:18:32.1917518Z OK (skipped=1)
2022-05-18T04:18:32.1917674Z 
2022-05-18T04:18:32.1917782Z Generating XML reports...
2022-05-18T04:18:32.1953753Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041832.xml
2022-05-18T04:18:33.4486943Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:33.4501684Z 
2022-05-18T04:18:33.4502041Z Running tests...
2022-05-18T04:18:33.4502489Z ----------------------------------------------------------------------
2022-05-18T04:18:33.4522382Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:18:33.4522732Z 
2022-05-18T04:18:33.4523033Z ----------------------------------------------------------------------
2022-05-18T04:18:33.4523373Z Ran 1 test in 0.002s
2022-05-18T04:18:33.4523540Z 
2022-05-18T04:18:33.4523653Z OK (skipped=1)
2022-05-18T04:18:33.4523792Z 
2022-05-18T04:18:33.4523922Z Generating XML reports...
2022-05-18T04:18:33.4559819Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041833.xml
2022-05-18T04:18:34.7128575Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:34.7143117Z 
2022-05-18T04:18:34.7143424Z Running tests...
2022-05-18T04:18:34.7143870Z ----------------------------------------------------------------------
2022-05-18T04:18:36.3079486Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:36.3474441Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17194
2022-05-18T04:18:36.3578747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17195
2022-05-18T04:18:37.5160018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:18:37.5208675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:18:37.5209474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:37.5261334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:37.5267620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:37.6223632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:38.7907841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsmqysg7p
2022-05-18T04:18:38.7908479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsmqysg7p/_remote_module_non_scriptable.py
2022-05-18T04:18:38.9021194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdj770sti
2022-05-18T04:18:38.9022031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdj770sti/_remote_module_non_scriptable.py
2022-05-18T04:18:39.4662167Z ok (4.751s)
2022-05-18T04:18:39.4662419Z 
2022-05-18T04:18:39.4662815Z ----------------------------------------------------------------------
2022-05-18T04:18:39.4663160Z Ran 1 test in 4.752s
2022-05-18T04:18:39.4663331Z 
2022-05-18T04:18:39.4663429Z OK
2022-05-18T04:18:39.4663575Z 
2022-05-18T04:18:39.4663692Z Generating XML reports...
2022-05-18T04:18:39.4705014Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041834.xml
2022-05-18T04:18:40.8905944Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:40.8920117Z 
2022-05-18T04:18:40.8920550Z Running tests...
2022-05-18T04:18:40.8921080Z ----------------------------------------------------------------------
2022-05-18T04:18:42.4681089Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:42.5075516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17314
2022-05-18T04:18:42.5178636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17315
2022-05-18T04:18:43.6801856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:18:43.7011294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:18:43.7012144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:43.7107652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:43.7114583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:43.8026412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:44.9577034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88djxoae
2022-05-18T04:18:44.9578206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88djxoae/_remote_module_non_scriptable.py
2022-05-18T04:18:45.0712215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87t_m_vt
2022-05-18T04:18:45.0713313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87t_m_vt/_remote_module_non_scriptable.py
2022-05-18T04:18:45.7260821Z ok (4.834s)
2022-05-18T04:18:45.7261047Z 
2022-05-18T04:18:45.7261463Z ----------------------------------------------------------------------
2022-05-18T04:18:45.7261804Z Ran 1 test in 4.834s
2022-05-18T04:18:45.7261974Z 
2022-05-18T04:18:45.7262354Z OK
2022-05-18T04:18:45.7262508Z 
2022-05-18T04:18:45.7262650Z Generating XML reports...
2022-05-18T04:18:45.7303671Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041840.xml
2022-05-18T04:18:47.1679311Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:47.1693942Z 
2022-05-18T04:18:47.1694329Z Running tests...
2022-05-18T04:18:47.1694907Z ----------------------------------------------------------------------
2022-05-18T04:18:48.7342504Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:48.7729014Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17434
2022-05-18T04:18:48.7831697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17435
2022-05-18T04:18:49.9373791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:18:49.9627428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:18:49.9628250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:49.9677628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:49.9684407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:50.0641945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:52.6928768Z ok (5.523s)
2022-05-18T04:18:52.6929050Z 
2022-05-18T04:18:52.6929695Z ----------------------------------------------------------------------
2022-05-18T04:18:52.6930159Z Ran 1 test in 5.523s
2022-05-18T04:18:52.6930331Z 
2022-05-18T04:18:52.6930437Z OK
2022-05-18T04:18:52.6930576Z 
2022-05-18T04:18:52.6930738Z Generating XML reports...
2022-05-18T04:18:52.6971237Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041847.xml
2022-05-18T04:18:54.0896536Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:18:54.0910281Z 
2022-05-18T04:18:54.0910548Z Running tests...
2022-05-18T04:18:54.0911003Z ----------------------------------------------------------------------
2022-05-18T04:18:55.6222222Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:55.6605284Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17551
2022-05-18T04:18:55.6709732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17552
2022-05-18T04:18:56.7790345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:18:56.7999616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:18:56.8000413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:56.8095909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:18:56.8102038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:56.9014869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:59.5802031Z ok (5.489s)
2022-05-18T04:18:59.5802280Z 
2022-05-18T04:18:59.5802687Z ----------------------------------------------------------------------
2022-05-18T04:18:59.5803012Z Ran 1 test in 5.489s
2022-05-18T04:18:59.5803179Z 
2022-05-18T04:18:59.5803275Z OK
2022-05-18T04:18:59.5804221Z 
2022-05-18T04:18:59.5804384Z Generating XML reports...
2022-05-18T04:18:59.5844285Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041854.xml
2022-05-18T04:19:00.9676761Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:00.9691004Z 
2022-05-18T04:19:00.9691304Z Running tests...
2022-05-18T04:19:00.9691755Z ----------------------------------------------------------------------
2022-05-18T04:19:02.5328910Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:02.5477090Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.578s)
2022-05-18T04:19:02.5477692Z 
2022-05-18T04:19:02.5477984Z ----------------------------------------------------------------------
2022-05-18T04:19:02.5478637Z Ran 1 test in 1.579s
2022-05-18T04:19:02.5478806Z 
2022-05-18T04:19:02.5478919Z OK (skipped=1)
2022-05-18T04:19:02.5479060Z 
2022-05-18T04:19:02.5479187Z Generating XML reports...
2022-05-18T04:19:02.5509877Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041900.xml
2022-05-18T04:19:03.9330554Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:03.9344805Z 
2022-05-18T04:19:03.9345064Z Running tests...
2022-05-18T04:19:03.9345496Z ----------------------------------------------------------------------
2022-05-18T04:19:05.5106163Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:05.5255530Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.591s)
2022-05-18T04:19:05.5256106Z 
2022-05-18T04:19:05.5256389Z ----------------------------------------------------------------------
2022-05-18T04:19:05.5256724Z Ran 1 test in 1.591s
2022-05-18T04:19:05.5256872Z 
2022-05-18T04:19:05.5256982Z OK (skipped=1)
2022-05-18T04:19:05.5257137Z 
2022-05-18T04:19:05.5257266Z Generating XML reports...
2022-05-18T04:19:05.5288627Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041903.xml
2022-05-18T04:19:06.9035474Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:06.9050371Z 
2022-05-18T04:19:06.9050829Z Running tests...
2022-05-18T04:19:06.9051509Z ----------------------------------------------------------------------
2022-05-18T04:19:08.4874069Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:08.5275198Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17736
2022-05-18T04:19:08.5382291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17737
2022-05-18T04:19:09.6494617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:09.6641019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:09.6641830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:09.6697103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:09.6703427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:09.7655360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:09.9434332Z skip: Need at least 4 CUDA devices (3.038s)
2022-05-18T04:19:09.9434579Z 
2022-05-18T04:19:09.9434959Z ----------------------------------------------------------------------
2022-05-18T04:19:09.9435299Z Ran 1 test in 3.038s
2022-05-18T04:19:09.9435446Z 
2022-05-18T04:19:09.9435563Z OK (skipped=1)
2022-05-18T04:19:09.9435722Z 
2022-05-18T04:19:09.9435851Z Generating XML reports...
2022-05-18T04:19:09.9477576Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041906.xml
2022-05-18T04:19:11.3541365Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:11.3555905Z 
2022-05-18T04:19:11.3556308Z Running tests...
2022-05-18T04:19:11.3556809Z ----------------------------------------------------------------------
2022-05-18T04:19:12.9395405Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:12.9790836Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17843
2022-05-18T04:19:12.9896829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17844
2022-05-18T04:19:14.0600193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:14.1202959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:14.1204033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:14.1207563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:14.1214175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:14.2217533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:14.3950259Z skip: Need at least 4 CUDA devices (3.039s)
2022-05-18T04:19:14.3950597Z 
2022-05-18T04:19:14.3951109Z ----------------------------------------------------------------------
2022-05-18T04:19:14.3951461Z Ran 1 test in 3.039s
2022-05-18T04:19:14.3951626Z 
2022-05-18T04:19:14.3951739Z OK (skipped=1)
2022-05-18T04:19:14.3951998Z 
2022-05-18T04:19:14.3952233Z Generating XML reports...
2022-05-18T04:19:14.3992269Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041911.xml
2022-05-18T04:19:15.7982975Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:15.7998097Z 
2022-05-18T04:19:15.7998258Z Running tests...
2022-05-18T04:19:15.7998692Z ----------------------------------------------------------------------
2022-05-18T04:19:15.8020541Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:15.8020865Z 
2022-05-18T04:19:15.8021156Z ----------------------------------------------------------------------
2022-05-18T04:19:15.8021495Z Ran 1 test in 0.002s
2022-05-18T04:19:15.8021644Z 
2022-05-18T04:19:15.8021754Z OK (skipped=1)
2022-05-18T04:19:15.8021910Z 
2022-05-18T04:19:15.8022036Z Generating XML reports...
2022-05-18T04:19:15.8058033Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041915.xml
2022-05-18T04:19:17.0640527Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:17.0656449Z 
2022-05-18T04:19:17.0656600Z Running tests...
2022-05-18T04:19:17.0657325Z ----------------------------------------------------------------------
2022-05-18T04:19:17.0678200Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:17.0679141Z 
2022-05-18T04:19:17.0679461Z ----------------------------------------------------------------------
2022-05-18T04:19:17.0679787Z Ran 1 test in 0.002s
2022-05-18T04:19:17.0679954Z 
2022-05-18T04:19:17.0680065Z OK (skipped=1)
2022-05-18T04:19:17.0680224Z 
2022-05-18T04:19:17.0680350Z Generating XML reports...
2022-05-18T04:19:17.0716618Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041917.xml
2022-05-18T04:19:18.3225211Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:18.3240510Z 
2022-05-18T04:19:18.3240930Z Running tests...
2022-05-18T04:19:18.3241425Z ----------------------------------------------------------------------
2022-05-18T04:19:18.3263517Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:18.3264243Z 
2022-05-18T04:19:18.3264700Z ----------------------------------------------------------------------
2022-05-18T04:19:18.3265061Z Ran 1 test in 0.002s
2022-05-18T04:19:18.3265229Z 
2022-05-18T04:19:18.3265324Z OK (skipped=1)
2022-05-18T04:19:18.3265482Z 
2022-05-18T04:19:18.3265609Z Generating XML reports...
2022-05-18T04:19:18.3302508Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041918.xml
2022-05-18T04:19:19.5794500Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:19.5809938Z 
2022-05-18T04:19:19.5810092Z Running tests...
2022-05-18T04:19:19.5810773Z ----------------------------------------------------------------------
2022-05-18T04:19:19.5832031Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:19.5832576Z 
2022-05-18T04:19:19.5832879Z ----------------------------------------------------------------------
2022-05-18T04:19:19.5833215Z Ran 1 test in 0.002s
2022-05-18T04:19:19.5833393Z 
2022-05-18T04:19:19.5833503Z OK (skipped=1)
2022-05-18T04:19:19.5833659Z 
2022-05-18T04:19:19.5833769Z Generating XML reports...
2022-05-18T04:19:19.5869792Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041919.xml
2022-05-18T04:19:20.8330206Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:20.8343952Z 
2022-05-18T04:19:20.8344207Z Running tests...
2022-05-18T04:19:20.8344653Z ----------------------------------------------------------------------
2022-05-18T04:19:20.8364891Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:20.8365225Z 
2022-05-18T04:19:20.8365498Z ----------------------------------------------------------------------
2022-05-18T04:19:20.8365826Z Ran 1 test in 0.002s
2022-05-18T04:19:20.8365997Z 
2022-05-18T04:19:20.8366114Z OK (skipped=1)
2022-05-18T04:19:20.8366284Z 
2022-05-18T04:19:20.8366412Z Generating XML reports...
2022-05-18T04:19:20.8400086Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041920.xml
2022-05-18T04:19:22.0902768Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:22.0917788Z 
2022-05-18T04:19:22.0918099Z Running tests...
2022-05-18T04:19:22.0918525Z ----------------------------------------------------------------------
2022-05-18T04:19:22.0939323Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:22.0939961Z 
2022-05-18T04:19:22.0940270Z ----------------------------------------------------------------------
2022-05-18T04:19:22.0940615Z Ran 1 test in 0.002s
2022-05-18T04:19:22.0940790Z 
2022-05-18T04:19:22.0940885Z OK (skipped=1)
2022-05-18T04:19:22.0941042Z 
2022-05-18T04:19:22.0941171Z Generating XML reports...
2022-05-18T04:19:22.0978122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041922.xml
2022-05-18T04:19:23.3600878Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:23.3616077Z 
2022-05-18T04:19:23.3616368Z Running tests...
2022-05-18T04:19:23.3616820Z ----------------------------------------------------------------------
2022-05-18T04:19:23.3638752Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:23.3639073Z 
2022-05-18T04:19:23.3639360Z ----------------------------------------------------------------------
2022-05-18T04:19:23.3639714Z Ran 1 test in 0.002s
2022-05-18T04:19:23.3639879Z 
2022-05-18T04:19:23.3639996Z OK (skipped=1)
2022-05-18T04:19:23.3640134Z 
2022-05-18T04:19:23.3640267Z Generating XML reports...
2022-05-18T04:19:23.3676782Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041923.xml
2022-05-18T04:19:24.6171892Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:24.6186597Z 
2022-05-18T04:19:24.6186759Z Running tests...
2022-05-18T04:19:24.6187492Z ----------------------------------------------------------------------
2022-05-18T04:19:24.6209074Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:24.6209374Z 
2022-05-18T04:19:24.6209669Z ----------------------------------------------------------------------
2022-05-18T04:19:24.6210009Z Ran 1 test in 0.002s
2022-05-18T04:19:24.6210495Z 
2022-05-18T04:19:24.6210611Z OK (skipped=1)
2022-05-18T04:19:24.6210776Z 
2022-05-18T04:19:24.6210908Z Generating XML reports...
2022-05-18T04:19:24.6247173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041924.xml
2022-05-18T04:19:25.8812735Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:25.8827735Z 
2022-05-18T04:19:25.8828096Z Running tests...
2022-05-18T04:19:25.8828548Z ----------------------------------------------------------------------
2022-05-18T04:19:25.8849433Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:25.8849957Z 
2022-05-18T04:19:25.8850249Z ----------------------------------------------------------------------
2022-05-18T04:19:25.8850585Z Ran 1 test in 0.002s
2022-05-18T04:19:25.8850750Z 
2022-05-18T04:19:25.8850862Z OK (skipped=1)
2022-05-18T04:19:25.8851001Z 
2022-05-18T04:19:25.8851150Z Generating XML reports...
2022-05-18T04:19:25.8888119Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041925.xml
2022-05-18T04:19:27.1550229Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:27.1565899Z 
2022-05-18T04:19:27.1566173Z Running tests...
2022-05-18T04:19:27.1566631Z ----------------------------------------------------------------------
2022-05-18T04:19:27.1586894Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:27.1587197Z 
2022-05-18T04:19:27.1587489Z ----------------------------------------------------------------------
2022-05-18T04:19:27.1587805Z Ran 1 test in 0.002s
2022-05-18T04:19:27.1587973Z 
2022-05-18T04:19:27.1588511Z OK (skipped=1)
2022-05-18T04:19:27.1588674Z 
2022-05-18T04:19:27.1589036Z Generating XML reports...
2022-05-18T04:19:27.1625819Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041927.xml
2022-05-18T04:19:28.3774417Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:28.3789623Z 
2022-05-18T04:19:28.3789956Z Running tests...
2022-05-18T04:19:28.3790704Z ----------------------------------------------------------------------
2022-05-18T04:19:29.9650215Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:30.0035660Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18280
2022-05-18T04:19:30.0141852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18281
2022-05-18T04:19:31.1556029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:31.1630139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:31.1631001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:31.1657080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:31.1663414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:31.2645620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:33.3227934Z ok (4.943s)
2022-05-18T04:19:33.3228333Z 
2022-05-18T04:19:33.3228991Z ----------------------------------------------------------------------
2022-05-18T04:19:33.3229608Z Ran 1 test in 4.944s
2022-05-18T04:19:33.3229911Z 
2022-05-18T04:19:33.3230077Z OK
2022-05-18T04:19:33.3230326Z 
2022-05-18T04:19:33.3230572Z Generating XML reports...
2022-05-18T04:19:33.3272415Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041928.xml
2022-05-18T04:19:34.7453479Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:34.7468410Z 
2022-05-18T04:19:34.7468568Z Running tests...
2022-05-18T04:19:34.7469265Z ----------------------------------------------------------------------
2022-05-18T04:19:34.7491856Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:34.7492179Z 
2022-05-18T04:19:34.7492457Z ----------------------------------------------------------------------
2022-05-18T04:19:34.7492797Z Ran 1 test in 0.002s
2022-05-18T04:19:34.7492967Z 
2022-05-18T04:19:34.7493087Z OK (skipped=1)
2022-05-18T04:19:34.7493246Z 
2022-05-18T04:19:34.7493375Z Generating XML reports...
2022-05-18T04:19:34.7530407Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041934.xml
2022-05-18T04:19:36.0159261Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:36.0174917Z 
2022-05-18T04:19:36.0175084Z Running tests...
2022-05-18T04:19:36.0176028Z ----------------------------------------------------------------------
2022-05-18T04:19:36.0196925Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:36.0197427Z 
2022-05-18T04:19:36.0197821Z ----------------------------------------------------------------------
2022-05-18T04:19:36.0198181Z Ran 1 test in 0.002s
2022-05-18T04:19:36.0198623Z 
2022-05-18T04:19:36.0198746Z OK (skipped=1)
2022-05-18T04:19:36.0198906Z 
2022-05-18T04:19:36.0199036Z Generating XML reports...
2022-05-18T04:19:36.0234368Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041936.xml
2022-05-18T04:19:37.2780646Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:37.2794898Z 
2022-05-18T04:19:37.2795416Z Running tests...
2022-05-18T04:19:37.2795906Z ----------------------------------------------------------------------
2022-05-18T04:19:38.8685160Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:38.9077192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18466
2022-05-18T04:19:38.9181091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18467
2022-05-18T04:19:40.0808720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:40.1060742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:40.1061593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:40.1112976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:40.1119399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:40.2075265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:42.2267612Z ok (4.947s)
2022-05-18T04:19:42.2268008Z 
2022-05-18T04:19:42.2268684Z ----------------------------------------------------------------------
2022-05-18T04:19:42.2269321Z Ran 1 test in 4.947s
2022-05-18T04:19:42.2269616Z 
2022-05-18T04:19:42.2269788Z OK
2022-05-18T04:19:42.2270045Z 
2022-05-18T04:19:42.2270274Z Generating XML reports...
2022-05-18T04:19:42.2311906Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041937.xml
2022-05-18T04:19:43.6212747Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:43.6226982Z 
2022-05-18T04:19:43.6227358Z Running tests...
2022-05-18T04:19:43.6227790Z ----------------------------------------------------------------------
2022-05-18T04:19:45.1612052Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:45.1994813Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18586
2022-05-18T04:19:45.2104856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18587
2022-05-18T04:19:46.3270809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:46.3679377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:46.3680203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:46.3778069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:46.3784969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:46.4694734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:48.5189514Z ok (4.896s)
2022-05-18T04:19:48.5189787Z 
2022-05-18T04:19:48.5190404Z ----------------------------------------------------------------------
2022-05-18T04:19:48.5190772Z Ran 1 test in 4.896s
2022-05-18T04:19:48.5190941Z 
2022-05-18T04:19:48.5191037Z OK
2022-05-18T04:19:48.5191174Z 
2022-05-18T04:19:48.5191325Z Generating XML reports...
2022-05-18T04:19:48.5233636Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041943.xml
2022-05-18T04:19:49.9313894Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:49.9329548Z 
2022-05-18T04:19:49.9329766Z Running tests...
2022-05-18T04:19:49.9330222Z ----------------------------------------------------------------------
2022-05-18T04:19:49.9351225Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:49.9351533Z 
2022-05-18T04:19:49.9351806Z ----------------------------------------------------------------------
2022-05-18T04:19:49.9352157Z Ran 1 test in 0.002s
2022-05-18T04:19:49.9352451Z 
2022-05-18T04:19:49.9352955Z OK (skipped=1)
2022-05-18T04:19:49.9353145Z 
2022-05-18T04:19:49.9353276Z Generating XML reports...
2022-05-18T04:19:49.9388560Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041949.xml
2022-05-18T04:19:51.1705886Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:51.1720906Z 
2022-05-18T04:19:51.1721059Z Running tests...
2022-05-18T04:19:51.1721799Z ----------------------------------------------------------------------
2022-05-18T04:19:51.1741333Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:51.1741828Z 
2022-05-18T04:19:51.1742121Z ----------------------------------------------------------------------
2022-05-18T04:19:51.1742472Z Ran 1 test in 0.002s
2022-05-18T04:19:51.1742643Z 
2022-05-18T04:19:51.1742737Z OK (skipped=1)
2022-05-18T04:19:51.1742894Z 
2022-05-18T04:19:51.1743032Z Generating XML reports...
2022-05-18T04:19:51.1778903Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041951.xml
2022-05-18T04:19:52.3933856Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:52.3948281Z 
2022-05-18T04:19:52.3948428Z Running tests...
2022-05-18T04:19:52.3949263Z ----------------------------------------------------------------------
2022-05-18T04:19:52.3975694Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:52.3976209Z 
2022-05-18T04:19:52.3976963Z ----------------------------------------------------------------------
2022-05-18T04:19:52.3977338Z Ran 1 test in 0.003s
2022-05-18T04:19:52.3977508Z 
2022-05-18T04:19:52.3977634Z OK (skipped=1)
2022-05-18T04:19:52.3977792Z 
2022-05-18T04:19:52.3977919Z Generating XML reports...
2022-05-18T04:19:52.4014372Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041952.xml
2022-05-18T04:19:53.6556470Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:53.6572228Z 
2022-05-18T04:19:53.6572658Z Running tests...
2022-05-18T04:19:53.6573144Z ----------------------------------------------------------------------
2022-05-18T04:19:53.6592430Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:19:53.6593101Z 
2022-05-18T04:19:53.6593407Z ----------------------------------------------------------------------
2022-05-18T04:19:53.6593781Z Ran 1 test in 0.002s
2022-05-18T04:19:53.6593953Z 
2022-05-18T04:19:53.6594047Z OK (skipped=1)
2022-05-18T04:19:53.6594207Z 
2022-05-18T04:19:53.6594335Z Generating XML reports...
2022-05-18T04:19:53.6630458Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041953.xml
2022-05-18T04:19:54.9198521Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:19:54.9213287Z 
2022-05-18T04:19:54.9214004Z Running tests...
2022-05-18T04:19:54.9214545Z ----------------------------------------------------------------------
2022-05-18T04:19:56.5113557Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:56.5509342Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18838
2022-05-18T04:19:56.5611767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18839
2022-05-18T04:19:57.6945306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:57.7087827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:57.7088848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:57.7147790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:57.7153767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:57.8102702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:59.8696043Z ok (4.948s)
2022-05-18T04:19:59.8696260Z 
2022-05-18T04:19:59.8696655Z ----------------------------------------------------------------------
2022-05-18T04:19:59.8697006Z Ran 1 test in 4.948s
2022-05-18T04:19:59.8697156Z 
2022-05-18T04:19:59.8697282Z OK
2022-05-18T04:19:59.8697421Z 
2022-05-18T04:19:59.8697559Z Generating XML reports...
2022-05-18T04:19:59.8739290Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041954.xml
2022-05-18T04:20:01.2704630Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:01.2718528Z 
2022-05-18T04:20:01.2718954Z Running tests...
2022-05-18T04:20:01.2719451Z ----------------------------------------------------------------------
2022-05-18T04:20:02.8103587Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:02.8492806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18958
2022-05-18T04:20:02.8600642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18959
2022-05-18T04:20:03.9983127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:04.0261276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:04.0262129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:04.0287945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:04.0295391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:04.1276314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:06.1686309Z ok (4.896s)
2022-05-18T04:20:06.1686534Z 
2022-05-18T04:20:06.1686949Z ----------------------------------------------------------------------
2022-05-18T04:20:06.1687297Z Ran 1 test in 4.897s
2022-05-18T04:20:06.1687450Z 
2022-05-18T04:20:06.1687552Z OK
2022-05-18T04:20:06.1688048Z 
2022-05-18T04:20:06.1688434Z Generating XML reports...
2022-05-18T04:20:06.1731198Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042001.xml
2022-05-18T04:20:07.5841071Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:07.5856015Z 
2022-05-18T04:20:07.5856292Z Running tests...
2022-05-18T04:20:07.5856743Z ----------------------------------------------------------------------
2022-05-18T04:20:07.5875420Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:20:07.5875745Z 
2022-05-18T04:20:07.5876034Z ----------------------------------------------------------------------
2022-05-18T04:20:07.5876371Z Ran 1 test in 0.002s
2022-05-18T04:20:07.5876538Z 
2022-05-18T04:20:07.5876656Z OK (skipped=1)
2022-05-18T04:20:07.5876796Z 
2022-05-18T04:20:07.5876926Z Generating XML reports...
2022-05-18T04:20:07.5912166Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042007.xml
2022-05-18T04:20:08.8455811Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:08.8470471Z 
2022-05-18T04:20:08.8470988Z Running tests...
2022-05-18T04:20:08.8471778Z ----------------------------------------------------------------------
2022-05-18T04:20:08.8490385Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:20:08.8490707Z 
2022-05-18T04:20:08.8491009Z ----------------------------------------------------------------------
2022-05-18T04:20:08.8491343Z Ran 1 test in 0.002s
2022-05-18T04:20:08.8491510Z 
2022-05-18T04:20:08.8491622Z OK (skipped=1)
2022-05-18T04:20:08.8491764Z 
2022-05-18T04:20:08.8491902Z Generating XML reports...
2022-05-18T04:20:08.8526798Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042008.xml
2022-05-18T04:20:10.1035769Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:10.1050321Z 
2022-05-18T04:20:10.1050716Z Running tests...
2022-05-18T04:20:10.1051240Z ----------------------------------------------------------------------
2022-05-18T04:20:10.1075443Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:20:10.1075774Z 
2022-05-18T04:20:10.1076051Z ----------------------------------------------------------------------
2022-05-18T04:20:10.1076374Z Ran 1 test in 0.003s
2022-05-18T04:20:10.1076541Z 
2022-05-18T04:20:10.1076653Z OK (skipped=1)
2022-05-18T04:20:10.1076810Z 
2022-05-18T04:20:10.1076936Z Generating XML reports...
2022-05-18T04:20:10.1112524Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042010.xml
2022-05-18T04:20:11.3412299Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:11.3426313Z 
2022-05-18T04:20:11.3426465Z Running tests...
2022-05-18T04:20:11.3427141Z ----------------------------------------------------------------------
2022-05-18T04:20:11.3445259Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:20:11.3445584Z 
2022-05-18T04:20:11.3445874Z ----------------------------------------------------------------------
2022-05-18T04:20:11.3446211Z Ran 1 test in 0.002s
2022-05-18T04:20:11.3446361Z 
2022-05-18T04:20:11.3446474Z OK (skipped=1)
2022-05-18T04:20:11.3446632Z 
2022-05-18T04:20:11.3446759Z Generating XML reports...
2022-05-18T04:20:11.3481536Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042011.xml
2022-05-18T04:20:12.5620461Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:12.5635062Z 
2022-05-18T04:20:12.5635284Z Running tests...
2022-05-18T04:20:12.5635727Z ----------------------------------------------------------------------
2022-05-18T04:20:12.5654678Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-05-18T04:20:12.5655020Z 
2022-05-18T04:20:12.5655317Z ----------------------------------------------------------------------
2022-05-18T04:20:12.5655633Z Ran 1 test in 0.002s
2022-05-18T04:20:12.5655799Z 
2022-05-18T04:20:12.5655910Z OK (skipped=1)
2022-05-18T04:20:12.5656072Z 
2022-05-18T04:20:12.5656200Z Generating XML reports...
2022-05-18T04:20:12.5692468Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042012.xml
2022-05-18T04:20:13.8308235Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:13.8323026Z 
2022-05-18T04:20:13.8323385Z Running tests...
2022-05-18T04:20:13.8323831Z ----------------------------------------------------------------------
2022-05-18T04:20:13.8342532Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-05-18T04:20:13.8342897Z 
2022-05-18T04:20:13.8343536Z ----------------------------------------------------------------------
2022-05-18T04:20:13.8343906Z Ran 1 test in 0.002s
2022-05-18T04:20:13.8344082Z 
2022-05-18T04:20:13.8344195Z OK (skipped=1)
2022-05-18T04:20:13.8344353Z 
2022-05-18T04:20:13.8344463Z Generating XML reports...
2022-05-18T04:20:13.8379613Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042013.xml
2022-05-18T04:20:15.0575633Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:15.0591528Z 
2022-05-18T04:20:15.0591686Z Running tests...
2022-05-18T04:20:15.0592409Z ----------------------------------------------------------------------
2022-05-18T04:20:15.0613206Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-05-18T04:20:15.0613564Z 
2022-05-18T04:20:15.0613842Z ----------------------------------------------------------------------
2022-05-18T04:20:15.0614194Z Ran 1 test in 0.002s
2022-05-18T04:20:15.0614360Z 
2022-05-18T04:20:15.0614473Z OK (skipped=1)
2022-05-18T04:20:15.0614631Z 
2022-05-18T04:20:15.0614758Z Generating XML reports...
2022-05-18T04:20:15.0651179Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042015.xml
2022-05-18T04:20:16.3287868Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:16.3301941Z 
2022-05-18T04:20:16.3302219Z Running tests...
2022-05-18T04:20:16.3302681Z ----------------------------------------------------------------------
2022-05-18T04:20:16.3321733Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:20:16.3322405Z 
2022-05-18T04:20:16.3322709Z ----------------------------------------------------------------------
2022-05-18T04:20:16.3323025Z Ran 1 test in 0.002s
2022-05-18T04:20:16.3323190Z 
2022-05-18T04:20:16.3323310Z OK (skipped=1)
2022-05-18T04:20:16.3323470Z 
2022-05-18T04:20:16.3323599Z Generating XML reports...
2022-05-18T04:20:16.3358334Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042016.xml
2022-05-18T04:20:17.5989449Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:17.6004370Z 
2022-05-18T04:20:17.6004646Z Running tests...
2022-05-18T04:20:17.6005095Z ----------------------------------------------------------------------
2022-05-18T04:20:19.2059059Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:19.2449317Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19342
2022-05-18T04:20:19.2554851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19343
2022-05-18T04:20:20.3931927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:20.4274630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:20.4275481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:20.4337277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:20.4343890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:20.5290177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:22.1632832Z ok (4.562s)
2022-05-18T04:20:22.1633060Z 
2022-05-18T04:20:22.1633453Z ----------------------------------------------------------------------
2022-05-18T04:20:22.1633799Z Ran 1 test in 4.563s
2022-05-18T04:20:22.1633955Z 
2022-05-18T04:20:22.1634053Z OK
2022-05-18T04:20:22.1634195Z 
2022-05-18T04:20:22.1634585Z Generating XML reports...
2022-05-18T04:20:22.1675240Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042017.xml
2022-05-18T04:20:23.5856909Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:23.5870849Z 
2022-05-18T04:20:23.5871293Z Running tests...
2022-05-18T04:20:23.5871794Z ----------------------------------------------------------------------
2022-05-18T04:20:25.1569604Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:25.1954914Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19458
2022-05-18T04:20:25.2057407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19459
2022-05-18T04:20:26.3498791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:26.3720317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:26.3721161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:26.3803572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:26.3810151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:26.4734565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:28.5142121Z ok (4.927s)
2022-05-18T04:20:28.5142636Z 
2022-05-18T04:20:28.5143181Z ----------------------------------------------------------------------
2022-05-18T04:20:28.5143713Z Ran 1 test in 4.927s
2022-05-18T04:20:28.5143883Z 
2022-05-18T04:20:28.5143960Z OK
2022-05-18T04:20:28.5144105Z 
2022-05-18T04:20:28.5144240Z Generating XML reports...
2022-05-18T04:20:28.5184796Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042023.xml
2022-05-18T04:20:29.9138578Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:29.9153107Z 
2022-05-18T04:20:29.9153397Z Running tests...
2022-05-18T04:20:29.9154081Z ----------------------------------------------------------------------
2022-05-18T04:20:31.4748157Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:31.5135226Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19578
2022-05-18T04:20:31.5241236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19579
2022-05-18T04:20:32.6518942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:32.6624745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:32.6625575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:32.6722088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:32.6728613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:32.7640395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:35.8344856Z ok (5.919s)
2022-05-18T04:20:35.8345085Z 
2022-05-18T04:20:35.8345488Z ----------------------------------------------------------------------
2022-05-18T04:20:35.8345861Z Ran 1 test in 5.919s
2022-05-18T04:20:35.8346037Z 
2022-05-18T04:20:35.8346116Z OK
2022-05-18T04:20:35.8346260Z 
2022-05-18T04:20:35.8346397Z Generating XML reports...
2022-05-18T04:20:35.8387832Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042029.xml
2022-05-18T04:20:37.2574536Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:37.2589648Z 
2022-05-18T04:20:37.2589831Z Running tests...
2022-05-18T04:20:37.2590520Z ----------------------------------------------------------------------
2022-05-18T04:20:37.2614407Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:20:37.2615091Z 
2022-05-18T04:20:37.2615397Z ----------------------------------------------------------------------
2022-05-18T04:20:37.2615752Z Ran 1 test in 0.003s
2022-05-18T04:20:37.2615937Z 
2022-05-18T04:20:37.2616051Z OK (skipped=1)
2022-05-18T04:20:37.2616192Z 
2022-05-18T04:20:37.2616323Z Generating XML reports...
2022-05-18T04:20:37.2659828Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042037.xml
2022-05-18T04:20:38.4847797Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:38.4862910Z 
2022-05-18T04:20:38.4863245Z Running tests...
2022-05-18T04:20:38.4863693Z ----------------------------------------------------------------------
2022-05-18T04:20:38.4883199Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:20:38.4883515Z 
2022-05-18T04:20:38.4883808Z ----------------------------------------------------------------------
2022-05-18T04:20:38.4884150Z Ran 1 test in 0.002s
2022-05-18T04:20:38.4884317Z 
2022-05-18T04:20:38.4884761Z OK (skipped=1)
2022-05-18T04:20:38.4884921Z 
2022-05-18T04:20:38.4885029Z Generating XML reports...
2022-05-18T04:20:38.4920991Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042038.xml
2022-05-18T04:20:39.7137983Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:39.7153234Z 
2022-05-18T04:20:39.7153483Z Running tests...
2022-05-18T04:20:39.7153933Z ----------------------------------------------------------------------
2022-05-18T04:20:39.7174774Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:20:39.7175116Z 
2022-05-18T04:20:39.7175412Z ----------------------------------------------------------------------
2022-05-18T04:20:39.7175723Z Ran 1 test in 0.002s
2022-05-18T04:20:39.7175893Z 
2022-05-18T04:20:39.7176006Z OK (skipped=1)
2022-05-18T04:20:39.7176164Z 
2022-05-18T04:20:39.7176302Z Generating XML reports...
2022-05-18T04:20:39.7212931Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042039.xml
2022-05-18T04:20:40.9609883Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:40.9624434Z 
2022-05-18T04:20:40.9624601Z Running tests...
2022-05-18T04:20:40.9625904Z ----------------------------------------------------------------------
2022-05-18T04:20:40.9645173Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:20:40.9646448Z 
2022-05-18T04:20:40.9646775Z ----------------------------------------------------------------------
2022-05-18T04:20:40.9647133Z Ran 1 test in 0.002s
2022-05-18T04:20:40.9647283Z 
2022-05-18T04:20:40.9647399Z OK (skipped=1)
2022-05-18T04:20:40.9647561Z 
2022-05-18T04:20:40.9647690Z Generating XML reports...
2022-05-18T04:20:40.9683304Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042040.xml
2022-05-18T04:20:42.2208480Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:42.2223392Z 
2022-05-18T04:20:42.2223554Z Running tests...
2022-05-18T04:20:42.2224534Z ----------------------------------------------------------------------
2022-05-18T04:20:42.2243538Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-05-18T04:20:42.2243878Z 
2022-05-18T04:20:42.2244173Z ----------------------------------------------------------------------
2022-05-18T04:20:42.2244488Z Ran 1 test in 0.002s
2022-05-18T04:20:42.2244654Z 
2022-05-18T04:20:42.2244767Z OK (skipped=1)
2022-05-18T04:20:42.2244924Z 
2022-05-18T04:20:42.2245054Z Generating XML reports...
2022-05-18T04:20:42.2282183Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042042.xml
2022-05-18T04:20:43.4833523Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:43.4848743Z 
2022-05-18T04:20:43.4849041Z Running tests...
2022-05-18T04:20:43.4849490Z ----------------------------------------------------------------------
2022-05-18T04:20:43.4869005Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-05-18T04:20:43.4869587Z 
2022-05-18T04:20:43.4869925Z ----------------------------------------------------------------------
2022-05-18T04:20:43.4870247Z Ran 1 test in 0.002s
2022-05-18T04:20:43.4870411Z 
2022-05-18T04:20:43.4870522Z OK (skipped=1)
2022-05-18T04:20:43.4870682Z 
2022-05-18T04:20:43.4870809Z Generating XML reports...
2022-05-18T04:20:43.4906796Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042043.xml
2022-05-18T04:20:44.7218181Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:44.7232630Z 
2022-05-18T04:20:44.7232882Z Running tests...
2022-05-18T04:20:44.7233579Z ----------------------------------------------------------------------
2022-05-18T04:20:46.2838980Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:46.3226159Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19896
2022-05-18T04:20:46.3331788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19897
2022-05-18T04:20:47.4771012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:47.4873996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:47.4874814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:47.4976029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:47.4982205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:47.5890526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:49.4413020Z ok (4.718s)
2022-05-18T04:20:49.4413376Z 
2022-05-18T04:20:49.4413787Z ----------------------------------------------------------------------
2022-05-18T04:20:49.4414140Z Ran 1 test in 4.718s
2022-05-18T04:20:49.4414309Z 
2022-05-18T04:20:49.4414405Z OK
2022-05-18T04:20:49.4414539Z 
2022-05-18T04:20:49.4414658Z Generating XML reports...
2022-05-18T04:20:49.4454619Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042044.xml
2022-05-18T04:20:50.8524807Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:50.8539354Z 
2022-05-18T04:20:50.8539700Z Running tests...
2022-05-18T04:20:50.8540285Z ----------------------------------------------------------------------
2022-05-18T04:20:50.8563769Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-05-18T04:20:50.8564402Z 
2022-05-18T04:20:50.8564838Z ----------------------------------------------------------------------
2022-05-18T04:20:50.8565184Z Ran 1 test in 0.002s
2022-05-18T04:20:50.8565355Z 
2022-05-18T04:20:50.8565467Z OK (skipped=1)
2022-05-18T04:20:50.8565624Z 
2022-05-18T04:20:50.8565733Z Generating XML reports...
2022-05-18T04:20:50.8601127Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042050.xml
2022-05-18T04:20:52.1175327Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:52.1189808Z 
2022-05-18T04:20:52.1190246Z Running tests...
2022-05-18T04:20:52.1190682Z ----------------------------------------------------------------------
2022-05-18T04:20:53.7065543Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:53.7465341Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20049
2022-05-18T04:20:53.7571691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20050
2022-05-18T04:20:54.8816401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:54.9238611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:54.9239692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:54.9322550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:54.9328886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:55.0253491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:56.2264217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5um56e4e
2022-05-18T04:20:56.2264947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5um56e4e/_remote_module_non_scriptable.py
2022-05-18T04:20:56.2723379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgezaycvw
2022-05-18T04:20:56.2724516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgezaycvw/_remote_module_non_scriptable.py
2022-05-18T04:20:56.6649102Z ok (4.545s)
2022-05-18T04:20:56.6649318Z 
2022-05-18T04:20:56.6649702Z ----------------------------------------------------------------------
2022-05-18T04:20:56.6650045Z Ran 1 test in 4.546s
2022-05-18T04:20:56.6650211Z 
2022-05-18T04:20:56.6650316Z OK
2022-05-18T04:20:56.6650454Z 
2022-05-18T04:20:56.6650590Z Generating XML reports...
2022-05-18T04:20:56.6690894Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042052.xml
2022-05-18T04:20:58.0964394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:20:58.0979176Z 
2022-05-18T04:20:58.0979493Z Running tests...
2022-05-18T04:20:58.0979931Z ----------------------------------------------------------------------
2022-05-18T04:20:59.6694048Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:59.7087344Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20165
2022-05-18T04:20:59.7190141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20166
2022-05-18T04:21:00.8486302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:00.8758076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:00.8759287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:00.8790595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:00.8796833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:00.9772517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:02.1549746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgv29eb8m
2022-05-18T04:21:02.1550676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgv29eb8m/_remote_module_non_scriptable.py
2022-05-18T04:21:02.2113695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgirrmaw
2022-05-18T04:21:02.2115157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgirrmaw/_remote_module_non_scriptable.py
2022-05-18T04:21:02.4468168Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:21:02.4512110Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:21:02.7270185Z ok (4.629s)
2022-05-18T04:21:02.7270394Z 
2022-05-18T04:21:02.7270834Z ----------------------------------------------------------------------
2022-05-18T04:21:02.7271179Z Ran 1 test in 4.629s
2022-05-18T04:21:02.7271346Z 
2022-05-18T04:21:02.7271447Z OK
2022-05-18T04:21:02.7271566Z 
2022-05-18T04:21:02.7271701Z Generating XML reports...
2022-05-18T04:21:02.7312333Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042058.xml
2022-05-18T04:21:04.1463560Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:21:04.1478301Z 
2022-05-18T04:21:04.1478654Z Running tests...
2022-05-18T04:21:04.1479599Z ----------------------------------------------------------------------
2022-05-18T04:21:05.7301969Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:05.7695310Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20285
2022-05-18T04:21:05.7801250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20286
2022-05-18T04:21:06.9145063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:06.9334088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:06.9334890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:06.9350007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:06.9356568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:07.0344720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:07.0557357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:21:07.0558593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:21:07.0559333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:21:07.0560066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:21:07.0561214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:21:07.0562533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:21:07.0563441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:21:07.0564335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:21:08.3360190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwstkvyx8
2022-05-18T04:21:08.3360810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwstkvyx8/_remote_module_non_scriptable.py
2022-05-18T04:21:08.3625301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbag__c4
2022-05-18T04:21:08.3627717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbag__c4/_remote_module_non_scriptable.py
2022-05-18T04:21:14.1978386Z ok (10.050s)
2022-05-18T04:21:14.1978600Z 
2022-05-18T04:21:14.1979020Z ----------------------------------------------------------------------
2022-05-18T04:21:14.1979669Z Ran 1 test in 10.050s
2022-05-18T04:21:14.1979837Z 
2022-05-18T04:21:14.1979932Z OK
2022-05-18T04:21:14.1980068Z 
2022-05-18T04:21:14.1980187Z Generating XML reports...
2022-05-18T04:21:14.2021328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042104.xml
2022-05-18T04:21:15.6046819Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:21:15.6060591Z 
2022-05-18T04:21:15.6060796Z Running tests...
2022-05-18T04:21:15.6061250Z ----------------------------------------------------------------------
2022-05-18T04:21:17.1548863Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:17.1935580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20409
2022-05-18T04:21:17.2037600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20410
2022-05-18T04:21:18.3356327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:18.3551473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:18.3552297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:18.3558976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:18.3566034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:18.4564660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:18.4686845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:21:18.4687342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:21:18.4688054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:21:18.4688747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:21:18.4691366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:21:18.4792391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:21:18.4793189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:21:18.4793879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:21:19.7625577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6_g2586
2022-05-18T04:21:19.7626206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6_g2586/_remote_module_non_scriptable.py
2022-05-18T04:21:19.7784918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpij0fuvni
2022-05-18T04:21:19.7788051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpij0fuvni/_remote_module_non_scriptable.py
2022-05-18T04:21:25.4211668Z ok (9.815s)
2022-05-18T04:21:25.4211898Z 
2022-05-18T04:21:25.4212306Z ----------------------------------------------------------------------
2022-05-18T04:21:25.4212774Z Ran 1 test in 9.815s
2022-05-18T04:21:25.4213051Z 
2022-05-18T04:21:25.4213210Z OK
2022-05-18T04:21:25.4213446Z 
2022-05-18T04:21:25.4214582Z Generating XML reports...
2022-05-18T04:21:25.4253762Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042115.xml
2022-05-18T04:21:25.8250430Z Running distributed tests for the nccl backend with file init_method
2022-05-18T04:21:25.8253217Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:21:25.824990]
2022-05-18T04:21:26.9831986Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:21:26.9875155Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-05-18T04:21:26.9909699Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9910334Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9910762Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9911186Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9911621Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9912107Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9912594Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9913093Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9913592Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9914145Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9914707Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9915218Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9915752Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9916395Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9916879Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9917351Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9917827Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9918868Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9919289Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9919745Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9920230Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9920707Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9921109Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9921511Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9921942Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9922351Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9922779Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9923207Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9923623Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9923992Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9924391Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9924798Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9925178Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9925579Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9925993Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9926401Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9926921Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9927367Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9927815Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9928243Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9928692Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9929122Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9929533Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9929977Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9930415Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9930838Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9931269Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9931711Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9932130Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9932528Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9932954Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9933384Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9933776Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9934196Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9934711Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9935114Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9935494Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9935901Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9936306Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9936676Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9937052Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9937446Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9937840Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9938249Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9938645Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9939039Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9939407Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9939803Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9940209Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9940595Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9941011Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9941402Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9941764Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9942147Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9942538Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9942941Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9943332Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9943728Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9944120Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9944566Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9945014Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9945456Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9945905Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9946343Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9946802Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9947254Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9947688Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9948125Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9948565Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9949018Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9949454Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9949918Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9950380Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9950820Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9951273Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9951767Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9952143Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9952528Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9952899Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9953275Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9953643Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9954044Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9954438Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9954807Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9955210Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9955624Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9956010Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9956418Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9956828Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9957257Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9957659Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9958650Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9959153Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9959553Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9959988Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9960420Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9960840Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9961225Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9961603Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9961997Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9962371Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9962851Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9963262Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9963702Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9964219Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9964680Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9965095Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9965498Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9965940Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9966439Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9966895Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9967342Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9967770Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9968219Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9968617Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9968997Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9969399Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9969800Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9970300Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9970740Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9971169Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9971599Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9972066Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9972570Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9973116Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9973732Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9974351Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9974963Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9975556Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9976164Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9976760Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9977364Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9977904Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9978404Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9978851Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9979329Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9979726Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9980140Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9980539Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9980945Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9981391Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9981856Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9982334Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9982736Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9983133Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9983557Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9983979Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9984413Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9984834Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9985235Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9985671Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9986106Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9986591Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9986986Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9987398Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9987824Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9988212Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9988635Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9989096Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9989528Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9989894Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9990297Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9990709Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9991144Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9991530Z test_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9991896Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9992254Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9992634Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9993013Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9993372Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9993760Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9994146Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9994495Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9994855Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9995235Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9995621Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9996015Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9996384Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9996784Z test_isend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9997160Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9997562Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9997977Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9999075Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9999530Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:26.9999946Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0000369Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0000796Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0001227Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0001660Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0002064Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0002481Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0002891Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0003284Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0003680Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0004083Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0004551Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0005124Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0005597Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0006053Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0006498Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0006956Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0007390Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0007812Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0008222Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0008660Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0009112Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0009574Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0010088Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0010545Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0010940Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0011327Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0011729Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0012117Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0012481Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0012865Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0013261Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0013613Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0013978Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0014354Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0014807Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0015177Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0015546Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0015935Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0016305Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0016674Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0017041Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0017404Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0017781Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0030103Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0030589Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0030994Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0031380Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0031760Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0032147Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0032555Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0033004Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0033439Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0033841Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0034233Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0034785Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0035207Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0035591Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0036019Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0036458Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0036855Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0037261Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0037671Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0038598Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0039104Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0039540Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0039997Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:27.0040426Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:21:28.1246219Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:21:28.1260261Z 
2022-05-18T04:21:28.1260790Z Running tests...
2022-05-18T04:21:28.1261300Z ----------------------------------------------------------------------
2022-05-18T04:21:29.6948447Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:29.7338881Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20566
2022-05-18T04:21:29.7439695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20567
2022-05-18T04:21:30.8845033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:30.9184586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:30.9185667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:30.9250141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:30.9256477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:31.0199496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:32.1890489Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:21:32.1891538Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:21:32.2688183Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:21:32.2689196Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:21:33.2839648Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:21:33.2840702Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:21:33.2841610Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:21:33.2842442Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:21:33.2976894Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:21:33.2977772Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:21:33.2980373Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:21:33.2981213Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:21:33.3114844Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:21:33.3115701Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:21:33.3118357Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:21:33.3119461Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:21:33.6537997Z ok (5.527s)
2022-05-18T04:21:33.6538224Z 
2022-05-18T04:21:33.6538623Z ----------------------------------------------------------------------
2022-05-18T04:21:33.6538950Z Ran 1 test in 5.528s
2022-05-18T04:21:33.6539117Z 
2022-05-18T04:21:33.6539221Z OK
2022-05-18T04:21:33.6539361Z 
2022-05-18T04:21:33.6539499Z Generating XML reports...
2022-05-18T04:21:33.6580505Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042128.xml
2022-05-18T04:21:35.0460467Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:21:35.0473980Z 
2022-05-18T04:21:35.0474414Z Running tests...
2022-05-18T04:21:35.0474916Z ----------------------------------------------------------------------
2022-05-18T04:21:35.0516770Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-05-18T04:21:35.0517166Z 
2022-05-18T04:21:35.0517504Z ----------------------------------------------------------------------
2022-05-18T04:21:35.0517840Z Ran 1 test in 0.004s
2022-05-18T04:21:35.0518015Z 
2022-05-18T04:21:35.0518451Z OK (skipped=1)
2022-05-18T04:21:35.0518623Z 
2022-05-18T04:21:35.0518754Z Generating XML reports...
2022-05-18T04:21:35.0551751Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042135.xml
2022-05-18T04:21:36.3055445Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:21:36.3069624Z 
2022-05-18T04:21:36.3069900Z Running tests...
2022-05-18T04:21:36.3070361Z ----------------------------------------------------------------------
2022-05-18T04:21:37.8714245Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:37.9105257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20716
2022-05-18T04:21:37.9207522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20717
2022-05-18T04:21:39.0557330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:39.0636052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:39.0636843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:39.0658888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:39.0665430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:39.1649335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:39.3260226Z ok (3.019s)
2022-05-18T04:21:39.3260464Z 
2022-05-18T04:21:39.3260838Z ----------------------------------------------------------------------
2022-05-18T04:21:39.3261190Z Ran 1 test in 3.019s
2022-05-18T04:21:39.3261357Z 
2022-05-18T04:21:39.3261461Z OK
2022-05-18T04:21:39.3261596Z 
2022-05-18T04:21:39.3261731Z Generating XML reports...
2022-05-18T04:21:39.3303044Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042136.xml
2022-05-18T04:21:40.7198803Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:21:40.7213214Z 
2022-05-18T04:21:40.7213472Z Running tests...
2022-05-18T04:21:40.7213942Z ----------------------------------------------------------------------
2022-05-18T04:21:42.2979409Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:42.3134307Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.592s)
2022-05-18T04:21:42.3134877Z 
2022-05-18T04:21:42.3135164Z ----------------------------------------------------------------------
2022-05-18T04:21:42.3135500Z Ran 1 test in 1.592s
2022-05-18T04:21:42.3135670Z 
2022-05-18T04:21:42.3135782Z OK (skipped=1)
2022-05-18T04:21:42.3135923Z 
2022-05-18T04:21:42.3136055Z Generating XML reports...
2022-05-18T04:21:42.3168647Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042140.xml
2022-05-18T04:21:43.6887467Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:21:43.6900353Z 
2022-05-18T04:21:43.6900614Z Running tests...
2022-05-18T04:21:43.6901036Z ----------------------------------------------------------------------
2022-05-18T04:21:43.6919238Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-05-18T04:21:43.6919636Z 
2022-05-18T04:21:43.6919946Z ----------------------------------------------------------------------
2022-05-18T04:21:43.6920297Z Ran 1 test in 0.002s
2022-05-18T04:21:43.6920452Z 
2022-05-18T04:21:43.6920566Z OK (skipped=1)
2022-05-18T04:21:43.6920730Z 
2022-05-18T04:21:43.6920861Z Generating XML reports...
2022-05-18T04:21:43.6955717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042143.xml
2022-05-18T04:21:44.9460606Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:21:44.9474601Z 
2022-05-18T04:21:44.9474982Z Running tests...
2022-05-18T04:21:44.9475430Z ----------------------------------------------------------------------
2022-05-18T04:21:44.9493423Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-05-18T04:21:44.9494462Z 
2022-05-18T04:21:44.9494770Z ----------------------------------------------------------------------
2022-05-18T04:21:44.9495112Z Ran 1 test in 0.002s
2022-05-18T04:21:44.9495283Z 
2022-05-18T04:21:44.9495395Z OK (skipped=1)
2022-05-18T04:21:44.9495555Z 
2022-05-18T04:21:44.9495682Z Generating XML reports...
2022-05-18T04:21:44.9530344Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042144.xml
2022-05-18T04:21:46.1665661Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:21:46.1680651Z 
2022-05-18T04:21:46.1680917Z Running tests...
2022-05-18T04:21:46.1681380Z ----------------------------------------------------------------------
2022-05-18T04:21:47.7691958Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:47.8081972Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20923
2022-05-18T04:21:47.8185212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20924
2022-05-18T04:21:49.0089597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:49.0137004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:49.0137782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:49.0191118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:49.0198017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:49.1151334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:50.2474039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3qwvv9i
2022-05-18T04:21:50.2475118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3qwvv9i/_remote_module_non_scriptable.py
2022-05-18T04:21:50.3647997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5k9vzx9m
2022-05-18T04:21:50.3649384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5k9vzx9m/_remote_module_non_scriptable.py
2022-05-18T04:21:51.1649279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:51.1649863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:51.1855014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:51.1855508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:51.2138591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:51.2139120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:51.2340761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:51.2341267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:51.3598355Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:51.3599096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:51.3807712Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:51.3808226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:51.7282459Z ok (5.560s)
2022-05-18T04:21:51.7282885Z 
2022-05-18T04:21:51.7283591Z ----------------------------------------------------------------------
2022-05-18T04:21:51.7284211Z Ran 1 test in 5.560s
2022-05-18T04:21:51.7284531Z 
2022-05-18T04:21:51.7284706Z OK
2022-05-18T04:21:51.7284958Z 
2022-05-18T04:21:51.7285174Z Generating XML reports...
2022-05-18T04:21:51.7326737Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042146.xml
2022-05-18T04:21:53.1095592Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:21:53.1109797Z 
2022-05-18T04:21:53.1110013Z Running tests...
2022-05-18T04:21:53.1110459Z ----------------------------------------------------------------------
2022-05-18T04:21:54.6919683Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:54.7311228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21043
2022-05-18T04:21:54.7413113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21044
2022-05-18T04:21:55.8850638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:55.9308926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:55.9309747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:55.9356987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:55.9363856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:56.0324452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:57.2109073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9dml_omk
2022-05-18T04:21:57.2109756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9dml_omk/_remote_module_non_scriptable.py
2022-05-18T04:21:57.2877817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphzgafw0r
2022-05-18T04:21:57.2879176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphzgafw0r/_remote_module_non_scriptable.py
2022-05-18T04:21:57.3865993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:57.3870365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:57.4014698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:57.4015194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:57.7493213Z ok (4.638s)
2022-05-18T04:21:57.7493429Z 
2022-05-18T04:21:57.7493821Z ----------------------------------------------------------------------
2022-05-18T04:21:57.7494164Z Ran 1 test in 4.638s
2022-05-18T04:21:57.7494578Z 
2022-05-18T04:21:57.7494674Z OK
2022-05-18T04:21:57.7494817Z 
2022-05-18T04:21:57.7494956Z Generating XML reports...
2022-05-18T04:21:57.7535335Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042153.xml
2022-05-18T04:21:59.1877672Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:21:59.1892756Z 
2022-05-18T04:21:59.1893226Z Running tests...
2022-05-18T04:21:59.1893733Z ----------------------------------------------------------------------
2022-05-18T04:22:00.7734879Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:00.8128271Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21163
2022-05-18T04:22:00.8230594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21164
2022-05-18T04:22:01.9630586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:01.9930196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:01.9931019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:01.9934580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:01.9941148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:02.0945250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:03.2439315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9dcbw6r
2022-05-18T04:22:03.2439953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9dcbw6r/_remote_module_non_scriptable.py
2022-05-18T04:22:03.3273286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrvhfjs_
2022-05-18T04:22:03.3274513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrvhfjs_/_remote_module_non_scriptable.py
2022-05-18T04:22:03.4208229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:03.4208767Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:03.4377661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:03.4379421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:03.7308460Z ok (4.541s)
2022-05-18T04:22:03.7308828Z 
2022-05-18T04:22:03.7309276Z ----------------------------------------------------------------------
2022-05-18T04:22:03.7309608Z Ran 1 test in 4.542s
2022-05-18T04:22:03.7309776Z 
2022-05-18T04:22:03.7309874Z OK
2022-05-18T04:22:03.7310011Z 
2022-05-18T04:22:03.7310168Z Generating XML reports...
2022-05-18T04:22:03.7351668Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042159.xml
2022-05-18T04:22:05.1657287Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:22:05.1672552Z 
2022-05-18T04:22:05.1672701Z Running tests...
2022-05-18T04:22:05.1673548Z ----------------------------------------------------------------------
2022-05-18T04:22:06.7559587Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:06.7954403Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21283
2022-05-18T04:22:06.8058709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21284
2022-05-18T04:22:07.9191886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:07.9385717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:07.9386597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:07.9394480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:07.9400516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:08.0400398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:09.1911985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyyaklca5
2022-05-18T04:22:09.1912789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyyaklca5/_remote_module_non_scriptable.py
2022-05-18T04:22:09.2883636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6at7kuyp
2022-05-18T04:22:09.2884869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6at7kuyp/_remote_module_non_scriptable.py
2022-05-18T04:22:09.3707787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:09.3708379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:09.8138275Z ok (4.646s)
2022-05-18T04:22:09.8138522Z 
2022-05-18T04:22:09.8138917Z ----------------------------------------------------------------------
2022-05-18T04:22:09.8139263Z Ran 1 test in 4.646s
2022-05-18T04:22:09.8139434Z 
2022-05-18T04:22:09.8139530Z OK
2022-05-18T04:22:09.8139928Z 
2022-05-18T04:22:09.8140067Z Generating XML reports...
2022-05-18T04:22:09.8180542Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042205.xml
2022-05-18T04:22:11.2341934Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:22:11.2356115Z 
2022-05-18T04:22:11.2356338Z Running tests...
2022-05-18T04:22:11.2356793Z ----------------------------------------------------------------------
2022-05-18T04:22:12.8057668Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:12.8448253Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21403
2022-05-18T04:22:12.8551516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21404
2022-05-18T04:22:13.9861175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:13.9902872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:13.9903699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:13.9962345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:13.9968629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:14.0917231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:15.2469769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8enx6ko
2022-05-18T04:22:15.2470418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8enx6ko/_remote_module_non_scriptable.py
2022-05-18T04:22:15.3129461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkkeyrlk
2022-05-18T04:22:15.3130879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkkeyrlk/_remote_module_non_scriptable.py
2022-05-18T04:22:16.1249636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:16.1250729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:16.1463302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:16.1463827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:16.4642153Z ok (5.228s)
2022-05-18T04:22:16.4642529Z 
2022-05-18T04:22:16.4643263Z ----------------------------------------------------------------------
2022-05-18T04:22:16.4643641Z Ran 1 test in 5.229s
2022-05-18T04:22:16.4643810Z 
2022-05-18T04:22:16.4643898Z OK
2022-05-18T04:22:16.4644057Z 
2022-05-18T04:22:16.4644298Z Generating XML reports...
2022-05-18T04:22:16.4685920Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042211.xml
2022-05-18T04:22:17.8677157Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:22:17.8690890Z 
2022-05-18T04:22:17.8691214Z Running tests...
2022-05-18T04:22:17.8691687Z ----------------------------------------------------------------------
2022-05-18T04:22:19.3962008Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:19.4341568Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21523
2022-05-18T04:22:19.4443315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21524
2022-05-18T04:22:20.5571829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:20.5758812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:20.5760195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:20.5775313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:20.5781381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:20.6773902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:21.8256195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvyzgnkt
2022-05-18T04:22:21.8257073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvyzgnkt/_remote_module_non_scriptable.py
2022-05-18T04:22:21.9442803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxr2xe2gq
2022-05-18T04:22:21.9444104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxr2xe2gq/_remote_module_non_scriptable.py
2022-05-18T04:22:22.2011004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:22.2014682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:22.2183842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:22.2186706Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:22.5526091Z ok (4.683s)
2022-05-18T04:22:22.5526338Z 
2022-05-18T04:22:22.5526970Z ----------------------------------------------------------------------
2022-05-18T04:22:22.5527718Z Ran 1 test in 4.683s
2022-05-18T04:22:22.5528034Z 
2022-05-18T04:22:22.5528189Z OK
2022-05-18T04:22:22.5528339Z 
2022-05-18T04:22:22.5528456Z Generating XML reports...
2022-05-18T04:22:22.5567938Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042217.xml
2022-05-18T04:22:23.9351134Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:22:23.9367286Z 
2022-05-18T04:22:23.9367804Z Running tests...
2022-05-18T04:22:23.9368427Z ----------------------------------------------------------------------
2022-05-18T04:22:25.5148371Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:25.5538692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21643
2022-05-18T04:22:25.5642211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21644
2022-05-18T04:22:26.6827047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:26.6935656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:26.6936518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:26.7030921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:26.7037417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:26.7950498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:27.9474641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7pc8x18
2022-05-18T04:22:27.9475290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7pc8x18/_remote_module_non_scriptable.py
2022-05-18T04:22:28.0567935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvi85cfwa
2022-05-18T04:22:28.0569809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvi85cfwa/_remote_module_non_scriptable.py
2022-05-18T04:22:28.1537362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:28.1537878Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:28.1674666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:28.1675174Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:28.4723027Z ok (4.535s)
2022-05-18T04:22:28.4723355Z 
2022-05-18T04:22:28.4724117Z ----------------------------------------------------------------------
2022-05-18T04:22:28.4724688Z Ran 1 test in 4.536s
2022-05-18T04:22:28.4724860Z 
2022-05-18T04:22:28.4724959Z OK
2022-05-18T04:22:28.4725098Z 
2022-05-18T04:22:28.4725232Z Generating XML reports...
2022-05-18T04:22:28.4765331Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042223.xml
2022-05-18T04:22:29.8903583Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:22:29.8917395Z 
2022-05-18T04:22:29.8917695Z Running tests...
2022-05-18T04:22:29.8918493Z ----------------------------------------------------------------------
2022-05-18T04:22:31.4700010Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:31.4853773Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.593s)
2022-05-18T04:22:31.4854345Z 
2022-05-18T04:22:31.4854627Z ----------------------------------------------------------------------
2022-05-18T04:22:31.4854948Z Ran 1 test in 1.594s
2022-05-18T04:22:31.4855115Z 
2022-05-18T04:22:31.4855227Z OK (skipped=1)
2022-05-18T04:22:31.4855397Z 
2022-05-18T04:22:31.4855525Z Generating XML reports...
2022-05-18T04:22:31.4888945Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042229.xml
2022-05-18T04:22:32.8670967Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:22:32.8686346Z 
2022-05-18T04:22:32.8686511Z Running tests...
2022-05-18T04:22:32.8686973Z ----------------------------------------------------------------------
2022-05-18T04:22:34.4319211Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:34.4702142Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21797
2022-05-18T04:22:34.4803181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21798
2022-05-18T04:22:35.6379424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:35.6564513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:35.6565327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:35.6582516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:35.6588514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:35.7576279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:35.9856740Z ok (3.117s)
2022-05-18T04:22:35.9856940Z 
2022-05-18T04:22:35.9857321Z ----------------------------------------------------------------------
2022-05-18T04:22:35.9857655Z Ran 1 test in 3.117s
2022-05-18T04:22:35.9857823Z 
2022-05-18T04:22:35.9857920Z OK
2022-05-18T04:22:35.9858055Z 
2022-05-18T04:22:35.9858477Z Generating XML reports...
2022-05-18T04:22:35.9899655Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042232.xml
2022-05-18T04:22:37.3850184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:22:37.3864123Z 
2022-05-18T04:22:37.3864431Z Running tests...
2022-05-18T04:22:37.3864862Z ----------------------------------------------------------------------
2022-05-18T04:22:38.9608084Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:38.9756498Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.589s)
2022-05-18T04:22:38.9757141Z 
2022-05-18T04:22:38.9757451Z ----------------------------------------------------------------------
2022-05-18T04:22:38.9757787Z Ran 1 test in 1.589s
2022-05-18T04:22:38.9757953Z 
2022-05-18T04:22:38.9758399Z OK (skipped=1)
2022-05-18T04:22:38.9758554Z 
2022-05-18T04:22:38.9758683Z Generating XML reports...
2022-05-18T04:22:38.9789999Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042237.xml
2022-05-18T04:22:40.3403473Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:22:40.3417660Z 
2022-05-18T04:22:40.3417940Z Running tests...
2022-05-18T04:22:40.3418394Z ----------------------------------------------------------------------
2022-05-18T04:22:41.9386640Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:41.9777573Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21938
2022-05-18T04:22:41.9879060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21939
2022-05-18T04:22:43.1487017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:43.1746911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:43.1748048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:43.1791293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:43.1797184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:43.2760728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:44.8957942Z ok (4.554s)
2022-05-18T04:22:44.8958595Z 
2022-05-18T04:22:44.8959006Z ----------------------------------------------------------------------
2022-05-18T04:22:44.8959465Z Ran 1 test in 4.554s
2022-05-18T04:22:44.8959772Z 
2022-05-18T04:22:44.8959890Z OK
2022-05-18T04:22:44.8960036Z 
2022-05-18T04:22:44.8960180Z Generating XML reports...
2022-05-18T04:22:44.9001011Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042240.xml
2022-05-18T04:22:46.3136157Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:22:46.3149955Z 
2022-05-18T04:22:46.3150256Z Running tests...
2022-05-18T04:22:46.3151017Z ----------------------------------------------------------------------
2022-05-18T04:22:46.3171811Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-05-18T04:22:46.3172298Z 
2022-05-18T04:22:46.3172604Z ----------------------------------------------------------------------
2022-05-18T04:22:46.3172941Z Ran 1 test in 0.002s
2022-05-18T04:22:46.3173111Z 
2022-05-18T04:22:46.3173463Z OK (skipped=1)
2022-05-18T04:22:46.3173631Z 
2022-05-18T04:22:46.3173761Z Generating XML reports...
2022-05-18T04:22:46.3208468Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042246.xml
2022-05-18T04:22:47.5731671Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:22:47.5745483Z 
2022-05-18T04:22:47.5745895Z Running tests...
2022-05-18T04:22:47.5746405Z ----------------------------------------------------------------------
2022-05-18T04:22:47.5764705Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:22:49.1622145Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:49.2012114Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22087
2022-05-18T04:22:49.2112859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22088
2022-05-18T04:22:50.3232056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:50.3437329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:50.3438682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:50.3535979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:50.3541872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:50.4451975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:51.6118394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq7k4c3ib
2022-05-18T04:22:51.6119226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq7k4c3ib/_remote_module_non_scriptable.py
2022-05-18T04:22:51.7162707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzmo7tfi
2022-05-18T04:22:51.7164197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzmo7tfi/_remote_module_non_scriptable.py
2022-05-18T04:22:52.2193152Z ok (4.644s)
2022-05-18T04:22:52.2193631Z 
2022-05-18T04:22:52.2194022Z ----------------------------------------------------------------------
2022-05-18T04:22:52.2194372Z Ran 1 test in 4.645s
2022-05-18T04:22:52.2194539Z 
2022-05-18T04:22:52.2194643Z OK
2022-05-18T04:22:52.2194779Z 
2022-05-18T04:22:52.2194915Z Generating XML reports...
2022-05-18T04:22:52.2236600Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042247.xml
2022-05-18T04:22:53.6015638Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:22:53.6029916Z 
2022-05-18T04:22:53.6030185Z Running tests...
2022-05-18T04:22:53.6030668Z ----------------------------------------------------------------------
2022-05-18T04:22:53.6052942Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:22:55.1728841Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:55.2118800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22207
2022-05-18T04:22:55.2220250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22208
2022-05-18T04:22:56.3842683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:56.3882162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:56.3882973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:56.3944468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:22:56.3950236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:56.4896997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:57.6371663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsl9_khea
2022-05-18T04:22:57.6372302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsl9_khea/_remote_module_non_scriptable.py
2022-05-18T04:22:57.7209310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvtzp80qw
2022-05-18T04:22:57.7210594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvtzp80qw/_remote_module_non_scriptable.py
2022-05-18T04:22:57.9745980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:57.9746505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:58.3301711Z ok (4.727s)
2022-05-18T04:22:58.3301931Z 
2022-05-18T04:22:58.3302307Z ----------------------------------------------------------------------
2022-05-18T04:22:58.3302631Z Ran 1 test in 4.727s
2022-05-18T04:22:58.3302795Z 
2022-05-18T04:22:58.3302890Z OK
2022-05-18T04:22:58.3303026Z 
2022-05-18T04:22:58.3303193Z Generating XML reports...
2022-05-18T04:22:58.3343930Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042253.xml
2022-05-18T04:22:59.7335448Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:22:59.7349378Z 
2022-05-18T04:22:59.7349620Z Running tests...
2022-05-18T04:22:59.7350062Z ----------------------------------------------------------------------
2022-05-18T04:22:59.7374029Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:01.2811688Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:01.3196371Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22327
2022-05-18T04:23:01.3300126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22328
2022-05-18T04:23:02.4773169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:02.5064216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:02.5065059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:02.5076960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:02.5083666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:02.6077847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:03.7842133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1g0xma3
2022-05-18T04:23:03.7843054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1g0xma3/_remote_module_non_scriptable.py
2022-05-18T04:23:03.8284508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnt1mcrwb
2022-05-18T04:23:03.8285869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnt1mcrwb/_remote_module_non_scriptable.py
2022-05-18T04:23:04.0848263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:04.0882789Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:04.4381249Z ok (4.703s)
2022-05-18T04:23:04.4381454Z 
2022-05-18T04:23:04.4381839Z ----------------------------------------------------------------------
2022-05-18T04:23:04.4382402Z Ran 1 test in 4.703s
2022-05-18T04:23:04.4382552Z 
2022-05-18T04:23:04.4382646Z OK
2022-05-18T04:23:04.4382786Z 
2022-05-18T04:23:04.4382918Z Generating XML reports...
2022-05-18T04:23:04.4423142Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042259.xml
2022-05-18T04:23:05.8374729Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:05.8388185Z 
2022-05-18T04:23:05.8388623Z Running tests...
2022-05-18T04:23:05.8389117Z ----------------------------------------------------------------------
2022-05-18T04:23:05.8406515Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:23:07.3606970Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:07.3993379Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22447
2022-05-18T04:23:07.4095376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22448
2022-05-18T04:23:08.5605093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:08.5898675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:08.5899522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:08.5909347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:08.5915164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:08.6913084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:09.8448093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48meg027
2022-05-18T04:23:09.8449350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48meg027/_remote_module_non_scriptable.py
2022-05-18T04:23:09.9596962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphyyga0tq
2022-05-18T04:23:09.9598701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphyyga0tq/_remote_module_non_scriptable.py
2022-05-18T04:23:10.5178218Z ok (4.679s)
2022-05-18T04:23:10.5178538Z 
2022-05-18T04:23:10.5179201Z ----------------------------------------------------------------------
2022-05-18T04:23:10.5179595Z Ran 1 test in 4.679s
2022-05-18T04:23:10.5179771Z 
2022-05-18T04:23:10.5179865Z OK
2022-05-18T04:23:10.5180001Z 
2022-05-18T04:23:10.5182160Z Generating XML reports...
2022-05-18T04:23:10.5221095Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042305.xml
2022-05-18T04:23:11.9505395Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:11.9519200Z 
2022-05-18T04:23:11.9519518Z Running tests...
2022-05-18T04:23:11.9520244Z ----------------------------------------------------------------------
2022-05-18T04:23:11.9538785Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:23:11.9539348Z 
2022-05-18T04:23:11.9539971Z ----------------------------------------------------------------------
2022-05-18T04:23:11.9540506Z Ran 1 test in 0.002s
2022-05-18T04:23:11.9540673Z 
2022-05-18T04:23:11.9540782Z OK (skipped=1)
2022-05-18T04:23:11.9540937Z 
2022-05-18T04:23:11.9541062Z Generating XML reports...
2022-05-18T04:23:11.9576197Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042311.xml
2022-05-18T04:23:13.2144980Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:13.2160117Z 
2022-05-18T04:23:13.2160307Z Running tests...
2022-05-18T04:23:13.2160746Z ----------------------------------------------------------------------
2022-05-18T04:23:13.2180146Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:23:13.2180694Z 
2022-05-18T04:23:13.2181149Z ----------------------------------------------------------------------
2022-05-18T04:23:13.2181553Z Ran 1 test in 0.002s
2022-05-18T04:23:13.2181855Z 
2022-05-18T04:23:13.2182048Z OK (skipped=1)
2022-05-18T04:23:13.2182264Z 
2022-05-18T04:23:13.2182377Z Generating XML reports...
2022-05-18T04:23:13.2218138Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042313.xml
2022-05-18T04:23:14.4770535Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:14.4784949Z 
2022-05-18T04:23:14.4785305Z Running tests...
2022-05-18T04:23:14.4786046Z ----------------------------------------------------------------------
2022-05-18T04:23:14.4805627Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:23:14.4805974Z 
2022-05-18T04:23:14.4806392Z ----------------------------------------------------------------------
2022-05-18T04:23:14.4806886Z Ran 1 test in 0.002s
2022-05-18T04:23:14.4807065Z 
2022-05-18T04:23:14.4807178Z OK (skipped=1)
2022-05-18T04:23:14.4807326Z 
2022-05-18T04:23:14.4807452Z Generating XML reports...
2022-05-18T04:23:14.4842454Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042314.xml
2022-05-18T04:23:15.7013847Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:15.7028602Z 
2022-05-18T04:23:15.7029060Z Running tests...
2022-05-18T04:23:15.7029764Z ----------------------------------------------------------------------
2022-05-18T04:23:15.7049408Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:23:15.7049776Z 
2022-05-18T04:23:15.7050110Z ----------------------------------------------------------------------
2022-05-18T04:23:15.7050682Z Ran 1 test in 0.002s
2022-05-18T04:23:15.7050850Z 
2022-05-18T04:23:15.7050944Z OK (skipped=1)
2022-05-18T04:23:15.7051349Z 
2022-05-18T04:23:15.7051500Z Generating XML reports...
2022-05-18T04:23:15.7086545Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042315.xml
2022-05-18T04:23:17.0150010Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:17.0164682Z 
2022-05-18T04:23:17.0165180Z Running tests...
2022-05-18T04:23:17.0165656Z ----------------------------------------------------------------------
2022-05-18T04:23:17.0185116Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-05-18T04:23:17.0185481Z 
2022-05-18T04:23:17.0185769Z ----------------------------------------------------------------------
2022-05-18T04:23:17.0186096Z Ran 1 test in 0.002s
2022-05-18T04:23:17.0186260Z 
2022-05-18T04:23:17.0186352Z OK (skipped=1)
2022-05-18T04:23:17.0186505Z 
2022-05-18T04:23:17.0186636Z Generating XML reports...
2022-05-18T04:23:17.0221996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042317.xml
2022-05-18T04:23:18.2811164Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:18.2825168Z 
2022-05-18T04:23:18.2825598Z Running tests...
2022-05-18T04:23:18.2826070Z ----------------------------------------------------------------------
2022-05-18T04:23:18.2855806Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s)
2022-05-18T04:23:18.2856153Z 
2022-05-18T04:23:18.2856788Z ----------------------------------------------------------------------
2022-05-18T04:23:18.2857120Z Ran 1 test in 0.003s
2022-05-18T04:23:18.2857268Z 
2022-05-18T04:23:18.2857381Z OK (skipped=1)
2022-05-18T04:23:18.2857534Z 
2022-05-18T04:23:18.2857667Z Generating XML reports...
2022-05-18T04:23:18.2892690Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042318.xml
2022-05-18T04:23:19.5478966Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:19.5493441Z 
2022-05-18T04:23:19.5493649Z Running tests...
2022-05-18T04:23:19.5494104Z ----------------------------------------------------------------------
2022-05-18T04:23:19.5513394Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:23:19.5513719Z 
2022-05-18T04:23:19.5513992Z ----------------------------------------------------------------------
2022-05-18T04:23:19.5514355Z Ran 1 test in 0.002s
2022-05-18T04:23:19.5514525Z 
2022-05-18T04:23:19.5514637Z OK (skipped=1)
2022-05-18T04:23:19.5514795Z 
2022-05-18T04:23:19.5514921Z Generating XML reports...
2022-05-18T04:23:19.5550633Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042319.xml
2022-05-18T04:23:20.8042495Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:20.8057218Z 
2022-05-18T04:23:20.8057659Z Running tests...
2022-05-18T04:23:20.8058251Z ----------------------------------------------------------------------
2022-05-18T04:23:22.3836375Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:22.4227385Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22798
2022-05-18T04:23:22.4330985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22799
2022-05-18T04:23:23.5661866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:23.5783372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:23.5784411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:23.5866192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:23.5872244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:23.6797812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:25.6414455Z ok (4.835s)
2022-05-18T04:23:25.6414802Z 
2022-05-18T04:23:25.6415571Z ----------------------------------------------------------------------
2022-05-18T04:23:25.6416229Z Ran 1 test in 4.836s
2022-05-18T04:23:25.6416429Z 
2022-05-18T04:23:25.6416530Z OK
2022-05-18T04:23:25.6416665Z 
2022-05-18T04:23:25.6416800Z Generating XML reports...
2022-05-18T04:23:25.6456749Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042320.xml
2022-05-18T04:23:27.0817442Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:27.0831294Z 
2022-05-18T04:23:27.0831617Z Running tests...
2022-05-18T04:23:27.0832565Z ----------------------------------------------------------------------
2022-05-18T04:23:28.6700207Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:28.7090638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22918
2022-05-18T04:23:28.7192329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22919
2022-05-18T04:23:29.8915185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:29.9127994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:29.9128838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:29.9216447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:29.9222905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:30.0142408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:32.0279015Z ok (4.944s)
2022-05-18T04:23:32.0279339Z 
2022-05-18T04:23:32.0279909Z ----------------------------------------------------------------------
2022-05-18T04:23:32.0280270Z Ran 1 test in 4.945s
2022-05-18T04:23:32.0280441Z 
2022-05-18T04:23:32.0280535Z OK
2022-05-18T04:23:32.0280674Z 
2022-05-18T04:23:32.0280815Z Generating XML reports...
2022-05-18T04:23:32.0320724Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042327.xml
2022-05-18T04:23:33.4462791Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:33.4478372Z 
2022-05-18T04:23:33.4478635Z Running tests...
2022-05-18T04:23:33.4479710Z ----------------------------------------------------------------------
2022-05-18T04:23:33.4499462Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:23:33.4499786Z 
2022-05-18T04:23:33.4500093Z ----------------------------------------------------------------------
2022-05-18T04:23:33.4500408Z Ran 1 test in 0.002s
2022-05-18T04:23:33.4500570Z 
2022-05-18T04:23:33.4500679Z OK (skipped=1)
2022-05-18T04:23:33.4500837Z 
2022-05-18T04:23:33.4500962Z Generating XML reports...
2022-05-18T04:23:33.4543703Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042333.xml
2022-05-18T04:23:34.7058654Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:34.7073640Z 
2022-05-18T04:23:34.7073946Z Running tests...
2022-05-18T04:23:34.7074702Z ----------------------------------------------------------------------
2022-05-18T04:23:34.7098483Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:23:34.7098880Z 
2022-05-18T04:23:34.7099406Z ----------------------------------------------------------------------
2022-05-18T04:23:34.7099755Z Ran 1 test in 0.002s
2022-05-18T04:23:34.7099921Z 
2022-05-18T04:23:34.7100036Z OK (skipped=1)
2022-05-18T04:23:34.7100192Z 
2022-05-18T04:23:34.7100301Z Generating XML reports...
2022-05-18T04:23:34.7142291Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042334.xml
2022-05-18T04:23:35.9582887Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:35.9598259Z 
2022-05-18T04:23:35.9598741Z Running tests...
2022-05-18T04:23:35.9599237Z ----------------------------------------------------------------------
2022-05-18T04:23:37.5539406Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:37.5928952Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23104
2022-05-18T04:23:37.6036095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23105
2022-05-18T04:23:38.7518659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:38.7878492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:38.7879590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:38.7924133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:38.7930413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:38.8892377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:40.9122044Z ok (4.952s)
2022-05-18T04:23:40.9122282Z 
2022-05-18T04:23:40.9122689Z ----------------------------------------------------------------------
2022-05-18T04:23:40.9123015Z Ran 1 test in 4.952s
2022-05-18T04:23:40.9123187Z 
2022-05-18T04:23:40.9123287Z OK
2022-05-18T04:23:40.9123425Z 
2022-05-18T04:23:40.9123564Z Generating XML reports...
2022-05-18T04:23:40.9164398Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042335.xml
2022-05-18T04:23:42.3266330Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:42.3280154Z 
2022-05-18T04:23:42.3280396Z Running tests...
2022-05-18T04:23:42.3280828Z ----------------------------------------------------------------------
2022-05-18T04:23:43.8886898Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:43.9268863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23224
2022-05-18T04:23:43.9370595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23225
2022-05-18T04:23:45.0694606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:45.1064063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:45.1064880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:45.1099979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:45.1106821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:45.2078464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:47.1456310Z ok (4.817s)
2022-05-18T04:23:47.1456538Z 
2022-05-18T04:23:47.1456911Z ----------------------------------------------------------------------
2022-05-18T04:23:47.1457262Z Ran 1 test in 4.818s
2022-05-18T04:23:47.1457432Z 
2022-05-18T04:23:47.1457529Z OK
2022-05-18T04:23:47.1457671Z 
2022-05-18T04:23:47.1457809Z Generating XML reports...
2022-05-18T04:23:47.1498265Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042342.xml
2022-05-18T04:23:48.5457893Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:48.5471491Z 
2022-05-18T04:23:48.5471660Z Running tests...
2022-05-18T04:23:48.5472382Z ----------------------------------------------------------------------
2022-05-18T04:23:50.0872179Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:50.1252880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23344
2022-05-18T04:23:50.1357921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23345
2022-05-18T04:23:51.2608710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:51.2732615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:51.2733426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:51.2811397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:51.2817780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:51.3747656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:54.0454502Z ok (5.498s)
2022-05-18T04:23:54.0454696Z 
2022-05-18T04:23:54.0455103Z ----------------------------------------------------------------------
2022-05-18T04:23:54.0455461Z Ran 1 test in 5.498s
2022-05-18T04:23:54.0455629Z 
2022-05-18T04:23:54.0455727Z OK
2022-05-18T04:23:54.0455867Z 
2022-05-18T04:23:54.0455986Z Generating XML reports...
2022-05-18T04:23:54.0497448Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042348.xml
2022-05-18T04:23:55.4786065Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:23:55.4800794Z 
2022-05-18T04:23:55.4801213Z Running tests...
2022-05-18T04:23:55.4801898Z ----------------------------------------------------------------------
2022-05-18T04:23:57.0511741Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:57.0905689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23461
2022-05-18T04:23:57.1007812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23462
2022-05-18T04:23:58.2910927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:58.2962162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:58.2962988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:58.3011926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:23:58.3018589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:58.3973092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:58.4083383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:23:58.4083938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:23:58.4084645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:23:58.4085356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:24:01.6915810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:24:01.6916390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:24:01.6917234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:24:01.6917960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:24:01.7313810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:24:01.7314366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:24:01.7315138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:24:01.7315843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:24:02.2141649Z ok (6.734s)
2022-05-18T04:24:02.2141884Z 
2022-05-18T04:24:02.2142539Z ----------------------------------------------------------------------
2022-05-18T04:24:02.2142894Z Ran 1 test in 6.734s
2022-05-18T04:24:02.2143061Z 
2022-05-18T04:24:02.2143158Z OK
2022-05-18T04:24:02.2143297Z 
2022-05-18T04:24:02.2143433Z Generating XML reports...
2022-05-18T04:24:02.2184701Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042355.xml
2022-05-18T04:24:03.6365519Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:03.6380490Z 
2022-05-18T04:24:03.6380644Z Running tests...
2022-05-18T04:24:03.6381360Z ----------------------------------------------------------------------
2022-05-18T04:24:03.6402215Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:24:03.6402570Z 
2022-05-18T04:24:03.6402843Z ----------------------------------------------------------------------
2022-05-18T04:24:03.6403188Z Ran 1 test in 0.002s
2022-05-18T04:24:03.6403361Z 
2022-05-18T04:24:03.6403478Z OK (skipped=1)
2022-05-18T04:24:03.6403637Z 
2022-05-18T04:24:03.6403770Z Generating XML reports...
2022-05-18T04:24:03.6439983Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042403.xml
2022-05-18T04:24:04.8979804Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:04.8995147Z 
2022-05-18T04:24:04.8995596Z Running tests...
2022-05-18T04:24:04.8996047Z ----------------------------------------------------------------------
2022-05-18T04:24:04.9016761Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:24:04.9017104Z 
2022-05-18T04:24:04.9017378Z ----------------------------------------------------------------------
2022-05-18T04:24:04.9017705Z Ran 1 test in 0.002s
2022-05-18T04:24:04.9017888Z 
2022-05-18T04:24:04.9018001Z OK (skipped=1)
2022-05-18T04:24:04.9018159Z 
2022-05-18T04:24:04.9018288Z Generating XML reports...
2022-05-18T04:24:04.9055871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042404.xml
2022-05-18T04:24:06.1615045Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:06.1629774Z 
2022-05-18T04:24:06.1630196Z Running tests...
2022-05-18T04:24:06.1630650Z ----------------------------------------------------------------------
2022-05-18T04:24:06.1651453Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:24:06.1651809Z 
2022-05-18T04:24:06.1652065Z ----------------------------------------------------------------------
2022-05-18T04:24:06.1652411Z Ran 1 test in 0.002s
2022-05-18T04:24:06.1652582Z 
2022-05-18T04:24:06.1652698Z OK (skipped=1)
2022-05-18T04:24:06.1652869Z 
2022-05-18T04:24:06.1652999Z Generating XML reports...
2022-05-18T04:24:06.1689332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042406.xml
2022-05-18T04:24:07.3883170Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:07.3898235Z 
2022-05-18T04:24:07.3898593Z Running tests...
2022-05-18T04:24:07.3899047Z ----------------------------------------------------------------------
2022-05-18T04:24:07.3919153Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:24:07.3919501Z 
2022-05-18T04:24:07.3919773Z ----------------------------------------------------------------------
2022-05-18T04:24:07.3920114Z Ran 1 test in 0.002s
2022-05-18T04:24:07.3920285Z 
2022-05-18T04:24:07.3920396Z OK (skipped=1)
2022-05-18T04:24:07.3920560Z 
2022-05-18T04:24:07.3921001Z Generating XML reports...
2022-05-18T04:24:07.3956833Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042407.xml
2022-05-18T04:24:08.6215868Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:08.6229213Z 
2022-05-18T04:24:08.6229510Z Running tests...
2022-05-18T04:24:08.6229943Z ----------------------------------------------------------------------
2022-05-18T04:24:08.6249402Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:24:08.6249747Z 
2022-05-18T04:24:08.6250023Z ----------------------------------------------------------------------
2022-05-18T04:24:08.6250359Z Ran 1 test in 0.002s
2022-05-18T04:24:08.6250528Z 
2022-05-18T04:24:08.6250622Z OK (skipped=1)
2022-05-18T04:24:08.6250781Z 
2022-05-18T04:24:08.6250910Z Generating XML reports...
2022-05-18T04:24:08.6285059Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042408.xml
2022-05-18T04:24:09.8518883Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:09.8532663Z 
2022-05-18T04:24:09.8533111Z Running tests...
2022-05-18T04:24:09.8533633Z ----------------------------------------------------------------------
2022-05-18T04:24:09.8552833Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:24:09.8553193Z 
2022-05-18T04:24:09.8553469Z ----------------------------------------------------------------------
2022-05-18T04:24:09.8553806Z Ran 1 test in 0.002s
2022-05-18T04:24:09.8553972Z 
2022-05-18T04:24:09.8554083Z OK (skipped=1)
2022-05-18T04:24:09.8554240Z 
2022-05-18T04:24:09.8554349Z Generating XML reports...
2022-05-18T04:24:09.8589280Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042409.xml
2022-05-18T04:24:11.1044756Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:11.1059773Z 
2022-05-18T04:24:11.1060076Z Running tests...
2022-05-18T04:24:11.1060508Z ----------------------------------------------------------------------
2022-05-18T04:24:11.1081066Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:24:11.1081435Z 
2022-05-18T04:24:11.1081717Z ----------------------------------------------------------------------
2022-05-18T04:24:11.1082107Z Ran 1 test in 0.002s
2022-05-18T04:24:11.1082274Z 
2022-05-18T04:24:11.1082370Z OK (skipped=1)
2022-05-18T04:24:11.1082537Z 
2022-05-18T04:24:11.1082668Z Generating XML reports...
2022-05-18T04:24:11.1119294Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042411.xml
2022-05-18T04:24:12.3659116Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:12.3674493Z 
2022-05-18T04:24:12.3674657Z Running tests...
2022-05-18T04:24:12.3675409Z ----------------------------------------------------------------------
2022-05-18T04:24:12.3696166Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:24:12.3696504Z 
2022-05-18T04:24:12.3696797Z ----------------------------------------------------------------------
2022-05-18T04:24:12.3697119Z Ran 1 test in 0.002s
2022-05-18T04:24:12.3697286Z 
2022-05-18T04:24:12.3697397Z OK (skipped=1)
2022-05-18T04:24:12.3697554Z 
2022-05-18T04:24:12.3697680Z Generating XML reports...
2022-05-18T04:24:12.3734465Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042412.xml
2022-05-18T04:24:13.6289905Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:13.6304990Z 
2022-05-18T04:24:13.6305402Z Running tests...
2022-05-18T04:24:13.6305899Z ----------------------------------------------------------------------
2022-05-18T04:24:13.6326126Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:24:13.6326466Z 
2022-05-18T04:24:13.6326743Z ----------------------------------------------------------------------
2022-05-18T04:24:13.6327061Z Ran 1 test in 0.002s
2022-05-18T04:24:13.6327230Z 
2022-05-18T04:24:13.6327345Z OK (skipped=1)
2022-05-18T04:24:13.6327509Z 
2022-05-18T04:24:13.6327635Z Generating XML reports...
2022-05-18T04:24:13.6364112Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042413.xml
2022-05-18T04:24:14.8837989Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:14.8853571Z 
2022-05-18T04:24:14.8853992Z Running tests...
2022-05-18T04:24:14.8854516Z ----------------------------------------------------------------------
2022-05-18T04:24:14.8875470Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:14.8875836Z 
2022-05-18T04:24:14.8876153Z ----------------------------------------------------------------------
2022-05-18T04:24:14.8876475Z Ran 1 test in 0.002s
2022-05-18T04:24:14.8876648Z 
2022-05-18T04:24:14.8876762Z OK (skipped=1)
2022-05-18T04:24:14.8876920Z 
2022-05-18T04:24:14.8877052Z Generating XML reports...
2022-05-18T04:24:14.8913147Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042414.xml
2022-05-18T04:24:16.1017678Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:16.1033062Z 
2022-05-18T04:24:16.1033473Z Running tests...
2022-05-18T04:24:16.1034000Z ----------------------------------------------------------------------
2022-05-18T04:24:16.1054388Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:24:16.1054900Z 
2022-05-18T04:24:16.1055771Z ----------------------------------------------------------------------
2022-05-18T04:24:16.1056247Z Ran 1 test in 0.002s
2022-05-18T04:24:16.1056419Z 
2022-05-18T04:24:16.1056546Z OK (skipped=1)
2022-05-18T04:24:16.1056704Z 
2022-05-18T04:24:16.1056836Z Generating XML reports...
2022-05-18T04:24:16.1093086Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042416.xml
2022-05-18T04:24:17.3262192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:17.3277102Z 
2022-05-18T04:24:17.3277262Z Running tests...
2022-05-18T04:24:17.3277956Z ----------------------------------------------------------------------
2022-05-18T04:24:17.3298762Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:24:17.3299100Z 
2022-05-18T04:24:17.3299388Z ----------------------------------------------------------------------
2022-05-18T04:24:17.3299706Z Ran 1 test in 0.002s
2022-05-18T04:24:17.3299894Z 
2022-05-18T04:24:17.3300012Z OK (skipped=1)
2022-05-18T04:24:17.3300172Z 
2022-05-18T04:24:17.3300301Z Generating XML reports...
2022-05-18T04:24:17.3336111Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042417.xml
2022-05-18T04:24:18.5544859Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:18.5559863Z 
2022-05-18T04:24:18.5560004Z Running tests...
2022-05-18T04:24:18.5560927Z ----------------------------------------------------------------------
2022-05-18T04:24:18.5581713Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:24:18.5582378Z 
2022-05-18T04:24:18.5582713Z ----------------------------------------------------------------------
2022-05-18T04:24:18.5583057Z Ran 1 test in 0.002s
2022-05-18T04:24:18.5583207Z 
2022-05-18T04:24:18.5583325Z OK (skipped=1)
2022-05-18T04:24:18.5583491Z 
2022-05-18T04:24:18.5583622Z Generating XML reports...
2022-05-18T04:24:18.5620071Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042418.xml
2022-05-18T04:24:19.8200105Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:19.8215460Z 
2022-05-18T04:24:19.8215757Z Running tests...
2022-05-18T04:24:19.8216209Z ----------------------------------------------------------------------
2022-05-18T04:24:19.8239522Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:19.8239870Z 
2022-05-18T04:24:19.8240161Z ----------------------------------------------------------------------
2022-05-18T04:24:19.8240502Z Ran 1 test in 0.002s
2022-05-18T04:24:19.8240672Z 
2022-05-18T04:24:19.8241004Z OK (skipped=1)
2022-05-18T04:24:19.8241146Z 
2022-05-18T04:24:19.8241590Z Generating XML reports...
2022-05-18T04:24:19.8277932Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042419.xml
2022-05-18T04:24:21.0743600Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:21.0763777Z 
2022-05-18T04:24:21.0764117Z Running tests...
2022-05-18T04:24:21.0764644Z ----------------------------------------------------------------------
2022-05-18T04:24:21.0784510Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:21.0784845Z 
2022-05-18T04:24:21.0785158Z ----------------------------------------------------------------------
2022-05-18T04:24:21.0785473Z Ran 1 test in 0.002s
2022-05-18T04:24:21.0785637Z 
2022-05-18T04:24:21.0785749Z OK (skipped=1)
2022-05-18T04:24:21.0785911Z 
2022-05-18T04:24:21.0786039Z Generating XML reports...
2022-05-18T04:24:21.0822808Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042421.xml
2022-05-18T04:24:22.2993317Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:22.3008500Z 
2022-05-18T04:24:22.3008797Z Running tests...
2022-05-18T04:24:22.3009249Z ----------------------------------------------------------------------
2022-05-18T04:24:22.3028913Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:22.3029232Z 
2022-05-18T04:24:22.3029523Z ----------------------------------------------------------------------
2022-05-18T04:24:22.3029862Z Ran 1 test in 0.002s
2022-05-18T04:24:22.3030031Z 
2022-05-18T04:24:22.3030151Z OK (skipped=1)
2022-05-18T04:24:22.3030314Z 
2022-05-18T04:24:22.3030443Z Generating XML reports...
2022-05-18T04:24:22.3067212Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042422.xml
2022-05-18T04:24:23.5617382Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:23.5632033Z 
2022-05-18T04:24:23.5632583Z Running tests...
2022-05-18T04:24:23.5633078Z ----------------------------------------------------------------------
2022-05-18T04:24:23.5655000Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:23.5655443Z 
2022-05-18T04:24:23.5655751Z ----------------------------------------------------------------------
2022-05-18T04:24:23.5656111Z Ran 1 test in 0.002s
2022-05-18T04:24:23.5656414Z 
2022-05-18T04:24:23.5656893Z OK (skipped=1)
2022-05-18T04:24:23.5657059Z 
2022-05-18T04:24:23.5657168Z Generating XML reports...
2022-05-18T04:24:23.5693191Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042423.xml
2022-05-18T04:24:24.8215847Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:24.8230714Z 
2022-05-18T04:24:24.8230869Z Running tests...
2022-05-18T04:24:24.8231502Z ----------------------------------------------------------------------
2022-05-18T04:24:24.8252412Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:24.8252742Z 
2022-05-18T04:24:24.8253028Z ----------------------------------------------------------------------
2022-05-18T04:24:24.8253364Z Ran 1 test in 0.002s
2022-05-18T04:24:24.8253529Z 
2022-05-18T04:24:24.8253640Z OK (skipped=1)
2022-05-18T04:24:24.8253783Z 
2022-05-18T04:24:24.8253923Z Generating XML reports...
2022-05-18T04:24:24.8290738Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042424.xml
2022-05-18T04:24:26.0781211Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:26.0796476Z 
2022-05-18T04:24:26.0796786Z Running tests...
2022-05-18T04:24:26.0797238Z ----------------------------------------------------------------------
2022-05-18T04:24:26.0817591Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:26.0817918Z 
2022-05-18T04:24:26.0818190Z ----------------------------------------------------------------------
2022-05-18T04:24:26.0818526Z Ran 1 test in 0.002s
2022-05-18T04:24:26.0818692Z 
2022-05-18T04:24:26.0818806Z OK (skipped=1)
2022-05-18T04:24:26.0818964Z 
2022-05-18T04:24:26.0819090Z Generating XML reports...
2022-05-18T04:24:26.0855878Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042426.xml
2022-05-18T04:24:27.3340920Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:27.3357001Z 
2022-05-18T04:24:27.3357327Z Running tests...
2022-05-18T04:24:27.3358367Z ----------------------------------------------------------------------
2022-05-18T04:24:27.3377315Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:27.3378443Z 
2022-05-18T04:24:27.3379116Z ----------------------------------------------------------------------
2022-05-18T04:24:27.3379477Z Ran 1 test in 0.002s
2022-05-18T04:24:27.3379648Z 
2022-05-18T04:24:27.3379743Z OK (skipped=1)
2022-05-18T04:24:27.3379903Z 
2022-05-18T04:24:27.3380034Z Generating XML reports...
2022-05-18T04:24:27.3416373Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042427.xml
2022-05-18T04:24:28.5947213Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:28.5962133Z 
2022-05-18T04:24:28.5962280Z Running tests...
2022-05-18T04:24:28.5963125Z ----------------------------------------------------------------------
2022-05-18T04:24:28.5984525Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:28.5984868Z 
2022-05-18T04:24:28.5985346Z ----------------------------------------------------------------------
2022-05-18T04:24:28.5985774Z Ran 1 test in 0.002s
2022-05-18T04:24:28.5985943Z 
2022-05-18T04:24:28.5986039Z OK (skipped=1)
2022-05-18T04:24:28.5986199Z 
2022-05-18T04:24:28.5986327Z Generating XML reports...
2022-05-18T04:24:28.6023047Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042428.xml
2022-05-18T04:24:29.8548653Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:29.8564281Z 
2022-05-18T04:24:29.8564728Z Running tests...
2022-05-18T04:24:29.8565221Z ----------------------------------------------------------------------
2022-05-18T04:24:29.8585542Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:29.8586041Z 
2022-05-18T04:24:29.8586352Z ----------------------------------------------------------------------
2022-05-18T04:24:29.8586690Z Ran 1 test in 0.002s
2022-05-18T04:24:29.8586839Z 
2022-05-18T04:24:29.8586964Z OK (skipped=1)
2022-05-18T04:24:29.8587122Z 
2022-05-18T04:24:29.8587252Z Generating XML reports...
2022-05-18T04:24:29.8623962Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042429.xml
2022-05-18T04:24:31.1163128Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:31.1178173Z 
2022-05-18T04:24:31.1178328Z Running tests...
2022-05-18T04:24:31.1178779Z ----------------------------------------------------------------------
2022-05-18T04:24:31.1199473Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:31.1200033Z 
2022-05-18T04:24:31.1200342Z ----------------------------------------------------------------------
2022-05-18T04:24:31.1200674Z Ran 1 test in 0.002s
2022-05-18T04:24:31.1200840Z 
2022-05-18T04:24:31.1200953Z OK (skipped=1)
2022-05-18T04:24:31.1201114Z 
2022-05-18T04:24:31.1201242Z Generating XML reports...
2022-05-18T04:24:31.1237579Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042431.xml
2022-05-18T04:24:32.3761944Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:32.3777114Z 
2022-05-18T04:24:32.3777267Z Running tests...
2022-05-18T04:24:32.3778026Z ----------------------------------------------------------------------
2022-05-18T04:24:32.3798347Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:32.3799029Z 
2022-05-18T04:24:32.3799336Z ----------------------------------------------------------------------
2022-05-18T04:24:32.3799960Z Ran 1 test in 0.002s
2022-05-18T04:24:32.3800129Z 
2022-05-18T04:24:32.3800247Z OK (skipped=1)
2022-05-18T04:24:32.3800406Z 
2022-05-18T04:24:32.3800537Z Generating XML reports...
2022-05-18T04:24:32.3836093Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042432.xml
2022-05-18T04:24:33.6372067Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:33.6387305Z 
2022-05-18T04:24:33.6387753Z Running tests...
2022-05-18T04:24:33.6388227Z ----------------------------------------------------------------------
2022-05-18T04:24:33.6411145Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-05-18T04:24:33.6411491Z 
2022-05-18T04:24:33.6411781Z ----------------------------------------------------------------------
2022-05-18T04:24:33.6412118Z Ran 1 test in 0.002s
2022-05-18T04:24:33.6412297Z 
2022-05-18T04:24:33.6412398Z OK (skipped=1)
2022-05-18T04:24:33.6412563Z 
2022-05-18T04:24:33.6412690Z Generating XML reports...
2022-05-18T04:24:33.6449399Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042433.xml
2022-05-18T04:24:34.8689540Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:34.8703069Z 
2022-05-18T04:24:34.8703345Z Running tests...
2022-05-18T04:24:34.8703809Z ----------------------------------------------------------------------
2022-05-18T04:24:34.8726731Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-05-18T04:24:34.8727421Z 
2022-05-18T04:24:34.8727741Z ----------------------------------------------------------------------
2022-05-18T04:24:34.8728104Z Ran 1 test in 0.002s
2022-05-18T04:24:34.8728281Z 
2022-05-18T04:24:34.8728397Z OK (skipped=1)
2022-05-18T04:24:34.8728545Z 
2022-05-18T04:24:34.8728688Z Generating XML reports...
2022-05-18T04:24:34.8762266Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042434.xml
2022-05-18T04:24:36.1196371Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:36.1212239Z 
2022-05-18T04:24:36.1212402Z Running tests...
2022-05-18T04:24:36.1212839Z ----------------------------------------------------------------------
2022-05-18T04:24:36.1234418Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:36.1234752Z 
2022-05-18T04:24:36.1235069Z ----------------------------------------------------------------------
2022-05-18T04:24:36.1235408Z Ran 1 test in 0.002s
2022-05-18T04:24:36.1235554Z 
2022-05-18T04:24:36.1235668Z OK (skipped=1)
2022-05-18T04:24:36.1235825Z 
2022-05-18T04:24:36.1235953Z Generating XML reports...
2022-05-18T04:24:36.1272501Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042436.xml
2022-05-18T04:24:37.3749120Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:37.3763466Z 
2022-05-18T04:24:37.3763965Z Running tests...
2022-05-18T04:24:37.3764491Z ----------------------------------------------------------------------
2022-05-18T04:24:38.9576955Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:38.9967224Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24509
2022-05-18T04:24:39.0070245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24510
2022-05-18T04:24:40.1724769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:40.1733086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:40.1734086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:40.1826013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:40.1832126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:40.2747990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:41.9149215Z ok (4.538s)
2022-05-18T04:24:41.9149460Z 
2022-05-18T04:24:41.9149863Z ----------------------------------------------------------------------
2022-05-18T04:24:41.9150231Z Ran 1 test in 4.539s
2022-05-18T04:24:41.9150404Z 
2022-05-18T04:24:41.9150506Z OK
2022-05-18T04:24:41.9150625Z 
2022-05-18T04:24:41.9150771Z Generating XML reports...
2022-05-18T04:24:41.9191434Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042437.xml
2022-05-18T04:24:43.3324808Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:43.3338819Z 
2022-05-18T04:24:43.3339126Z Running tests...
2022-05-18T04:24:43.3339566Z ----------------------------------------------------------------------
2022-05-18T04:24:43.3360116Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:43.3360690Z 
2022-05-18T04:24:43.3361037Z ----------------------------------------------------------------------
2022-05-18T04:24:43.3361378Z Ran 1 test in 0.002s
2022-05-18T04:24:43.3361840Z 
2022-05-18T04:24:43.3361936Z OK (skipped=1)
2022-05-18T04:24:43.3362094Z 
2022-05-18T04:24:43.3362222Z Generating XML reports...
2022-05-18T04:24:43.3397088Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042443.xml
2022-05-18T04:24:44.5954676Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:44.5969342Z 
2022-05-18T04:24:44.5969752Z Running tests...
2022-05-18T04:24:44.5970211Z ----------------------------------------------------------------------
2022-05-18T04:24:44.5990990Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:44.5991561Z 
2022-05-18T04:24:44.5991909Z ----------------------------------------------------------------------
2022-05-18T04:24:44.5992231Z Ran 1 test in 0.002s
2022-05-18T04:24:44.5992397Z 
2022-05-18T04:24:44.5992585Z OK (skipped=1)
2022-05-18T04:24:44.5992880Z 
2022-05-18T04:24:44.5993125Z Generating XML reports...
2022-05-18T04:24:44.6028770Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042444.xml
2022-05-18T04:24:45.8742604Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:45.8757350Z 
2022-05-18T04:24:45.8757707Z Running tests...
2022-05-18T04:24:45.8758419Z ----------------------------------------------------------------------
2022-05-18T04:24:45.8779815Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:24:45.8780641Z 
2022-05-18T04:24:45.8780957Z ----------------------------------------------------------------------
2022-05-18T04:24:45.8781292Z Ran 1 test in 0.002s
2022-05-18T04:24:45.8781460Z 
2022-05-18T04:24:45.8781577Z OK (skipped=1)
2022-05-18T04:24:45.8781736Z 
2022-05-18T04:24:45.8781869Z Generating XML reports...
2022-05-18T04:24:45.8817183Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042445.xml
2022-05-18T04:24:47.1368022Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:47.1382341Z 
2022-05-18T04:24:47.1382763Z Running tests...
2022-05-18T04:24:47.1383506Z ----------------------------------------------------------------------
2022-05-18T04:24:48.7346891Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:48.7727654Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24724
2022-05-18T04:24:48.7829042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24725
2022-05-18T04:24:49.9198926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:49.9490324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:49.9491162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:49.9502856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:49.9509144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:50.0505482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:51.9911898Z ok (4.853s)
2022-05-18T04:24:51.9912207Z 
2022-05-18T04:24:51.9912619Z ----------------------------------------------------------------------
2022-05-18T04:24:51.9912950Z Ran 1 test in 4.853s
2022-05-18T04:24:51.9913120Z 
2022-05-18T04:24:51.9913219Z OK
2022-05-18T04:24:51.9913356Z 
2022-05-18T04:24:51.9913491Z Generating XML reports...
2022-05-18T04:24:51.9953810Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042447.xml
2022-05-18T04:24:53.4066024Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:53.4080257Z 
2022-05-18T04:24:53.4080787Z Running tests...
2022-05-18T04:24:53.4081301Z ----------------------------------------------------------------------
2022-05-18T04:24:54.9863955Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:55.0255798Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24844
2022-05-18T04:24:55.0359465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24845
2022-05-18T04:24:56.1525278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:56.1670235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:56.1671267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:56.1728423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:56.1734505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:56.2684808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:58.2443767Z ok (4.836s)
2022-05-18T04:24:58.2444152Z 
2022-05-18T04:24:58.2444708Z ----------------------------------------------------------------------
2022-05-18T04:24:58.2445053Z Ran 1 test in 4.836s
2022-05-18T04:24:58.2445222Z 
2022-05-18T04:24:58.2445302Z OK
2022-05-18T04:24:58.2445436Z 
2022-05-18T04:24:58.2445575Z Generating XML reports...
2022-05-18T04:24:58.2485396Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042453.xml
2022-05-18T04:24:59.6683746Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:24:59.6697882Z 
2022-05-18T04:24:59.6698443Z Running tests...
2022-05-18T04:24:59.6698949Z ----------------------------------------------------------------------
2022-05-18T04:25:01.2548141Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:01.2938420Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24964
2022-05-18T04:25:01.3042080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24965
2022-05-18T04:25:02.4537514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:02.4798522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:02.4799619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:02.4841492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:02.4848421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:02.5812762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:04.5126373Z ok (4.842s)
2022-05-18T04:25:04.5126674Z 
2022-05-18T04:25:04.5127260Z ----------------------------------------------------------------------
2022-05-18T04:25:04.5127602Z Ran 1 test in 4.843s
2022-05-18T04:25:04.5127769Z 
2022-05-18T04:25:04.5127865Z OK
2022-05-18T04:25:04.5128010Z 
2022-05-18T04:25:04.5128146Z Generating XML reports...
2022-05-18T04:25:04.5168319Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042459.xml
2022-05-18T04:25:05.9392127Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:05.9408615Z 
2022-05-18T04:25:05.9409033Z Running tests...
2022-05-18T04:25:05.9409529Z ----------------------------------------------------------------------
2022-05-18T04:25:05.9429047Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:25:05.9429753Z 
2022-05-18T04:25:05.9430075Z ----------------------------------------------------------------------
2022-05-18T04:25:05.9430428Z Ran 1 test in 0.002s
2022-05-18T04:25:05.9430598Z 
2022-05-18T04:25:05.9430712Z OK (skipped=1)
2022-05-18T04:25:05.9430851Z 
2022-05-18T04:25:05.9430980Z Generating XML reports...
2022-05-18T04:25:05.9468205Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042505.xml
2022-05-18T04:25:07.2043038Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:07.2058240Z 
2022-05-18T04:25:07.2058408Z Running tests...
2022-05-18T04:25:07.2058871Z ----------------------------------------------------------------------
2022-05-18T04:25:07.2078788Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:25:07.2079106Z 
2022-05-18T04:25:07.2079408Z ----------------------------------------------------------------------
2022-05-18T04:25:07.2079747Z Ran 1 test in 0.002s
2022-05-18T04:25:07.2079896Z 
2022-05-18T04:25:07.2080011Z OK (skipped=1)
2022-05-18T04:25:07.2080168Z 
2022-05-18T04:25:07.2080297Z Generating XML reports...
2022-05-18T04:25:07.2116738Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042507.xml
2022-05-18T04:25:08.4693582Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:08.4708779Z 
2022-05-18T04:25:08.4709091Z Running tests...
2022-05-18T04:25:08.4709528Z ----------------------------------------------------------------------
2022-05-18T04:25:10.0459506Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:10.0844873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25150
2022-05-18T04:25:10.0945747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25151
2022-05-18T04:25:11.2622600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:11.2884976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:11.2885777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:11.2926340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:11.2932333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:11.3898878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:13.0024064Z ok (4.531s)
2022-05-18T04:25:13.0024398Z 
2022-05-18T04:25:13.0024802Z ----------------------------------------------------------------------
2022-05-18T04:25:13.0025131Z Ran 1 test in 4.531s
2022-05-18T04:25:13.0025323Z 
2022-05-18T04:25:13.0025420Z OK
2022-05-18T04:25:13.0025558Z 
2022-05-18T04:25:13.0025693Z Generating XML reports...
2022-05-18T04:25:13.0066597Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042508.xml
2022-05-18T04:25:14.4184780Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:14.4199239Z 
2022-05-18T04:25:14.4199510Z Running tests...
2022-05-18T04:25:14.4200154Z ----------------------------------------------------------------------
2022-05-18T04:25:15.9931398Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:16.0313869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25266
2022-05-18T04:25:16.0417429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25267
2022-05-18T04:25:17.1871497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:17.2154100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:17.2154917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:17.2175312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:17.2181162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:17.3168118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:18.9494674Z ok (4.529s)
2022-05-18T04:25:18.9494935Z 
2022-05-18T04:25:18.9495350Z ----------------------------------------------------------------------
2022-05-18T04:25:18.9495700Z Ran 1 test in 4.530s
2022-05-18T04:25:18.9495869Z 
2022-05-18T04:25:18.9495968Z OK
2022-05-18T04:25:18.9496129Z 
2022-05-18T04:25:18.9496250Z Generating XML reports...
2022-05-18T04:25:18.9537253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042514.xml
2022-05-18T04:25:20.3756040Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:20.3770805Z 
2022-05-18T04:25:20.3771159Z Running tests...
2022-05-18T04:25:20.3771944Z ----------------------------------------------------------------------
2022-05-18T04:25:20.3790296Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:25:20.3790880Z 
2022-05-18T04:25:20.3791499Z ----------------------------------------------------------------------
2022-05-18T04:25:20.3792194Z Ran 1 test in 0.002s
2022-05-18T04:25:20.3792391Z 
2022-05-18T04:25:20.3792504Z OK (skipped=1)
2022-05-18T04:25:20.3792662Z 
2022-05-18T04:25:20.3792799Z Generating XML reports...
2022-05-18T04:25:20.3828882Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042520.xml
2022-05-18T04:25:21.6299809Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:21.6314188Z 
2022-05-18T04:25:21.6314643Z Running tests...
2022-05-18T04:25:21.6315144Z ----------------------------------------------------------------------
2022-05-18T04:25:23.2058118Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:23.2449592Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25415
2022-05-18T04:25:23.2552233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25416
2022-05-18T04:25:24.3733483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:24.3758242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:24.3759333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:24.3834977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:24.3840981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:24.3843918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:25:24.4769116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:24.4772439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:25:24.4773143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:25:24.4861779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:25:26.0629246Z ok (4.431s)
2022-05-18T04:25:26.0629469Z 
2022-05-18T04:25:26.0629880Z ----------------------------------------------------------------------
2022-05-18T04:25:26.0630207Z Ran 1 test in 4.431s
2022-05-18T04:25:26.0630371Z 
2022-05-18T04:25:26.0630472Z OK
2022-05-18T04:25:26.0630610Z 
2022-05-18T04:25:26.0630745Z Generating XML reports...
2022-05-18T04:25:26.0671780Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042521.xml
2022-05-18T04:25:27.4806144Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:27.4820580Z 
2022-05-18T04:25:27.4820879Z Running tests...
2022-05-18T04:25:27.4821323Z ----------------------------------------------------------------------
2022-05-18T04:25:27.4840445Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:25:27.4840967Z 
2022-05-18T04:25:27.4841430Z ----------------------------------------------------------------------
2022-05-18T04:25:27.4841933Z Ran 1 test in 0.002s
2022-05-18T04:25:27.4842099Z 
2022-05-18T04:25:27.4842223Z OK (skipped=1)
2022-05-18T04:25:27.4842381Z 
2022-05-18T04:25:27.4842510Z Generating XML reports...
2022-05-18T04:25:27.4877388Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042527.xml
2022-05-18T04:25:28.6969357Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:28.6983403Z 
2022-05-18T04:25:28.6983626Z Running tests...
2022-05-18T04:25:28.6984069Z ----------------------------------------------------------------------
2022-05-18T04:25:30.2736576Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:30.3127950Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25568
2022-05-18T04:25:30.3231989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25569
2022-05-18T04:25:31.4663816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:31.4717136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:31.4717937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:31.4765193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:31.4771777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:31.5731574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:31.7282320Z skip: Skipped due to small world size. (3.029s)
2022-05-18T04:25:31.7282605Z 
2022-05-18T04:25:31.7282993Z ----------------------------------------------------------------------
2022-05-18T04:25:31.7283338Z Ran 1 test in 3.030s
2022-05-18T04:25:31.7283500Z 
2022-05-18T04:25:31.7283611Z OK (skipped=1)
2022-05-18T04:25:31.7283774Z 
2022-05-18T04:25:31.7283885Z Generating XML reports...
2022-05-18T04:25:31.7324799Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042528.xml
2022-05-18T04:25:33.1304540Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:33.1318602Z 
2022-05-18T04:25:33.1318885Z Running tests...
2022-05-18T04:25:33.1319327Z ----------------------------------------------------------------------
2022-05-18T04:25:33.1338747Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:25:33.1339096Z 
2022-05-18T04:25:33.1339422Z ----------------------------------------------------------------------
2022-05-18T04:25:33.1339747Z Ran 1 test in 0.002s
2022-05-18T04:25:33.1339914Z 
2022-05-18T04:25:33.1340026Z OK (skipped=1)
2022-05-18T04:25:33.1340183Z 
2022-05-18T04:25:33.1340312Z Generating XML reports...
2022-05-18T04:25:33.1375907Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042533.xml
2022-05-18T04:25:34.4012448Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:34.4026751Z 
2022-05-18T04:25:34.4027389Z Running tests...
2022-05-18T04:25:34.4027925Z ----------------------------------------------------------------------
2022-05-18T04:25:34.4046826Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:25:34.4047168Z 
2022-05-18T04:25:34.4047463Z ----------------------------------------------------------------------
2022-05-18T04:25:34.4047792Z Ran 1 test in 0.002s
2022-05-18T04:25:34.4047958Z 
2022-05-18T04:25:34.4048072Z OK (skipped=1)
2022-05-18T04:25:34.4048230Z 
2022-05-18T04:25:34.4048360Z Generating XML reports...
2022-05-18T04:25:34.4084973Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042534.xml
2022-05-18T04:25:35.6617075Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:35.6630814Z 
2022-05-18T04:25:35.6631123Z Running tests...
2022-05-18T04:25:35.6631571Z ----------------------------------------------------------------------
2022-05-18T04:25:37.2395347Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:37.2777607Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25741
2022-05-18T04:25:37.2879686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25742
2022-05-18T04:25:38.4496472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:38.4608522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:38.4609325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:38.4699520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:38.4707236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:38.5622376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:40.4962925Z ok (4.833s)
2022-05-18T04:25:40.4963159Z 
2022-05-18T04:25:40.4963770Z ----------------------------------------------------------------------
2022-05-18T04:25:40.4964244Z Ran 1 test in 4.833s
2022-05-18T04:25:40.4964397Z 
2022-05-18T04:25:40.4964500Z OK
2022-05-18T04:25:40.4964647Z 
2022-05-18T04:25:40.4964786Z Generating XML reports...
2022-05-18T04:25:40.5005083Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042535.xml
2022-05-18T04:25:41.9041427Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:41.9055657Z 
2022-05-18T04:25:41.9055901Z Running tests...
2022-05-18T04:25:41.9056350Z ----------------------------------------------------------------------
2022-05-18T04:25:43.4408346Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:43.4796175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25861
2022-05-18T04:25:43.4905754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25862
2022-05-18T04:25:44.6063584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:44.6285273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:44.6286081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:44.6369358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:44.6375517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:44.7301197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:46.7987716Z ok (4.893s)
2022-05-18T04:25:46.7987942Z 
2022-05-18T04:25:46.7988352Z ----------------------------------------------------------------------
2022-05-18T04:25:46.7988685Z Ran 1 test in 4.893s
2022-05-18T04:25:46.7988858Z 
2022-05-18T04:25:46.7988974Z OK
2022-05-18T04:25:46.7989124Z 
2022-05-18T04:25:46.7989259Z Generating XML reports...
2022-05-18T04:25:46.8030948Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042541.xml
2022-05-18T04:25:48.2183405Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:48.2197663Z 
2022-05-18T04:25:48.2198270Z Running tests...
2022-05-18T04:25:48.2198803Z ----------------------------------------------------------------------
2022-05-18T04:25:48.2217601Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:25:48.2217977Z 
2022-05-18T04:25:48.2218273Z ----------------------------------------------------------------------
2022-05-18T04:25:48.2218592Z Ran 1 test in 0.002s
2022-05-18T04:25:48.2218757Z 
2022-05-18T04:25:48.2218869Z OK (skipped=1)
2022-05-18T04:25:48.2219031Z 
2022-05-18T04:25:48.2219444Z Generating XML reports...
2022-05-18T04:25:48.2254386Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042548.xml
2022-05-18T04:25:49.4705659Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:49.4719687Z 
2022-05-18T04:25:49.4719961Z Running tests...
2022-05-18T04:25:49.4720408Z ----------------------------------------------------------------------
2022-05-18T04:25:51.0446618Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:51.0839364Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26014
2022-05-18T04:25:51.0942389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26015
2022-05-18T04:25:52.2393829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:52.2559793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:52.2560594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:52.2596240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:52.2602407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:52.2605525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:25:52.3570440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:52.3573479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:25:52.3574212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:25:52.3621385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:25:54.5029204Z ok (5.030s)
2022-05-18T04:25:54.5031010Z 
2022-05-18T04:25:54.5031521Z ----------------------------------------------------------------------
2022-05-18T04:25:54.5031862Z Ran 1 test in 5.031s
2022-05-18T04:25:54.5032036Z 
2022-05-18T04:25:54.5032134Z OK
2022-05-18T04:25:54.5032270Z 
2022-05-18T04:25:54.5032408Z Generating XML reports...
2022-05-18T04:25:54.5071051Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042549.xml
2022-05-18T04:25:55.9193990Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:55.9208837Z 
2022-05-18T04:25:55.9209172Z Running tests...
2022-05-18T04:25:55.9209612Z ----------------------------------------------------------------------
2022-05-18T04:25:55.9228249Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:25:55.9228595Z 
2022-05-18T04:25:55.9228882Z ----------------------------------------------------------------------
2022-05-18T04:25:55.9229197Z Ran 1 test in 0.002s
2022-05-18T04:25:55.9229361Z 
2022-05-18T04:25:55.9229473Z OK (skipped=1)
2022-05-18T04:25:55.9229633Z 
2022-05-18T04:25:55.9229761Z Generating XML reports...
2022-05-18T04:25:55.9265747Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042555.xml
2022-05-18T04:25:57.1611132Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:25:57.1625363Z 
2022-05-18T04:25:57.1625830Z Running tests...
2022-05-18T04:25:57.1626330Z ----------------------------------------------------------------------
2022-05-18T04:25:58.6938348Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:58.7322735Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26171
2022-05-18T04:25:58.7426810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26172
2022-05-18T04:25:59.9231994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:59.9864174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:59.9865279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:59.9940554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:25:59.9946988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:00.0878822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:00.2478866Z skip: Skipped due to small world size. (3.085s)
2022-05-18T04:26:00.2479179Z 
2022-05-18T04:26:00.2479581Z ----------------------------------------------------------------------
2022-05-18T04:26:00.2480178Z Ran 1 test in 3.085s
2022-05-18T04:26:00.2480350Z 
2022-05-18T04:26:00.2480462Z OK (skipped=1)
2022-05-18T04:26:00.2480602Z 
2022-05-18T04:26:00.2480730Z Generating XML reports...
2022-05-18T04:26:00.2521661Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042557.xml
2022-05-18T04:26:01.6514324Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:01.6529486Z 
2022-05-18T04:26:01.6529810Z Running tests...
2022-05-18T04:26:01.6530259Z ----------------------------------------------------------------------
2022-05-18T04:26:01.6548826Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:26:01.6549161Z 
2022-05-18T04:26:01.6549631Z ----------------------------------------------------------------------
2022-05-18T04:26:01.6550230Z Ran 1 test in 0.002s
2022-05-18T04:26:01.6550439Z 
2022-05-18T04:26:01.6550562Z OK (skipped=1)
2022-05-18T04:26:01.6550720Z 
2022-05-18T04:26:01.6550848Z Generating XML reports...
2022-05-18T04:26:01.6585869Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042601.xml
2022-05-18T04:26:02.9204425Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:02.9218749Z 
2022-05-18T04:26:02.9219121Z Running tests...
2022-05-18T04:26:02.9219570Z ----------------------------------------------------------------------
2022-05-18T04:26:02.9238589Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:26:02.9238932Z 
2022-05-18T04:26:02.9239577Z ----------------------------------------------------------------------
2022-05-18T04:26:02.9239955Z Ran 1 test in 0.002s
2022-05-18T04:26:02.9240123Z 
2022-05-18T04:26:02.9240238Z OK (skipped=1)
2022-05-18T04:26:02.9240396Z 
2022-05-18T04:26:02.9240507Z Generating XML reports...
2022-05-18T04:26:02.9276528Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042602.xml
2022-05-18T04:26:04.1883006Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:04.1897299Z 
2022-05-18T04:26:04.1897638Z Running tests...
2022-05-18T04:26:04.1898069Z ----------------------------------------------------------------------
2022-05-18T04:26:05.7852623Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:05.8244488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26344
2022-05-18T04:26:05.8345520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26345
2022-05-18T04:26:07.0201478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:07.0225498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:07.0226303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:07.0302789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:07.0308719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:07.1240585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:08.7435049Z ok (4.553s)
2022-05-18T04:26:08.7435282Z 
2022-05-18T04:26:08.7435701Z ----------------------------------------------------------------------
2022-05-18T04:26:08.7436048Z Ran 1 test in 4.554s
2022-05-18T04:26:08.7436224Z 
2022-05-18T04:26:08.7436303Z OK
2022-05-18T04:26:08.7436439Z 
2022-05-18T04:26:08.7436575Z Generating XML reports...
2022-05-18T04:26:08.7478649Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042604.xml
2022-05-18T04:26:10.1748047Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:10.1762933Z 
2022-05-18T04:26:10.1763088Z Running tests...
2022-05-18T04:26:10.1763809Z ----------------------------------------------------------------------
2022-05-18T04:26:11.7481600Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:11.7874287Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26460
2022-05-18T04:26:11.7977435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26461
2022-05-18T04:26:12.9351383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:12.9706301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:12.9707114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:12.9756509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:12.9763297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:13.0720639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:14.7057158Z ok (4.529s)
2022-05-18T04:26:14.7057385Z 
2022-05-18T04:26:14.7057812Z ----------------------------------------------------------------------
2022-05-18T04:26:14.7058155Z Ran 1 test in 4.529s
2022-05-18T04:26:14.7058320Z 
2022-05-18T04:26:14.7058399Z OK
2022-05-18T04:26:14.7058534Z 
2022-05-18T04:26:14.7058668Z Generating XML reports...
2022-05-18T04:26:14.7098882Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042610.xml
2022-05-18T04:26:16.1072042Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:16.1086883Z 
2022-05-18T04:26:16.1087301Z Running tests...
2022-05-18T04:26:16.1087789Z ----------------------------------------------------------------------
2022-05-18T04:26:16.1106205Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:26:16.1106830Z 
2022-05-18T04:26:16.1107183Z ----------------------------------------------------------------------
2022-05-18T04:26:16.1107823Z Ran 1 test in 0.002s
2022-05-18T04:26:16.1107991Z 
2022-05-18T04:26:16.1108109Z OK (skipped=1)
2022-05-18T04:26:16.1108266Z 
2022-05-18T04:26:16.1108395Z Generating XML reports...
2022-05-18T04:26:16.1142808Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042616.xml
2022-05-18T04:26:17.3675425Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:17.3690472Z 
2022-05-18T04:26:17.3690773Z Running tests...
2022-05-18T04:26:17.3691222Z ----------------------------------------------------------------------
2022-05-18T04:26:18.9678879Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:19.0070708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26609
2022-05-18T04:26:19.0173502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26610
2022-05-18T04:26:20.1686004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:20.1788261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:20.1789054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:20.1888618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:20.1894512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:20.1897742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:26:20.2799266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:20.2803262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:26:20.2803972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:26:20.2809324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:26:21.9251186Z ok (4.556s)
2022-05-18T04:26:21.9254583Z 
2022-05-18T04:26:21.9255447Z ----------------------------------------------------------------------
2022-05-18T04:26:21.9256084Z Ran 1 test in 4.556s
2022-05-18T04:26:21.9256377Z 
2022-05-18T04:26:21.9256535Z OK
2022-05-18T04:26:21.9256789Z 
2022-05-18T04:26:21.9257023Z Generating XML reports...
2022-05-18T04:26:21.9296835Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042617.xml
2022-05-18T04:26:23.3473266Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:23.3488287Z 
2022-05-18T04:26:23.3488826Z Running tests...
2022-05-18T04:26:23.3489522Z ----------------------------------------------------------------------
2022-05-18T04:26:23.3508928Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:26:23.3509286Z 
2022-05-18T04:26:23.3509815Z ----------------------------------------------------------------------
2022-05-18T04:26:23.3510238Z Ran 1 test in 0.002s
2022-05-18T04:26:23.3510408Z 
2022-05-18T04:26:23.3510521Z OK (skipped=1)
2022-05-18T04:26:23.3510683Z 
2022-05-18T04:26:23.3510812Z Generating XML reports...
2022-05-18T04:26:23.3547365Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042623.xml
2022-05-18T04:26:24.6056014Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:24.6070639Z 
2022-05-18T04:26:24.6070959Z Running tests...
2022-05-18T04:26:24.6071728Z ----------------------------------------------------------------------
2022-05-18T04:26:26.1931785Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:26.2321923Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26762
2022-05-18T04:26:26.2425671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26763
2022-05-18T04:26:27.3666256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:27.3824676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:27.3825529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:27.3868943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:27.3875324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:27.4839200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:27.6476172Z skip: Skipped due to small world size. (3.040s)
2022-05-18T04:26:27.6476454Z 
2022-05-18T04:26:27.6476851Z ----------------------------------------------------------------------
2022-05-18T04:26:27.6477204Z Ran 1 test in 3.040s
2022-05-18T04:26:27.6477353Z 
2022-05-18T04:26:27.6477466Z OK (skipped=1)
2022-05-18T04:26:27.6477627Z 
2022-05-18T04:26:27.6477756Z Generating XML reports...
2022-05-18T04:26:27.6518488Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042624.xml
2022-05-18T04:26:29.0461530Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:29.0476319Z 
2022-05-18T04:26:29.0476630Z Running tests...
2022-05-18T04:26:29.0477084Z ----------------------------------------------------------------------
2022-05-18T04:26:30.6200489Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:30.6596300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26869
2022-05-18T04:26:30.6701789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26870
2022-05-18T04:26:31.8153931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:31.8376713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:31.8377811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:31.8457448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:31.8464406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:31.9390973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:34.1496003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:26:34.1496558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:26:34.1497383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:26:34.1498089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:26:34.4797499Z ok (5.432s)
2022-05-18T04:26:34.4797855Z 
2022-05-18T04:26:34.4798977Z ----------------------------------------------------------------------
2022-05-18T04:26:34.4799498Z Ran 1 test in 5.432s
2022-05-18T04:26:34.4799650Z 
2022-05-18T04:26:34.4799953Z OK
2022-05-18T04:26:34.4800117Z 
2022-05-18T04:26:34.4800257Z Generating XML reports...
2022-05-18T04:26:34.4841476Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042629.xml
2022-05-18T04:26:35.9031040Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:35.9046621Z 
2022-05-18T04:26:35.9046771Z Running tests...
2022-05-18T04:26:35.9047682Z ----------------------------------------------------------------------
2022-05-18T04:26:37.4939049Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:37.5332769Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26997
2022-05-18T04:26:37.5437779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26998
2022-05-18T04:26:38.7078312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:38.7113030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:38.7113875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:38.7179495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:38.7186013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:38.8127646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:38.9489793Z skip: Need at least 3 CUDA devices (3.044s)
2022-05-18T04:26:38.9490236Z 
2022-05-18T04:26:38.9490873Z ----------------------------------------------------------------------
2022-05-18T04:26:38.9491509Z Ran 1 test in 3.044s
2022-05-18T04:26:38.9491823Z 
2022-05-18T04:26:38.9492020Z OK (skipped=1)
2022-05-18T04:26:38.9492323Z 
2022-05-18T04:26:38.9492558Z Generating XML reports...
2022-05-18T04:26:38.9533907Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042635.xml
2022-05-18T04:26:40.3367792Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:40.3381978Z 
2022-05-18T04:26:40.3382251Z Running tests...
2022-05-18T04:26:40.3382692Z ----------------------------------------------------------------------
2022-05-18T04:26:40.3402525Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-05-18T04:26:40.3402863Z 
2022-05-18T04:26:40.3403136Z ----------------------------------------------------------------------
2022-05-18T04:26:40.3403471Z Ran 1 test in 0.002s
2022-05-18T04:26:40.3403635Z 
2022-05-18T04:26:40.3403751Z OK (skipped=1)
2022-05-18T04:26:40.3403911Z 
2022-05-18T04:26:40.3404021Z Generating XML reports...
2022-05-18T04:26:40.3438481Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042640.xml
2022-05-18T04:26:41.5812116Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:41.5826197Z 
2022-05-18T04:26:41.5826687Z Running tests...
2022-05-18T04:26:41.5827193Z ----------------------------------------------------------------------
2022-05-18T04:26:41.5846137Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-05-18T04:26:41.5846513Z 
2022-05-18T04:26:41.5847035Z ----------------------------------------------------------------------
2022-05-18T04:26:41.5847504Z Ran 1 test in 0.002s
2022-05-18T04:26:41.5847670Z 
2022-05-18T04:26:41.5847782Z OK (skipped=1)
2022-05-18T04:26:41.5847921Z 
2022-05-18T04:26:41.5848050Z Generating XML reports...
2022-05-18T04:26:41.5883123Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042641.xml
2022-05-18T04:26:42.8411457Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:42.8426234Z 
2022-05-18T04:26:42.8426515Z Running tests...
2022-05-18T04:26:42.8426945Z ----------------------------------------------------------------------
2022-05-18T04:26:44.4307732Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:44.4704004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27170
2022-05-18T04:26:44.4811290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27171
2022-05-18T04:26:45.6379762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:45.6380687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:45.6381559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:45.6481635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:45.6487646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:45.7396377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:48.0900638Z ok (5.247s)
2022-05-18T04:26:48.0900889Z 
2022-05-18T04:26:48.0901272Z ----------------------------------------------------------------------
2022-05-18T04:26:48.0901622Z Ran 1 test in 5.247s
2022-05-18T04:26:48.0902089Z 
2022-05-18T04:26:48.0902194Z OK
2022-05-18T04:26:48.0902341Z 
2022-05-18T04:26:48.0902481Z Generating XML reports...
2022-05-18T04:26:48.0950367Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042642.xml
2022-05-18T04:26:49.5188273Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:49.5203771Z 
2022-05-18T04:26:49.5204143Z Running tests...
2022-05-18T04:26:49.5204640Z ----------------------------------------------------------------------
2022-05-18T04:26:49.5223766Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-05-18T04:26:49.5224083Z 
2022-05-18T04:26:49.5224353Z ----------------------------------------------------------------------
2022-05-18T04:26:49.5224692Z Ran 1 test in 0.002s
2022-05-18T04:26:49.5224864Z 
2022-05-18T04:26:49.5224978Z OK (skipped=1)
2022-05-18T04:26:49.5225151Z 
2022-05-18T04:26:49.5225277Z Generating XML reports...
2022-05-18T04:26:49.5261196Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042649.xml
2022-05-18T04:26:50.7913181Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:50.7928489Z 
2022-05-18T04:26:50.7928885Z Running tests...
2022-05-18T04:26:50.7929375Z ----------------------------------------------------------------------
2022-05-18T04:26:52.3702606Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:52.4099924Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27319
2022-05-18T04:26:52.4205399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27320
2022-05-18T04:26:53.5726401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:53.5839216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:53.5840255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:53.5929465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:53.5936017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:53.6853941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:53.8256074Z skip: Skipped due to small world size. (3.032s)
2022-05-18T04:26:53.8256348Z 
2022-05-18T04:26:53.8256728Z ----------------------------------------------------------------------
2022-05-18T04:26:53.8257073Z Ran 1 test in 3.033s
2022-05-18T04:26:53.8257241Z 
2022-05-18T04:26:53.8257352Z OK (skipped=1)
2022-05-18T04:26:53.8257513Z 
2022-05-18T04:26:53.8257662Z Generating XML reports...
2022-05-18T04:26:53.8299160Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042650.xml
2022-05-18T04:26:55.2310687Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:55.2325911Z 
2022-05-18T04:26:55.2326078Z Running tests...
2022-05-18T04:26:55.2326537Z ----------------------------------------------------------------------
2022-05-18T04:26:55.2346038Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-05-18T04:26:55.2346358Z 
2022-05-18T04:26:55.2346726Z ----------------------------------------------------------------------
2022-05-18T04:26:55.2347315Z Ran 1 test in 0.002s
2022-05-18T04:26:55.2347532Z 
2022-05-18T04:26:55.2347630Z OK (skipped=1)
2022-05-18T04:26:55.2349527Z 
2022-05-18T04:26:55.2349919Z Generating XML reports...
2022-05-18T04:26:55.2384709Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042655.xml
2022-05-18T04:26:56.4955517Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:26:56.4970767Z 
2022-05-18T04:26:56.4970947Z Running tests...
2022-05-18T04:26:56.4971408Z ----------------------------------------------------------------------
2022-05-18T04:26:58.0634816Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:58.1021088Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27459
2022-05-18T04:26:58.1124375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27460
2022-05-18T04:26:59.2409128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:59.2738615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:59.2739448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:59.2814214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:59.2820711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:59.3752880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:59.5175710Z skip: Skipped due to small world size. (3.020s)
2022-05-18T04:26:59.5175984Z 
2022-05-18T04:26:59.5176369Z ----------------------------------------------------------------------
2022-05-18T04:26:59.5176697Z Ran 1 test in 3.020s
2022-05-18T04:26:59.5176862Z 
2022-05-18T04:26:59.5176979Z OK (skipped=1)
2022-05-18T04:26:59.5178353Z 
2022-05-18T04:26:59.5178673Z Generating XML reports...
2022-05-18T04:26:59.5218587Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042656.xml
2022-05-18T04:27:00.9301138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:00.9315918Z 
2022-05-18T04:27:00.9316282Z Running tests...
2022-05-18T04:27:00.9317116Z ----------------------------------------------------------------------
2022-05-18T04:27:00.9337774Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-05-18T04:27:00.9338116Z 
2022-05-18T04:27:00.9338427Z ----------------------------------------------------------------------
2022-05-18T04:27:00.9338746Z Ran 1 test in 0.002s
2022-05-18T04:27:00.9338915Z 
2022-05-18T04:27:00.9339032Z OK (skipped=1)
2022-05-18T04:27:00.9339190Z 
2022-05-18T04:27:00.9339318Z Generating XML reports...
2022-05-18T04:27:00.9375318Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042700.xml
2022-05-18T04:27:02.2048625Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:02.2063024Z 
2022-05-18T04:27:02.2063286Z Running tests...
2022-05-18T04:27:02.2063741Z ----------------------------------------------------------------------
2022-05-18T04:27:02.2087557Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-05-18T04:27:02.2087892Z 
2022-05-18T04:27:02.2088162Z ----------------------------------------------------------------------
2022-05-18T04:27:02.2088504Z Ran 1 test in 0.002s
2022-05-18T04:27:02.2088669Z 
2022-05-18T04:27:02.2088783Z OK (skipped=1)
2022-05-18T04:27:02.2088940Z 
2022-05-18T04:27:02.2089072Z Generating XML reports...
2022-05-18T04:27:02.2125480Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042702.xml
2022-05-18T04:27:03.4807367Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:03.4822363Z 
2022-05-18T04:27:03.4822576Z Running tests...
2022-05-18T04:27:03.4823030Z ----------------------------------------------------------------------
2022-05-18T04:27:03.4843667Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-05-18T04:27:03.4844000Z 
2022-05-18T04:27:03.4844287Z ----------------------------------------------------------------------
2022-05-18T04:27:03.4844607Z Ran 1 test in 0.002s
2022-05-18T04:27:03.4844775Z 
2022-05-18T04:27:03.4844892Z OK (skipped=1)
2022-05-18T04:27:03.4845054Z 
2022-05-18T04:27:03.4845190Z Generating XML reports...
2022-05-18T04:27:03.4881362Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042703.xml
2022-05-18T04:27:04.7424934Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:04.7439460Z 
2022-05-18T04:27:04.7439872Z Running tests...
2022-05-18T04:27:04.7440405Z ----------------------------------------------------------------------
2022-05-18T04:27:04.7465513Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-05-18T04:27:04.7465837Z 
2022-05-18T04:27:04.7466150Z ----------------------------------------------------------------------
2022-05-18T04:27:04.7466471Z Ran 1 test in 0.003s
2022-05-18T04:27:04.7466636Z 
2022-05-18T04:27:04.7466748Z OK (skipped=1)
2022-05-18T04:27:04.7466909Z 
2022-05-18T04:27:04.7467037Z Generating XML reports...
2022-05-18T04:27:04.7503041Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042704.xml
2022-05-18T04:27:05.9662042Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:05.9676113Z 
2022-05-18T04:27:05.9676343Z Running tests...
2022-05-18T04:27:05.9676776Z ----------------------------------------------------------------------
2022-05-18T04:27:05.9701886Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-05-18T04:27:05.9702206Z 
2022-05-18T04:27:05.9702495Z ----------------------------------------------------------------------
2022-05-18T04:27:05.9703151Z Ran 1 test in 0.003s
2022-05-18T04:27:05.9703318Z 
2022-05-18T04:27:05.9703435Z OK (skipped=1)
2022-05-18T04:27:05.9703594Z 
2022-05-18T04:27:05.9703723Z Generating XML reports...
2022-05-18T04:27:05.9739172Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042705.xml
2022-05-18T04:27:07.2233135Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:07.2248266Z 
2022-05-18T04:27:07.2248651Z Running tests...
2022-05-18T04:27:07.2249159Z ----------------------------------------------------------------------
2022-05-18T04:27:08.8036935Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:08.8429836Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27731
2022-05-18T04:27:08.8534780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27732
2022-05-18T04:27:09.9942948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:10.0152415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:10.0153244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:10.0248491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:10.0254647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:10.1163699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:10.1386126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:27:10.1386655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:27:10.1387346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:27:10.1388053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:27:10.1391012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:27:10.1491978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:27:10.1492648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:27:10.1493626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:27:10.4588074Z ok (3.234s)
2022-05-18T04:27:10.4588378Z 
2022-05-18T04:27:10.4589075Z ----------------------------------------------------------------------
2022-05-18T04:27:10.4589716Z Ran 1 test in 3.234s
2022-05-18T04:27:10.4590028Z 
2022-05-18T04:27:10.4590216Z OK
2022-05-18T04:27:10.4590474Z 
2022-05-18T04:27:10.4590696Z Generating XML reports...
2022-05-18T04:27:10.4632625Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042707.xml
2022-05-18T04:27:11.8797836Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:11.8813524Z 
2022-05-18T04:27:11.8813783Z Running tests...
2022-05-18T04:27:11.8814249Z ----------------------------------------------------------------------
2022-05-18T04:27:13.4549526Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:13.4930306Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27848
2022-05-18T04:27:13.5032499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27849
2022-05-18T04:27:14.6540447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:14.6950112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:14.6950930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:14.7046931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:14.7053469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:14.7960563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:16.4107933Z ok (4.529s)
2022-05-18T04:27:16.4108163Z 
2022-05-18T04:27:16.4108562Z ----------------------------------------------------------------------
2022-05-18T04:27:16.4108886Z Ran 1 test in 4.529s
2022-05-18T04:27:16.4109077Z 
2022-05-18T04:27:16.4109177Z OK
2022-05-18T04:27:16.4109315Z 
2022-05-18T04:27:16.4109459Z Generating XML reports...
2022-05-18T04:27:16.4150956Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042711.xml
2022-05-18T04:27:17.8053839Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:17.8067359Z 
2022-05-18T04:27:17.8067626Z Running tests...
2022-05-18T04:27:17.8068063Z ----------------------------------------------------------------------
2022-05-18T04:27:19.3453078Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:19.3836222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27964
2022-05-18T04:27:19.3943405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27965
2022-05-18T04:27:20.5264681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:20.5336838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:20.5337649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:20.5365987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:20.5372206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:20.6351820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:20.7992595Z skip: Skipped due to small world size. (2.992s)
2022-05-18T04:27:20.7992860Z 
2022-05-18T04:27:20.7993256Z ----------------------------------------------------------------------
2022-05-18T04:27:20.7993606Z Ran 1 test in 2.992s
2022-05-18T04:27:20.7993798Z 
2022-05-18T04:27:20.7993923Z OK (skipped=1)
2022-05-18T04:27:20.7994063Z 
2022-05-18T04:27:20.7994193Z Generating XML reports...
2022-05-18T04:27:20.8034728Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042717.xml
2022-05-18T04:27:22.2021682Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:22.2035786Z 
2022-05-18T04:27:22.2036041Z Running tests...
2022-05-18T04:27:22.2036485Z ----------------------------------------------------------------------
2022-05-18T04:27:23.7832088Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:23.8224235Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28071
2022-05-18T04:27:23.8325994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28072
2022-05-18T04:27:24.9808119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:25.0145301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:25.0146155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:25.0213316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:25.0219560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:25.1157206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:26.5401149Z ok (4.336s)
2022-05-18T04:27:26.5401378Z 
2022-05-18T04:27:26.5401778Z ----------------------------------------------------------------------
2022-05-18T04:27:26.5402106Z Ran 1 test in 4.336s
2022-05-18T04:27:26.5402288Z 
2022-05-18T04:27:26.5402386Z OK
2022-05-18T04:27:26.5402545Z 
2022-05-18T04:27:26.5402683Z Generating XML reports...
2022-05-18T04:27:26.5444559Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042722.xml
2022-05-18T04:27:27.9678689Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:27.9693578Z 
2022-05-18T04:27:27.9693849Z Running tests...
2022-05-18T04:27:27.9694303Z ----------------------------------------------------------------------
2022-05-18T04:27:29.5342722Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:29.5737520Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28179
2022-05-18T04:27:29.5840883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28180
2022-05-18T04:27:30.7112982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:30.7222337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:30.7223150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:30.7315680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:30.7322262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:30.8233248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:31.0893434Z ok (3.120s)
2022-05-18T04:27:31.0893689Z 
2022-05-18T04:27:31.0894078Z ----------------------------------------------------------------------
2022-05-18T04:27:31.0894429Z Ran 1 test in 3.120s
2022-05-18T04:27:31.0894604Z 
2022-05-18T04:27:31.0894706Z OK
2022-05-18T04:27:31.0896189Z 
2022-05-18T04:27:31.0896816Z Generating XML reports...
2022-05-18T04:27:31.0936276Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042727.xml
2022-05-18T04:27:32.5049445Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:32.5063865Z 
2022-05-18T04:27:32.5064365Z Running tests...
2022-05-18T04:27:32.5064845Z ----------------------------------------------------------------------
2022-05-18T04:27:34.0779590Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:34.1171179Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28286
2022-05-18T04:27:34.1275670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28287
2022-05-18T04:27:35.2602422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:35.3085973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:35.3086828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:35.3108613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:35.3114557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:35.4096009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:37.0352166Z ok (4.528s)
2022-05-18T04:27:37.0352407Z 
2022-05-18T04:27:37.0352804Z ----------------------------------------------------------------------
2022-05-18T04:27:37.0353126Z Ran 1 test in 4.529s
2022-05-18T04:27:37.0353296Z 
2022-05-18T04:27:37.0353394Z OK
2022-05-18T04:27:37.0353530Z 
2022-05-18T04:27:37.0353665Z Generating XML reports...
2022-05-18T04:27:37.0394621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042732.xml
2022-05-18T04:27:38.4774485Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:38.4788699Z 
2022-05-18T04:27:38.4789006Z Running tests...
2022-05-18T04:27:38.4789437Z ----------------------------------------------------------------------
2022-05-18T04:27:40.0476228Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:40.0869475Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28402
2022-05-18T04:27:40.0972763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28403
2022-05-18T04:27:41.2767660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:41.2907289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:41.2908138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:41.2970364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:41.2976890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:41.3918024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:43.1050109Z ok (4.626s)
2022-05-18T04:27:43.1050339Z 
2022-05-18T04:27:43.1050978Z ----------------------------------------------------------------------
2022-05-18T04:27:43.1051362Z Ran 1 test in 4.626s
2022-05-18T04:27:43.1051542Z 
2022-05-18T04:27:43.1051642Z OK
2022-05-18T04:27:43.1051762Z 
2022-05-18T04:27:43.1051897Z Generating XML reports...
2022-05-18T04:27:43.1092467Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042738.xml
2022-05-18T04:27:44.5254703Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:44.5269598Z 
2022-05-18T04:27:44.5269921Z Running tests...
2022-05-18T04:27:44.5270619Z ----------------------------------------------------------------------
2022-05-18T04:27:46.1138703Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:46.1537776Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28518
2022-05-18T04:27:46.1643807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28519
2022-05-18T04:27:47.3329390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:47.3463836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:47.3465403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:47.3531908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:47.3538113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:47.4474698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:47.7698778Z ok (3.242s)
2022-05-18T04:27:47.7699159Z 
2022-05-18T04:27:47.7702376Z ----------------------------------------------------------------------
2022-05-18T04:27:47.7703160Z Ran 1 test in 3.243s
2022-05-18T04:27:47.7703470Z 
2022-05-18T04:27:47.7703570Z OK
2022-05-18T04:27:47.7703712Z 
2022-05-18T04:27:47.7703852Z Generating XML reports...
2022-05-18T04:27:47.7741642Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042744.xml
2022-05-18T04:27:49.1735336Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:49.1749902Z 
2022-05-18T04:27:49.1750056Z Running tests...
2022-05-18T04:27:49.1751126Z ----------------------------------------------------------------------
2022-05-18T04:27:49.1769738Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:27:49.1770356Z 
2022-05-18T04:27:49.1770949Z ----------------------------------------------------------------------
2022-05-18T04:27:49.1771452Z Ran 1 test in 0.002s
2022-05-18T04:27:49.1771622Z 
2022-05-18T04:27:49.1771741Z OK (skipped=1)
2022-05-18T04:27:49.1772117Z 
2022-05-18T04:27:49.1772251Z Generating XML reports...
2022-05-18T04:27:49.1805817Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042749.xml
2022-05-18T04:27:50.4307418Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:50.4322775Z 
2022-05-18T04:27:50.4323223Z Running tests...
2022-05-18T04:27:50.4323730Z ----------------------------------------------------------------------
2022-05-18T04:27:52.0400781Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:52.0788604Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28658
2022-05-18T04:27:52.0893477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28659
2022-05-18T04:27:53.2308300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:53.2980933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:53.2981775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:53.3016721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:27:53.3022908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:53.3995609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:55.4980897Z ok (5.065s)
2022-05-18T04:27:55.4981125Z 
2022-05-18T04:27:55.4981535Z ----------------------------------------------------------------------
2022-05-18T04:27:55.4981879Z Ran 1 test in 5.066s
2022-05-18T04:27:55.4982032Z 
2022-05-18T04:27:55.4982133Z OK
2022-05-18T04:27:55.4982270Z 
2022-05-18T04:27:55.4982408Z Generating XML reports...
2022-05-18T04:27:55.5023159Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042750.xml
2022-05-18T04:27:56.9070378Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:56.9084305Z 
2022-05-18T04:27:56.9084786Z Running tests...
2022-05-18T04:27:56.9085612Z ----------------------------------------------------------------------
2022-05-18T04:27:56.9103270Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:27:56.9103602Z 
2022-05-18T04:27:56.9103972Z ----------------------------------------------------------------------
2022-05-18T04:27:56.9104507Z Ran 1 test in 0.002s
2022-05-18T04:27:56.9104689Z 
2022-05-18T04:27:56.9104803Z OK (skipped=1)
2022-05-18T04:27:56.9104962Z 
2022-05-18T04:27:56.9105090Z Generating XML reports...
2022-05-18T04:27:56.9139260Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042756.xml
2022-05-18T04:27:58.1631271Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:58.1646279Z 
2022-05-18T04:27:58.1646548Z Running tests...
2022-05-18T04:27:58.1647022Z ----------------------------------------------------------------------
2022-05-18T04:27:58.1666048Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:27:58.1666372Z 
2022-05-18T04:27:58.1666664Z ----------------------------------------------------------------------
2022-05-18T04:27:58.1666999Z Ran 1 test in 0.002s
2022-05-18T04:27:58.1667167Z 
2022-05-18T04:27:58.1667283Z OK (skipped=1)
2022-05-18T04:27:58.1667448Z 
2022-05-18T04:27:58.1667556Z Generating XML reports...
2022-05-18T04:27:58.1703410Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042758.xml
2022-05-18T04:27:59.4267681Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:27:59.4282844Z 
2022-05-18T04:27:59.4283111Z Running tests...
2022-05-18T04:27:59.4283563Z ----------------------------------------------------------------------
2022-05-18T04:27:59.4303698Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-05-18T04:27:59.4304024Z 
2022-05-18T04:27:59.4304313Z ----------------------------------------------------------------------
2022-05-18T04:27:59.4304651Z Ran 1 test in 0.002s
2022-05-18T04:27:59.4304822Z 
2022-05-18T04:27:59.4304916Z OK (skipped=1)
2022-05-18T04:27:59.4305080Z 
2022-05-18T04:27:59.4305207Z Generating XML reports...
2022-05-18T04:27:59.4341531Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042759.xml
2022-05-18T04:28:00.6737267Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:00.6751965Z 
2022-05-18T04:28:00.6752125Z Running tests...
2022-05-18T04:28:00.6752862Z ----------------------------------------------------------------------
2022-05-18T04:28:02.2381463Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:02.2774367Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28877
2022-05-18T04:28:02.2879992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28878
2022-05-18T04:28:03.4203443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:03.4306022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:03.4306815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:03.4406445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:03.4413211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:03.5321116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:06.1977114Z ok (5.522s)
2022-05-18T04:28:06.1977595Z 
2022-05-18T04:28:06.1978028Z ----------------------------------------------------------------------
2022-05-18T04:28:06.1978396Z Ran 1 test in 5.522s
2022-05-18T04:28:06.1978553Z 
2022-05-18T04:28:06.1978649Z OK
2022-05-18T04:28:06.1978798Z 
2022-05-18T04:28:06.1978934Z Generating XML reports...
2022-05-18T04:28:06.2018860Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042800.xml
2022-05-18T04:28:07.6215284Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:07.6229169Z 
2022-05-18T04:28:07.6229579Z Running tests...
2022-05-18T04:28:07.6230100Z ----------------------------------------------------------------------
2022-05-18T04:28:09.2067192Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:09.2460191Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28994
2022-05-18T04:28:09.2565557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28995
2022-05-18T04:28:10.4001706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:10.4037302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:10.4038505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:10.4104086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:10.4110227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:10.5049445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:10.5259928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:10.5260866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:10.5261582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:28:10.5262280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:28:10.5263396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:28:10.5263903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:28:10.5264587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:28:10.5265276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:28:11.8153911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7bwk1a1e
2022-05-18T04:28:11.8154528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7bwk1a1e/_remote_module_non_scriptable.py
2022-05-18T04:28:11.8410477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvtxsv2zj
2022-05-18T04:28:11.8412908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvtxsv2zj/_remote_module_non_scriptable.py
2022-05-18T04:28:12.2645650Z ok (4.641s)
2022-05-18T04:28:12.2645873Z 
2022-05-18T04:28:12.2646264Z ----------------------------------------------------------------------
2022-05-18T04:28:12.2646615Z Ran 1 test in 4.642s
2022-05-18T04:28:12.2646787Z 
2022-05-18T04:28:12.2646889Z OK
2022-05-18T04:28:12.2647026Z 
2022-05-18T04:28:12.2647162Z Generating XML reports...
2022-05-18T04:28:12.2687562Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042807.xml
2022-05-18T04:28:13.6624361Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:13.6637952Z 
2022-05-18T04:28:13.6638406Z Running tests...
2022-05-18T04:28:13.6639160Z ----------------------------------------------------------------------
2022-05-18T04:28:15.2042159Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:15.2424610Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29118
2022-05-18T04:28:15.2530580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29119
2022-05-18T04:28:16.3902990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:16.4129070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:16.4129881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:16.4207020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:16.4213398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:16.5139680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:16.5334025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:16.5334756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:16.5335828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:28:16.5336514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:28:16.5337373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:28:16.5440625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:28:16.5441319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:28:16.5541706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:28:17.8071052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5c9i6_8
2022-05-18T04:28:17.8071752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5c9i6_8/_remote_module_non_scriptable.py
2022-05-18T04:28:17.8325140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuha_tcoy
2022-05-18T04:28:17.8328556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuha_tcoy/_remote_module_non_scriptable.py
2022-05-18T04:28:18.2609689Z ok (4.597s)
2022-05-18T04:28:18.2609915Z 
2022-05-18T04:28:18.2610308Z ----------------------------------------------------------------------
2022-05-18T04:28:18.2610634Z Ran 1 test in 4.597s
2022-05-18T04:28:18.2610805Z 
2022-05-18T04:28:18.2610908Z OK
2022-05-18T04:28:18.2611047Z 
2022-05-18T04:28:18.2611184Z Generating XML reports...
2022-05-18T04:28:18.2652300Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042813.xml
2022-05-18T04:28:19.6914129Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:19.6928984Z 
2022-05-18T04:28:19.6929437Z Running tests...
2022-05-18T04:28:19.6929941Z ----------------------------------------------------------------------
2022-05-18T04:28:21.2745594Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:21.3144387Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29242
2022-05-18T04:28:21.3250587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29243
2022-05-18T04:28:22.4558718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:22.4748862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:22.4750419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:22.4760939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:22.4767781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:22.5763410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:23.7263728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdkcmoqid
2022-05-18T04:28:23.7264337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdkcmoqid/_remote_module_non_scriptable.py
2022-05-18T04:28:23.8347867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3cfxcc9
2022-05-18T04:28:23.8348452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3cfxcc9/_remote_module_non_scriptable.py
2022-05-18T04:28:24.0946758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:24.0947303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:24.4333643Z ok (4.740s)
2022-05-18T04:28:24.4333993Z 
2022-05-18T04:28:24.4334676Z ----------------------------------------------------------------------
2022-05-18T04:28:24.4335032Z Ran 1 test in 4.740s
2022-05-18T04:28:24.4335202Z 
2022-05-18T04:28:24.4335308Z OK
2022-05-18T04:28:24.4335462Z 
2022-05-18T04:28:24.4335692Z Generating XML reports...
2022-05-18T04:28:24.4376261Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042819.xml
2022-05-18T04:28:25.8334077Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:25.8348493Z 
2022-05-18T04:28:25.8348911Z Running tests...
2022-05-18T04:28:25.8349433Z ----------------------------------------------------------------------
2022-05-18T04:28:27.3756566Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:27.4141584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29362
2022-05-18T04:28:27.4246296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29363
2022-05-18T04:28:28.5564067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:28.6171516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:28.6172334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:28.6173042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:28.6178959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:28.6179460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:29.8839242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqu_wlt_s
2022-05-18T04:28:29.8839857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqu_wlt_s/_remote_module_non_scriptable.py
2022-05-18T04:28:29.9250905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl2nfqomf
2022-05-18T04:28:29.9252984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl2nfqomf/_remote_module_non_scriptable.py
2022-05-18T04:28:30.1877630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:30.1878534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:30.1885199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:30.1885681Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:30.5327351Z ok (4.698s)
2022-05-18T04:28:30.5327567Z 
2022-05-18T04:28:30.5327956Z ----------------------------------------------------------------------
2022-05-18T04:28:30.5328292Z Ran 1 test in 4.698s
2022-05-18T04:28:30.5328460Z 
2022-05-18T04:28:30.5328557Z OK
2022-05-18T04:28:30.5328695Z 
2022-05-18T04:28:30.5328827Z Generating XML reports...
2022-05-18T04:28:30.5370204Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042825.xml
2022-05-18T04:28:31.9531621Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:31.9545940Z 
2022-05-18T04:28:31.9546112Z Running tests...
2022-05-18T04:28:31.9546828Z ----------------------------------------------------------------------
2022-05-18T04:28:33.5304655Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:33.5697084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29482
2022-05-18T04:28:33.5801831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29483
2022-05-18T04:28:34.7515432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:34.7522422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:34.7523213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:34.7617350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:34.7623554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:34.8537121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:36.0181503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjj803c19
2022-05-18T04:28:36.0182133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjj803c19/_remote_module_non_scriptable.py
2022-05-18T04:28:36.1171875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rfbdtd_
2022-05-18T04:28:36.1172797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rfbdtd_/_remote_module_non_scriptable.py
2022-05-18T04:28:36.3758117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:36.3759087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:36.3767736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:36.3769855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:36.3891285Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:36.3891766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:36.3900015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:36.3900515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:36.6885341Z ok (4.733s)
2022-05-18T04:28:36.6885953Z 
2022-05-18T04:28:36.6886358Z ----------------------------------------------------------------------
2022-05-18T04:28:36.6886714Z Ran 1 test in 4.734s
2022-05-18T04:28:36.6886882Z 
2022-05-18T04:28:36.6886978Z OK
2022-05-18T04:28:36.6887112Z 
2022-05-18T04:28:36.6887248Z Generating XML reports...
2022-05-18T04:28:36.6928159Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042831.xml
2022-05-18T04:28:38.1126807Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:38.1140695Z 
2022-05-18T04:28:38.1140980Z Running tests...
2022-05-18T04:28:38.1141426Z ----------------------------------------------------------------------
2022-05-18T04:28:39.7185815Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:39.7341100Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.620s)
2022-05-18T04:28:39.7341685Z 
2022-05-18T04:28:39.7341964Z ----------------------------------------------------------------------
2022-05-18T04:28:39.7342299Z Ran 1 test in 1.620s
2022-05-18T04:28:39.7342469Z 
2022-05-18T04:28:39.7342564Z OK (skipped=1)
2022-05-18T04:28:39.7342720Z 
2022-05-18T04:28:39.7345390Z Generating XML reports...
2022-05-18T04:28:39.7375484Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042838.xml
2022-05-18T04:28:41.0852378Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:41.0866156Z 
2022-05-18T04:28:41.0866628Z Running tests...
2022-05-18T04:28:41.0867195Z ----------------------------------------------------------------------
2022-05-18T04:28:42.6263913Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:42.6647119Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29636
2022-05-18T04:28:42.6754477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29637
2022-05-18T04:28:43.8434561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:43.8575885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:43.8576699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:43.8637103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:43.8644405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:43.9590705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:45.1232709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbzkqfsm
2022-05-18T04:28:45.1233578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbzkqfsm/_remote_module_non_scriptable.py
2022-05-18T04:28:45.2235904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbb62kj9r
2022-05-18T04:28:45.2237468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbb62kj9r/_remote_module_non_scriptable.py
2022-05-18T04:28:45.2979351Z <class 'torch.nn.parameter.Parameter'>
2022-05-18T04:28:45.5828573Z ok (4.496s)
2022-05-18T04:28:45.5828802Z 
2022-05-18T04:28:45.5829171Z ----------------------------------------------------------------------
2022-05-18T04:28:45.5829517Z Ran 1 test in 4.496s
2022-05-18T04:28:45.5829687Z 
2022-05-18T04:28:45.5829784Z OK
2022-05-18T04:28:45.5830179Z 
2022-05-18T04:28:45.5830330Z Generating XML reports...
2022-05-18T04:28:45.5870084Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042841.xml
2022-05-18T04:28:46.9939516Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:46.9953237Z 
2022-05-18T04:28:46.9953567Z Running tests...
2022-05-18T04:28:46.9954072Z ----------------------------------------------------------------------
2022-05-18T04:28:48.5709340Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:48.6097246Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29752
2022-05-18T04:28:48.6200891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29753
2022-05-18T04:28:49.7871374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:49.8072851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:49.8073668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:49.8074353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:49.8081985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:49.8082584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:51.1060890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8y9_hw1z
2022-05-18T04:28:51.1061507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8y9_hw1z/_remote_module_non_scriptable.py
2022-05-18T04:28:51.1155311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpse9h6h1n
2022-05-18T04:28:51.1158448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpse9h6h1n/_remote_module_non_scriptable.py
2022-05-18T04:28:51.5279737Z ok (4.532s)
2022-05-18T04:28:51.5279932Z 
2022-05-18T04:28:51.5280484Z ----------------------------------------------------------------------
2022-05-18T04:28:51.5281029Z Ran 1 test in 4.533s
2022-05-18T04:28:51.5281205Z 
2022-05-18T04:28:51.5281303Z OK
2022-05-18T04:28:51.5281425Z 
2022-05-18T04:28:51.5281561Z Generating XML reports...
2022-05-18T04:28:51.5322637Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042846.xml
2022-05-18T04:28:52.9254383Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:52.9269960Z 
2022-05-18T04:28:52.9270512Z Running tests...
2022-05-18T04:28:52.9271405Z ----------------------------------------------------------------------
2022-05-18T04:28:54.5130339Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:54.5525289Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29868
2022-05-18T04:28:54.5627628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29869
2022-05-18T04:28:55.6733594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:55.6809829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:55.6810987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:55.6834645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:28:55.6841074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:55.7825322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:56.9279390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6ywvrnw
2022-05-18T04:28:56.9280352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6ywvrnw/_remote_module_non_scriptable.py
2022-05-18T04:28:56.9946030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2i4yx72
2022-05-18T04:28:56.9947187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2i4yx72/_remote_module_non_scriptable.py
2022-05-18T04:28:57.2491783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:57.2492344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:57.5708441Z ok (4.643s)
2022-05-18T04:28:57.5708886Z 
2022-05-18T04:28:57.5709558Z ----------------------------------------------------------------------
2022-05-18T04:28:57.5710210Z Ran 1 test in 4.644s
2022-05-18T04:28:57.5710524Z 
2022-05-18T04:28:57.5710667Z OK
2022-05-18T04:28:57.5710920Z 
2022-05-18T04:28:57.5711182Z Generating XML reports...
2022-05-18T04:28:57.5752340Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042852.xml
2022-05-18T04:28:58.9943061Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:28:58.9957075Z 
2022-05-18T04:28:58.9957424Z Running tests...
2022-05-18T04:28:58.9958410Z ----------------------------------------------------------------------
2022-05-18T04:29:00.5737582Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:00.6131875Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29988
2022-05-18T04:29:00.6237380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29989
2022-05-18T04:29:01.7686804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:01.7819900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:01.7820716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:01.7889646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:01.7895898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:01.8834415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:03.0742359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkbq0nda9
2022-05-18T04:29:03.0743633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkbq0nda9/_remote_module_non_scriptable.py
2022-05-18T04:29:03.1549579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsba4ro13
2022-05-18T04:29:03.1550404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsba4ro13/_remote_module_non_scriptable.py
2022-05-18T04:29:03.4047662Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:29:03.7321304Z ok (4.736s)
2022-05-18T04:29:03.7322026Z 
2022-05-18T04:29:03.7322605Z ----------------------------------------------------------------------
2022-05-18T04:29:03.7323726Z Ran 1 test in 4.736s
2022-05-18T04:29:03.7323985Z 
2022-05-18T04:29:03.7324066Z OK
2022-05-18T04:29:03.7324204Z 
2022-05-18T04:29:03.7324340Z Generating XML reports...
2022-05-18T04:29:03.7364823Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042858.xml
2022-05-18T04:29:05.1541180Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:05.1554821Z 
2022-05-18T04:29:05.1555325Z Running tests...
2022-05-18T04:29:05.1556271Z ----------------------------------------------------------------------
2022-05-18T04:29:06.7348048Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:06.7740373Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30108
2022-05-18T04:29:06.7844689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30109
2022-05-18T04:29:07.9144793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:07.9338765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:07.9339562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:07.9347636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:07.9353821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:08.0353199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:09.2084808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqyr76pqc
2022-05-18T04:29:09.2085723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqyr76pqc/_remote_module_non_scriptable.py
2022-05-18T04:29:09.2729899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2kikoth
2022-05-18T04:29:09.2730831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2kikoth/_remote_module_non_scriptable.py
2022-05-18T04:29:09.5228327Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:29:09.5238377Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:29:09.8938533Z ok (4.738s)
2022-05-18T04:29:09.8938920Z 
2022-05-18T04:29:09.8939355Z ----------------------------------------------------------------------
2022-05-18T04:29:09.8939687Z Ran 1 test in 4.738s
2022-05-18T04:29:09.8939856Z 
2022-05-18T04:29:09.8939956Z OK
2022-05-18T04:29:09.8940093Z 
2022-05-18T04:29:09.8940252Z Generating XML reports...
2022-05-18T04:29:09.8980681Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042905.xml
2022-05-18T04:29:11.3170592Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:11.3184915Z 
2022-05-18T04:29:11.3185516Z Running tests...
2022-05-18T04:29:11.3186028Z ----------------------------------------------------------------------
2022-05-18T04:29:11.3210096Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2022-05-18T04:29:11.3210472Z 
2022-05-18T04:29:11.3210763Z ----------------------------------------------------------------------
2022-05-18T04:29:11.3211110Z Ran 1 test in 0.003s
2022-05-18T04:29:11.3211277Z 
2022-05-18T04:29:11.3211392Z OK (skipped=1)
2022-05-18T04:29:11.3211532Z 
2022-05-18T04:29:11.3211663Z Generating XML reports...
2022-05-18T04:29:11.3247246Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042911.xml
2022-05-18T04:29:12.5763717Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:12.5778198Z 
2022-05-18T04:29:12.5778687Z Running tests...
2022-05-18T04:29:12.5779158Z ----------------------------------------------------------------------
2022-05-18T04:29:14.1611601Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:14.2005265Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30261
2022-05-18T04:29:14.2109287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30262
2022-05-18T04:29:15.3405373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:15.3844316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:15.3845125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:15.3911456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:15.3918295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:15.4859145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:16.6285733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqvm_fg3
2022-05-18T04:29:16.6286835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqvm_fg3/_remote_module_non_scriptable.py
2022-05-18T04:29:16.7489446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3xui8128
2022-05-18T04:29:16.7490692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3xui8128/_remote_module_non_scriptable.py
2022-05-18T04:29:16.9967474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:16.9967991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:17.4191906Z ok (4.841s)
2022-05-18T04:29:17.4192301Z 
2022-05-18T04:29:17.4192763Z ----------------------------------------------------------------------
2022-05-18T04:29:17.4193319Z Ran 1 test in 4.841s
2022-05-18T04:29:17.4193501Z 
2022-05-18T04:29:17.4193600Z OK
2022-05-18T04:29:17.4193743Z 
2022-05-18T04:29:17.4193883Z Generating XML reports...
2022-05-18T04:29:17.4234982Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042912.xml
2022-05-18T04:29:18.8252328Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:18.8265675Z 
2022-05-18T04:29:18.8265846Z Running tests...
2022-05-18T04:29:18.8266294Z ----------------------------------------------------------------------
2022-05-18T04:29:20.3748079Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:20.4134240Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30381
2022-05-18T04:29:20.4237681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30382
2022-05-18T04:29:21.5633616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:21.5657328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:21.5658614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:21.5734928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:21.5741593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:21.6673136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:22.8290815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytn7x8w_
2022-05-18T04:29:22.8291847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytn7x8w_/_remote_module_non_scriptable.py
2022-05-18T04:29:22.9399743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmply1d8pgd
2022-05-18T04:29:22.9401800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmply1d8pgd/_remote_module_non_scriptable.py
2022-05-18T04:29:23.0173553Z /opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T04:29:23.0177712Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T04:29:23.0180195Z /opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T04:29:23.0182388Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T04:29:23.5318629Z ok (4.705s)
2022-05-18T04:29:23.5318820Z 
2022-05-18T04:29:23.5319724Z ----------------------------------------------------------------------
2022-05-18T04:29:23.5320060Z Ran 1 test in 4.705s
2022-05-18T04:29:23.5320239Z 
2022-05-18T04:29:23.5320336Z OK
2022-05-18T04:29:23.5320471Z 
2022-05-18T04:29:23.5320601Z Generating XML reports...
2022-05-18T04:29:23.5360669Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042918.xml
2022-05-18T04:29:24.9485148Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:24.9500250Z 
2022-05-18T04:29:24.9500491Z Running tests...
2022-05-18T04:29:24.9501421Z ----------------------------------------------------------------------
2022-05-18T04:29:26.5555670Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:26.5950286Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30531
2022-05-18T04:29:26.6054767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30532
2022-05-18T04:29:27.7118091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:27.7358430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:27.7359537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:27.7421723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:27.7428677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:27.8373647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:28.9988900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88g8wrh_
2022-05-18T04:29:28.9989523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88g8wrh_/_remote_module_non_scriptable.py
2022-05-18T04:29:29.0883054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwppg894y
2022-05-18T04:29:29.0884489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwppg894y/_remote_module_non_scriptable.py
2022-05-18T04:29:29.3481370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:29.3481959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:29.3656102Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp88g8wrh_'>
2022-05-18T04:29:29.3656589Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:29:29.3657172Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpwppg894y'>
2022-05-18T04:29:29.3657631Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:29:29.7137729Z ok (4.763s)
2022-05-18T04:29:29.7137954Z 
2022-05-18T04:29:29.7138360Z ----------------------------------------------------------------------
2022-05-18T04:29:29.7138987Z Ran 1 test in 4.764s
2022-05-18T04:29:29.7139158Z 
2022-05-18T04:29:29.7139256Z OK
2022-05-18T04:29:29.7139392Z 
2022-05-18T04:29:29.7139836Z Generating XML reports...
2022-05-18T04:29:29.7179526Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042924.xml
2022-05-18T04:29:31.1038018Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:31.1053482Z 
2022-05-18T04:29:31.1054009Z Running tests...
2022-05-18T04:29:31.1054540Z ----------------------------------------------------------------------
2022-05-18T04:29:32.6375470Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:32.6524232Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.547s)
2022-05-18T04:29:32.6524794Z 
2022-05-18T04:29:32.6525111Z ----------------------------------------------------------------------
2022-05-18T04:29:32.6525453Z Ran 1 test in 1.547s
2022-05-18T04:29:32.6525621Z 
2022-05-18T04:29:32.6525734Z OK (skipped=1)
2022-05-18T04:29:32.6525891Z 
2022-05-18T04:29:32.6526000Z Generating XML reports...
2022-05-18T04:29:32.6556556Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042931.xml
2022-05-18T04:29:34.0169770Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:34.0184241Z 
2022-05-18T04:29:34.0184493Z Running tests...
2022-05-18T04:29:34.0184932Z ----------------------------------------------------------------------
2022-05-18T04:29:35.5803715Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:35.6187186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30685
2022-05-18T04:29:35.6291654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30686
2022-05-18T04:29:36.7749581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:36.7814228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:36.7815254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:36.7850790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:36.7856780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:36.7860055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:36.8825812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:36.8829148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:36.8829833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:29:36.8877173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:29:38.1693064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkezvatm
2022-05-18T04:29:38.1693704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkezvatm/_remote_module_non_scriptable.py
2022-05-18T04:29:38.1847963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrf7w8ya
2022-05-18T04:29:38.1850593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrf7w8ya/_remote_module_non_scriptable.py
2022-05-18T04:29:38.4269146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:38.4269952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:38.4289114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:38.4289599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:38.9389916Z ok (4.920s)
2022-05-18T04:29:38.9390098Z 
2022-05-18T04:29:38.9390485Z ----------------------------------------------------------------------
2022-05-18T04:29:38.9390817Z Ran 1 test in 4.921s
2022-05-18T04:29:38.9390985Z 
2022-05-18T04:29:38.9391089Z OK
2022-05-18T04:29:38.9391232Z 
2022-05-18T04:29:38.9391364Z Generating XML reports...
2022-05-18T04:29:38.9432221Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042934.xml
2022-05-18T04:29:40.3510565Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:40.3526016Z 
2022-05-18T04:29:40.3526165Z Running tests...
2022-05-18T04:29:40.3526928Z ----------------------------------------------------------------------
2022-05-18T04:29:41.9248014Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:41.9640236Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30809
2022-05-18T04:29:41.9743770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30810
2022-05-18T04:29:43.1360970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:43.1442419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:43.1443195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:43.1462044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:43.1468368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:43.1470459Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:29:43.2457814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:43.2458427Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:29:44.3985604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05nkorb9
2022-05-18T04:29:44.3986208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05nkorb9/_remote_module_non_scriptable.py
2022-05-18T04:29:44.5133298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvoqb_vts
2022-05-18T04:29:44.5134172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvoqb_vts/_remote_module_non_scriptable.py
2022-05-18T04:29:44.7651924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:44.7655632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:44.7675008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:44.7678824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:44.7879533Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:29:44.7882851Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:29:44.9932578Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:29:44.9935772Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:29:44.9993295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:44.9997061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:45.0005571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:45.0009884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:45.0208888Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:29:45.0212250Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:29:45.1483079Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T04:29:45.1491935Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T04:29:45.1548490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:45.1552437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:45.1560248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:45.1565331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:45.6834536Z ok (5.330s)
2022-05-18T04:29:45.6834753Z 
2022-05-18T04:29:45.6835146Z ----------------------------------------------------------------------
2022-05-18T04:29:45.6835492Z Ran 1 test in 5.331s
2022-05-18T04:29:45.6835658Z 
2022-05-18T04:29:45.6835762Z OK
2022-05-18T04:29:45.6835898Z 
2022-05-18T04:29:45.6836035Z Generating XML reports...
2022-05-18T04:29:45.6876073Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042940.xml
2022-05-18T04:29:47.1001514Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:47.1017029Z 
2022-05-18T04:29:47.1017456Z Running tests...
2022-05-18T04:29:47.1018224Z ----------------------------------------------------------------------
2022-05-18T04:29:48.6767046Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:48.7161344Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30929
2022-05-18T04:29:48.7266252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30930
2022-05-18T04:29:49.8332359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:49.8551253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:49.8552681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:49.8636286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:49.8642345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:49.8645028Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:29:49.9562640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:49.9563961Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:29:51.0979800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplyxplndr
2022-05-18T04:29:51.0980442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplyxplndr/_remote_module_non_scriptable.py
2022-05-18T04:29:51.2103755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwjmbzuv3
2022-05-18T04:29:51.2104662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwjmbzuv3/_remote_module_non_scriptable.py
2022-05-18T04:29:51.4677194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:51.4678613Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:51.4699083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:51.4700944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:51.4706064Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:29:51.4707156Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:29:51.4732417Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:29:51.4734612Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:29:51.4735263Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:29:51.4735910Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T04:29:51.4736755Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:29:51.4738798Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T04:29:51.7475816Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:29:51.7478771Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:29:51.7540228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:51.7540719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:51.7551575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:51.7552072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:29:51.7557424Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:29:51.7557983Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:29:51.7583431Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:29:51.7584237Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:29:51.7584887Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:29:51.7585559Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:29:52.3355946Z ok (5.234s)
2022-05-18T04:29:52.3356200Z 
2022-05-18T04:29:52.3356598Z ----------------------------------------------------------------------
2022-05-18T04:29:52.3356946Z Ran 1 test in 5.234s
2022-05-18T04:29:52.3357121Z 
2022-05-18T04:29:52.3357216Z OK
2022-05-18T04:29:52.3357339Z 
2022-05-18T04:29:52.3357477Z Generating XML reports...
2022-05-18T04:29:52.3398595Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042947.xml
2022-05-18T04:29:53.7420792Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:53.7434242Z 
2022-05-18T04:29:53.7434656Z Running tests...
2022-05-18T04:29:53.7435167Z ----------------------------------------------------------------------
2022-05-18T04:29:53.7455992Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:29:53.7456434Z 
2022-05-18T04:29:53.7456730Z ----------------------------------------------------------------------
2022-05-18T04:29:53.7457070Z Ran 1 test in 0.002s
2022-05-18T04:29:53.7457216Z 
2022-05-18T04:29:53.7457330Z OK (skipped=1)
2022-05-18T04:29:53.7457487Z 
2022-05-18T04:29:53.7457613Z Generating XML reports...
2022-05-18T04:29:53.7491761Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042953.xml
2022-05-18T04:29:54.9607421Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:54.9623440Z 
2022-05-18T04:29:54.9623807Z Running tests...
2022-05-18T04:29:54.9625075Z ----------------------------------------------------------------------
2022-05-18T04:29:54.9647584Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:29:54.9648456Z 
2022-05-18T04:29:54.9649045Z ----------------------------------------------------------------------
2022-05-18T04:29:54.9649707Z Ran 1 test in 0.002s
2022-05-18T04:29:54.9650028Z 
2022-05-18T04:29:54.9650210Z OK (skipped=1)
2022-05-18T04:29:54.9650524Z 
2022-05-18T04:29:54.9650766Z Generating XML reports...
2022-05-18T04:29:54.9687031Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042954.xml
2022-05-18T04:29:56.2185556Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:56.2199849Z 
2022-05-18T04:29:56.2200588Z Running tests...
2022-05-18T04:29:56.2201107Z ----------------------------------------------------------------------
2022-05-18T04:29:56.2223471Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:29:56.2223978Z 
2022-05-18T04:29:56.2224274Z ----------------------------------------------------------------------
2022-05-18T04:29:56.2224605Z Ran 1 test in 0.002s
2022-05-18T04:29:56.2224769Z 
2022-05-18T04:29:56.2224862Z OK (skipped=1)
2022-05-18T04:29:56.2225022Z 
2022-05-18T04:29:56.2225151Z Generating XML reports...
2022-05-18T04:29:56.2260631Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042956.xml
2022-05-18T04:29:57.4465940Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:57.4480364Z 
2022-05-18T04:29:57.4480824Z Running tests...
2022-05-18T04:29:57.4481346Z ----------------------------------------------------------------------
2022-05-18T04:29:57.4503720Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:29:57.4504237Z 
2022-05-18T04:29:57.4504528Z ----------------------------------------------------------------------
2022-05-18T04:29:57.4504865Z Ran 1 test in 0.002s
2022-05-18T04:29:57.4505032Z 
2022-05-18T04:29:57.4505146Z OK (skipped=1)
2022-05-18T04:29:57.4505299Z 
2022-05-18T04:29:57.4505421Z Generating XML reports...
2022-05-18T04:29:57.4540872Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042957.xml
2022-05-18T04:29:58.6695930Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:58.6709931Z 
2022-05-18T04:29:58.6710148Z Running tests...
2022-05-18T04:29:58.6710607Z ----------------------------------------------------------------------
2022-05-18T04:29:58.6733782Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:29:58.6734295Z 
2022-05-18T04:29:58.6734587Z ----------------------------------------------------------------------
2022-05-18T04:29:58.6734919Z Ran 1 test in 0.002s
2022-05-18T04:29:58.6735068Z 
2022-05-18T04:29:58.6735178Z OK (skipped=1)
2022-05-18T04:29:58.6735346Z 
2022-05-18T04:29:58.6735471Z Generating XML reports...
2022-05-18T04:29:58.6771286Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042958.xml
2022-05-18T04:29:59.9351794Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:29:59.9366964Z 
2022-05-18T04:29:59.9367129Z Running tests...
2022-05-18T04:29:59.9367566Z ----------------------------------------------------------------------
2022-05-18T04:29:59.9390543Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:29:59.9391040Z 
2022-05-18T04:29:59.9391326Z ----------------------------------------------------------------------
2022-05-18T04:29:59.9391657Z Ran 1 test in 0.002s
2022-05-18T04:29:59.9391813Z 
2022-05-18T04:29:59.9391932Z OK (skipped=1)
2022-05-18T04:29:59.9392085Z 
2022-05-18T04:29:59.9392210Z Generating XML reports...
2022-05-18T04:29:59.9427671Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042959.xml
2022-05-18T04:30:01.1979396Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:01.1993130Z 
2022-05-18T04:30:01.1993458Z Running tests...
2022-05-18T04:30:01.1993905Z ----------------------------------------------------------------------
2022-05-18T04:30:01.2016963Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:30:01.2017792Z 
2022-05-18T04:30:01.2018105Z ----------------------------------------------------------------------
2022-05-18T04:30:01.2018452Z Ran 1 test in 0.002s
2022-05-18T04:30:01.2018925Z 
2022-05-18T04:30:01.2019039Z OK (skipped=1)
2022-05-18T04:30:01.2019203Z 
2022-05-18T04:30:01.2019332Z Generating XML reports...
2022-05-18T04:30:01.2053997Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043001.xml
2022-05-18T04:30:02.4332912Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:02.4347768Z 
2022-05-18T04:30:02.4348086Z Running tests...
2022-05-18T04:30:02.4348529Z ----------------------------------------------------------------------
2022-05-18T04:30:02.4371128Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:30:02.4371627Z 
2022-05-18T04:30:02.4372086Z ----------------------------------------------------------------------
2022-05-18T04:30:02.4372588Z Ran 1 test in 0.002s
2022-05-18T04:30:02.4372762Z 
2022-05-18T04:30:02.4372874Z OK (skipped=1)
2022-05-18T04:30:02.4373032Z 
2022-05-18T04:30:02.4373160Z Generating XML reports...
2022-05-18T04:30:02.4408766Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043002.xml
2022-05-18T04:30:03.6994760Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:03.7009031Z 
2022-05-18T04:30:03.7009187Z Running tests...
2022-05-18T04:30:03.7010024Z ----------------------------------------------------------------------
2022-05-18T04:30:03.7032936Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:30:03.7033443Z 
2022-05-18T04:30:03.7033734Z ----------------------------------------------------------------------
2022-05-18T04:30:03.7034069Z Ran 1 test in 0.002s
2022-05-18T04:30:03.7034237Z 
2022-05-18T04:30:03.7034349Z OK (skipped=1)
2022-05-18T04:30:03.7034507Z 
2022-05-18T04:30:03.7034636Z Generating XML reports...
2022-05-18T04:30:03.7069862Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043003.xml
2022-05-18T04:30:04.9618200Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:04.9633061Z 
2022-05-18T04:30:04.9633381Z Running tests...
2022-05-18T04:30:04.9633823Z ----------------------------------------------------------------------
2022-05-18T04:30:04.9656695Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:30:04.9657222Z 
2022-05-18T04:30:04.9657507Z ----------------------------------------------------------------------
2022-05-18T04:30:04.9657820Z Ran 1 test in 0.002s
2022-05-18T04:30:04.9657988Z 
2022-05-18T04:30:04.9658099Z OK (skipped=1)
2022-05-18T04:30:04.9658256Z 
2022-05-18T04:30:04.9658383Z Generating XML reports...
2022-05-18T04:30:04.9694043Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043004.xml
2022-05-18T04:30:06.2236352Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:06.2250827Z 
2022-05-18T04:30:06.2250975Z Running tests...
2022-05-18T04:30:06.2251867Z ----------------------------------------------------------------------
2022-05-18T04:30:06.2273134Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:30:06.2273955Z 
2022-05-18T04:30:06.2274467Z ----------------------------------------------------------------------
2022-05-18T04:30:06.2274794Z Ran 1 test in 0.002s
2022-05-18T04:30:06.2274960Z 
2022-05-18T04:30:06.2275071Z OK (skipped=1)
2022-05-18T04:30:06.2275230Z 
2022-05-18T04:30:06.2275448Z Generating XML reports...
2022-05-18T04:30:06.2310192Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043006.xml
2022-05-18T04:30:07.4824067Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:07.4838387Z 
2022-05-18T04:30:07.4838972Z Running tests...
2022-05-18T04:30:07.4839496Z ----------------------------------------------------------------------
2022-05-18T04:30:07.4860800Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-05-18T04:30:07.4861326Z 
2022-05-18T04:30:07.4861805Z ----------------------------------------------------------------------
2022-05-18T04:30:07.4862129Z Ran 1 test in 0.002s
2022-05-18T04:30:07.4862294Z 
2022-05-18T04:30:07.4862413Z OK (skipped=1)
2022-05-18T04:30:07.4862585Z 
2022-05-18T04:30:07.4862814Z Generating XML reports...
2022-05-18T04:30:07.4897724Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043007.xml
2022-05-18T04:30:08.7388748Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:08.7403147Z 
2022-05-18T04:30:08.7403503Z Running tests...
2022-05-18T04:30:08.7403957Z ----------------------------------------------------------------------
2022-05-18T04:30:10.3276764Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:10.3431804Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.602s)
2022-05-18T04:30:10.3432387Z 
2022-05-18T04:30:10.3432666Z ----------------------------------------------------------------------
2022-05-18T04:30:10.3433287Z Ran 1 test in 1.603s
2022-05-18T04:30:10.3433475Z 
2022-05-18T04:30:10.3433571Z OK (skipped=1)
2022-05-18T04:30:10.3433733Z 
2022-05-18T04:30:10.3433864Z Generating XML reports...
2022-05-18T04:30:10.3465823Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043008.xml
2022-05-18T04:30:11.7210586Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:11.7224445Z 
2022-05-18T04:30:11.7224683Z Running tests...
2022-05-18T04:30:11.7225128Z ----------------------------------------------------------------------
2022-05-18T04:30:13.3097935Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:13.3492376Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31479
2022-05-18T04:30:13.3597703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31480
2022-05-18T04:30:14.4827645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:14.4983203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:14.4996932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:14.5030698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:14.5036928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:14.5997449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:15.7492145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpucnb7wfl
2022-05-18T04:30:15.7492752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpucnb7wfl/_remote_module_non_scriptable.py
2022-05-18T04:30:15.8365547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dtrlyde
2022-05-18T04:30:15.8366593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dtrlyde/_remote_module_non_scriptable.py
2022-05-18T04:30:17.0693039Z ok (5.346s)
2022-05-18T04:30:17.0693230Z 
2022-05-18T04:30:17.0693623Z ----------------------------------------------------------------------
2022-05-18T04:30:17.0693973Z Ran 1 test in 5.347s
2022-05-18T04:30:17.0694143Z 
2022-05-18T04:30:17.0694249Z OK
2022-05-18T04:30:17.0694391Z 
2022-05-18T04:30:17.0694508Z Generating XML reports...
2022-05-18T04:30:17.0736400Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043011.xml
2022-05-18T04:30:18.4519386Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:18.4534886Z 
2022-05-18T04:30:18.4535047Z Running tests...
2022-05-18T04:30:18.4535773Z ----------------------------------------------------------------------
2022-05-18T04:30:20.0070293Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:20.0459021Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31595
2022-05-18T04:30:20.0559406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31596
2022-05-18T04:30:21.1824700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:21.2381192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:21.2382066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:21.2432030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:21.2439336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:21.3396250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:22.6525514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pju4jvg
2022-05-18T04:30:22.6530825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pju4jvg/_remote_module_non_scriptable.py
2022-05-18T04:30:22.7537068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7cbge6e
2022-05-18T04:30:22.7538274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7cbge6e/_remote_module_non_scriptable.py
2022-05-18T04:30:22.8429435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:22.8429973Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:22.8509736Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp4pju4jvg'>
2022-05-18T04:30:22.8510222Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:30:22.8510823Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpc7cbge6e'>
2022-05-18T04:30:22.8511262Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:30:23.1640509Z ok (4.710s)
2022-05-18T04:30:23.1640891Z 
2022-05-18T04:30:23.1641552Z ----------------------------------------------------------------------
2022-05-18T04:30:23.1641925Z Ran 1 test in 4.710s
2022-05-18T04:30:23.1642095Z 
2022-05-18T04:30:23.1642194Z OK
2022-05-18T04:30:23.1642606Z 
2022-05-18T04:30:23.1642766Z Generating XML reports...
2022-05-18T04:30:23.1682876Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043018.xml
2022-05-18T04:30:24.5651975Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:24.5665334Z 
2022-05-18T04:30:24.5666082Z Running tests...
2022-05-18T04:30:24.5666585Z ----------------------------------------------------------------------
2022-05-18T04:30:24.5739059Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.007s)
2022-05-18T04:30:24.5739667Z 
2022-05-18T04:30:24.5740062Z ----------------------------------------------------------------------
2022-05-18T04:30:24.5740405Z Ran 1 test in 0.007s
2022-05-18T04:30:24.5740572Z 
2022-05-18T04:30:24.5740684Z OK (skipped=1)
2022-05-18T04:30:24.5740843Z 
2022-05-18T04:30:24.5740952Z Generating XML reports...
2022-05-18T04:30:24.5774935Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043024.xml
2022-05-18T04:30:25.8064723Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:25.8079122Z 
2022-05-18T04:30:25.8079600Z Running tests...
2022-05-18T04:30:25.8080113Z ----------------------------------------------------------------------
2022-05-18T04:30:27.3685989Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:27.4072193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31748
2022-05-18T04:30:27.4176487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31749
2022-05-18T04:30:28.5789999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:28.6051932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:28.6052780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:28.6093883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:28.6099687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:28.7066535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:29.8787874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ia95u2a
2022-05-18T04:30:29.8788501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ia95u2a/_remote_module_non_scriptable.py
2022-05-18T04:30:29.9774627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprspnejk9
2022-05-18T04:30:29.9775634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprspnejk9/_remote_module_non_scriptable.py
2022-05-18T04:30:30.2301636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:30.2302204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:30:30.5257216Z ok (4.717s)
2022-05-18T04:30:30.5257615Z 
2022-05-18T04:30:30.5258282Z ----------------------------------------------------------------------
2022-05-18T04:30:30.5258901Z Ran 1 test in 4.718s
2022-05-18T04:30:30.5259206Z 
2022-05-18T04:30:30.5259375Z OK
2022-05-18T04:30:30.5259617Z 
2022-05-18T04:30:30.5259879Z Generating XML reports...
2022-05-18T04:30:30.5300872Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043025.xml
2022-05-18T04:30:31.9289413Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:31.9302755Z 
2022-05-18T04:30:31.9303180Z Running tests...
2022-05-18T04:30:31.9304017Z ----------------------------------------------------------------------
2022-05-18T04:30:33.4696916Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:33.5081962Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31868
2022-05-18T04:30:33.5187883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31869
2022-05-18T04:30:34.6707151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:34.6784999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:34.6785843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:34.6808494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:34.6814618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:34.7797339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:34.8008565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:30:34.8009517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:30:34.8010248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:30:34.8010951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:30:34.8012068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:30:34.8012572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:30:34.8013259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:30:34.8013958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:30:36.0873124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7r43ptm3
2022-05-18T04:30:36.0873948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7r43ptm3/_remote_module_non_scriptable.py
2022-05-18T04:30:36.1095648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6s0hwwu
2022-05-18T04:30:36.1098242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6s0hwwu/_remote_module_non_scriptable.py
2022-05-18T04:30:36.5266196Z ok (4.596s)
2022-05-18T04:30:36.5266392Z 
2022-05-18T04:30:36.5266785Z ----------------------------------------------------------------------
2022-05-18T04:30:36.5267141Z Ran 1 test in 4.596s
2022-05-18T04:30:36.5267309Z 
2022-05-18T04:30:36.5267407Z OK
2022-05-18T04:30:36.5267549Z 
2022-05-18T04:30:36.5267689Z Generating XML reports...
2022-05-18T04:30:36.5308070Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043031.xml
2022-05-18T04:30:37.9472975Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:37.9487698Z 
2022-05-18T04:30:37.9488012Z Running tests...
2022-05-18T04:30:37.9488689Z ----------------------------------------------------------------------
2022-05-18T04:30:39.5174184Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:39.5559916Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31992
2022-05-18T04:30:39.5663258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31993
2022-05-18T04:30:40.7418039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:40.7465587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:40.7466431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:40.7519450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:40.7525887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:40.8478204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:40.8646905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:30:40.8647719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:30:40.8648436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:30:40.8649115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:30:40.8650538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:30:40.8754558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:30:40.8755245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:30:40.8856133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:30:42.1610197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfn89o6xp
2022-05-18T04:30:42.1611102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfn89o6xp/_remote_module_non_scriptable.py
2022-05-18T04:30:42.1755470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xf41thw
2022-05-18T04:30:42.1758278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xf41thw/_remote_module_non_scriptable.py
2022-05-18T04:30:52.8452070Z [W ProcessGroupNCCL.cpp:865] [Rank 0] Found key in store: NCCLABORTEDCOMM:20c6fdac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-05-18T04:30:52.8453168Z [W ProcessGroupNCCL.cpp:865] [Rank 1] Found key in store: NCCLABORTEDCOMM:20c6fdac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-05-18T04:30:53.1935242Z ok (15.244s)
2022-05-18T04:30:53.1935475Z 
2022-05-18T04:30:53.1935854Z ----------------------------------------------------------------------
2022-05-18T04:30:53.1936200Z Ran 1 test in 15.245s
2022-05-18T04:30:53.1936367Z 
2022-05-18T04:30:53.1936493Z OK
2022-05-18T04:30:53.1936631Z 
2022-05-18T04:30:53.1936772Z Generating XML reports...
2022-05-18T04:30:53.1976985Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043037.xml
2022-05-18T04:30:54.5926518Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:30:54.5940559Z 
2022-05-18T04:30:54.5940849Z Running tests...
2022-05-18T04:30:54.5941274Z ----------------------------------------------------------------------
2022-05-18T04:30:56.1196157Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:56.1581451Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32116
2022-05-18T04:30:56.1684753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32117
2022-05-18T04:30:57.2993155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:57.3136621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:57.3137444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:57.3195809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:57.3201962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:57.4150428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:58.5611483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6lec2sc
2022-05-18T04:30:58.5612078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6lec2sc/_remote_module_non_scriptable.py
2022-05-18T04:30:58.6695772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46rnei_f
2022-05-18T04:30:58.6697066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46rnei_f/_remote_module_non_scriptable.py
2022-05-18T04:30:59.8778352Z ok (5.283s)
2022-05-18T04:30:59.8778614Z 
2022-05-18T04:30:59.8779263Z ----------------------------------------------------------------------
2022-05-18T04:30:59.8779642Z Ran 1 test in 5.284s
2022-05-18T04:30:59.8779815Z 
2022-05-18T04:30:59.8779921Z OK
2022-05-18T04:30:59.8780039Z 
2022-05-18T04:30:59.8780179Z Generating XML reports...
2022-05-18T04:30:59.8820127Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043054.xml
2022-05-18T04:31:01.2958606Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:01.2972619Z 
2022-05-18T04:31:01.2972881Z Running tests...
2022-05-18T04:31:01.2973328Z ----------------------------------------------------------------------
2022-05-18T04:31:02.8890529Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:02.9283818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32236
2022-05-18T04:31:02.9388629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32237
2022-05-18T04:31:04.1326821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:04.1571580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:04.1572397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:04.1630701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:04.1636876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:04.2585708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:05.4039185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlsh1k7l
2022-05-18T04:31:05.4040418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlsh1k7l/_remote_module_non_scriptable.py
2022-05-18T04:31:05.5223341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyluxwar2
2022-05-18T04:31:05.5224333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyluxwar2/_remote_module_non_scriptable.py
2022-05-18T04:31:06.6480881Z ok (5.350s)
2022-05-18T04:31:06.6481117Z 
2022-05-18T04:31:06.6481503Z ----------------------------------------------------------------------
2022-05-18T04:31:06.6481852Z Ran 1 test in 5.351s
2022-05-18T04:31:06.6482020Z 
2022-05-18T04:31:06.6482126Z OK
2022-05-18T04:31:06.6482263Z 
2022-05-18T04:31:06.6482398Z Generating XML reports...
2022-05-18T04:31:06.6522550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043101.xml
2022-05-18T04:31:08.0774582Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:08.0789012Z 
2022-05-18T04:31:08.0789286Z Running tests...
2022-05-18T04:31:08.0789738Z ----------------------------------------------------------------------
2022-05-18T04:31:09.6419309Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:09.6808460Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32356
2022-05-18T04:31:09.6912243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32357
2022-05-18T04:31:10.8324721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:10.8571118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:10.8571923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:10.8629208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:10.8635418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:10.9585932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:12.1170662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0v4buqeh
2022-05-18T04:31:12.1171783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0v4buqeh/_remote_module_non_scriptable.py
2022-05-18T04:31:12.1787843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30v40niv
2022-05-18T04:31:12.1789205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30v40niv/_remote_module_non_scriptable.py
2022-05-18T04:31:12.7993804Z ok (4.720s)
2022-05-18T04:31:12.7994031Z 
2022-05-18T04:31:12.7994431Z ----------------------------------------------------------------------
2022-05-18T04:31:12.7994758Z Ran 1 test in 4.720s
2022-05-18T04:31:12.7994933Z 
2022-05-18T04:31:12.7995043Z OK
2022-05-18T04:31:12.7995183Z 
2022-05-18T04:31:12.7995321Z Generating XML reports...
2022-05-18T04:31:12.8036299Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043108.xml
2022-05-18T04:31:14.2221147Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:14.2235393Z 
2022-05-18T04:31:14.2235828Z Running tests...
2022-05-18T04:31:14.2236349Z ----------------------------------------------------------------------
2022-05-18T04:31:15.8018349Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:15.8415331Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32472
2022-05-18T04:31:15.8520957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32473
2022-05-18T04:31:16.9825911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:17.0092737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:17.0093561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:17.0129727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:17.0136198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:17.1107438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:18.2723087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_wxestww
2022-05-18T04:31:18.2723958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_wxestww/_remote_module_non_scriptable.py
2022-05-18T04:31:18.3333256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23qwwy8z
2022-05-18T04:31:18.3334487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23qwwy8z/_remote_module_non_scriptable.py
2022-05-18T04:31:18.5660730Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:31:18.5720242Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:31:18.5890480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:18.5891047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:18.8601878Z ok (4.636s)
2022-05-18T04:31:18.8602110Z 
2022-05-18T04:31:18.8602516Z ----------------------------------------------------------------------
2022-05-18T04:31:18.8603115Z Ran 1 test in 4.637s
2022-05-18T04:31:18.8603317Z 
2022-05-18T04:31:18.8603423Z OK
2022-05-18T04:31:18.8603569Z 
2022-05-18T04:31:18.8603714Z Generating XML reports...
2022-05-18T04:31:18.8644191Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043114.xml
2022-05-18T04:31:20.2827516Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:20.2842402Z 
2022-05-18T04:31:20.2842817Z Running tests...
2022-05-18T04:31:20.2843321Z ----------------------------------------------------------------------
2022-05-18T04:31:21.8629510Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:21.9017073Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32592
2022-05-18T04:31:21.9123409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32593
2022-05-18T04:31:23.0693809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:23.0751349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:23.0752143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:23.0795230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:23.0801222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:23.1767574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:24.3153158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz46euq2m
2022-05-18T04:31:24.3153798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz46euq2m/_remote_module_non_scriptable.py
2022-05-18T04:31:24.4426111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsn31zjww
2022-05-18T04:31:24.4427245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsn31zjww/_remote_module_non_scriptable.py
2022-05-18T04:31:24.5170883Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:31:24.5171711Z   warnings.warn(
2022-05-18T04:31:24.5172777Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:31:24.5173491Z   warnings.warn(
2022-05-18T04:31:25.0205261Z ok (4.736s)
2022-05-18T04:31:25.0205580Z 
2022-05-18T04:31:25.0206030Z ----------------------------------------------------------------------
2022-05-18T04:31:25.0206380Z Ran 1 test in 4.736s
2022-05-18T04:31:25.0206549Z 
2022-05-18T04:31:25.0206645Z OK
2022-05-18T04:31:25.0206782Z 
2022-05-18T04:31:25.0206899Z Generating XML reports...
2022-05-18T04:31:25.0247659Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043120.xml
2022-05-18T04:31:26.4489538Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:26.4504731Z 
2022-05-18T04:31:26.4505238Z Running tests...
2022-05-18T04:31:26.4505759Z ----------------------------------------------------------------------
2022-05-18T04:31:28.0444667Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:28.0599585Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.609s)
2022-05-18T04:31:28.0600215Z 
2022-05-18T04:31:28.0600490Z ----------------------------------------------------------------------
2022-05-18T04:31:28.0600834Z Ran 1 test in 1.609s
2022-05-18T04:31:28.0601006Z 
2022-05-18T04:31:28.0601119Z OK (skipped=1)
2022-05-18T04:31:28.0601298Z 
2022-05-18T04:31:28.0601427Z Generating XML reports...
2022-05-18T04:31:28.0634594Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043126.xml
2022-05-18T04:31:29.4744673Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:29.4759983Z 
2022-05-18T04:31:29.4760287Z Running tests...
2022-05-18T04:31:29.4760724Z ----------------------------------------------------------------------
2022-05-18T04:31:31.0704720Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:31.1108609Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32746
2022-05-18T04:31:31.1215164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32747
2022-05-18T04:31:32.2887620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:32.3184839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:32.3185657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:32.3191254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:32.3197358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:32.4200166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:33.5886505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphy3jg_5m
2022-05-18T04:31:33.5887135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphy3jg_5m/_remote_module_non_scriptable.py
2022-05-18T04:31:33.6701551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4v1z1z48
2022-05-18T04:31:33.6702478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4v1z1z48/_remote_module_non_scriptable.py
2022-05-18T04:31:35.4035567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:35.4036624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:35.9808755Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:31:35.9811046Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:31:37.2352421Z ok (7.759s)
2022-05-18T04:31:37.2352823Z 
2022-05-18T04:31:37.2353473Z ----------------------------------------------------------------------
2022-05-18T04:31:37.2354080Z Ran 1 test in 7.759s
2022-05-18T04:31:37.2354391Z 
2022-05-18T04:31:37.2354559Z OK
2022-05-18T04:31:37.2354820Z 
2022-05-18T04:31:37.2355063Z Generating XML reports...
2022-05-18T04:31:37.2397341Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043129.xml
2022-05-18T04:31:38.6746794Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:38.6761711Z 
2022-05-18T04:31:38.6762231Z Running tests...
2022-05-18T04:31:38.6762864Z ----------------------------------------------------------------------
2022-05-18T04:31:40.2650086Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:40.3048683Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32870
2022-05-18T04:31:40.3155012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32871
2022-05-18T04:31:41.4239251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:41.4387861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:41.4389402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:41.4440785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:41.4447349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:41.5403565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:42.6939903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zcw49c7
2022-05-18T04:31:42.6941127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zcw49c7/_remote_module_non_scriptable.py
2022-05-18T04:31:42.7653968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3kqc2z3y
2022-05-18T04:31:42.7655120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3kqc2z3y/_remote_module_non_scriptable.py
2022-05-18T04:31:43.1230506Z ok (4.446s)
2022-05-18T04:31:43.1230740Z 
2022-05-18T04:31:43.1231376Z ----------------------------------------------------------------------
2022-05-18T04:31:43.1231763Z Ran 1 test in 4.447s
2022-05-18T04:31:43.1231929Z 
2022-05-18T04:31:43.1232013Z OK
2022-05-18T04:31:43.1232148Z 
2022-05-18T04:31:43.1232283Z Generating XML reports...
2022-05-18T04:31:43.1273013Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043138.xml
2022-05-18T04:31:44.5538546Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:44.5553064Z 
2022-05-18T04:31:44.5553228Z Running tests...
2022-05-18T04:31:44.5553687Z ----------------------------------------------------------------------
2022-05-18T04:31:46.1464249Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:46.1861537Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32986
2022-05-18T04:31:46.1966678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32987
2022-05-18T04:31:47.3572924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:47.3984366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:47.3985402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:47.4078943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:47.4085912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:47.4998161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:48.7148669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_x0qh2t
2022-05-18T04:31:48.7149998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_x0qh2t/_remote_module_non_scriptable.py
2022-05-18T04:31:48.7527160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozzvj3rb
2022-05-18T04:31:48.7529475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozzvj3rb/_remote_module_non_scriptable.py
2022-05-18T04:31:48.8368097Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:31:48.8368897Z   warnings.warn(
2022-05-18T04:31:48.8369933Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:31:48.8370997Z   warnings.warn(
2022-05-18T04:31:49.0277020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:49.0277558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:49.0333833Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:31:49.0335419Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:31:49.0440332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:49.0441017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:49.0504578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:49.0505090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:49.4050915Z ok (4.849s)
2022-05-18T04:31:49.4051131Z 
2022-05-18T04:31:49.4052115Z ----------------------------------------------------------------------
2022-05-18T04:31:49.4052500Z Ran 1 test in 4.850s
2022-05-18T04:31:49.4052672Z 
2022-05-18T04:31:49.4052753Z OK
2022-05-18T04:31:49.4052903Z 
2022-05-18T04:31:49.4053299Z Generating XML reports...
2022-05-18T04:31:49.4093976Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043144.xml
2022-05-18T04:31:50.8238830Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:50.8253588Z 
2022-05-18T04:31:50.8253721Z Running tests...
2022-05-18T04:31:50.8254489Z ----------------------------------------------------------------------
2022-05-18T04:31:52.4218384Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:52.4612134Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33106
2022-05-18T04:31:52.4718349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33107
2022-05-18T04:31:53.6011633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:53.6122726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:53.6123542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:53.6214298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:53.6220398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:53.7136845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:54.8792561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprn85rfut
2022-05-18T04:31:54.8793408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprn85rfut/_remote_module_non_scriptable.py
2022-05-18T04:31:54.9526544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9pry6ef7
2022-05-18T04:31:54.9527874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9pry6ef7/_remote_module_non_scriptable.py
2022-05-18T04:31:55.0305131Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:31:55.0306014Z   warnings.warn(
2022-05-18T04:31:55.0307072Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:31:55.0307800Z   warnings.warn(
2022-05-18T04:31:55.2187540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:55.2188079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:55.5801244Z ok (4.754s)
2022-05-18T04:31:55.5801925Z 
2022-05-18T04:31:55.5802340Z ----------------------------------------------------------------------
2022-05-18T04:31:55.5802710Z Ran 1 test in 4.755s
2022-05-18T04:31:55.5802877Z 
2022-05-18T04:31:55.5802956Z OK
2022-05-18T04:31:55.5803095Z 
2022-05-18T04:31:55.5803227Z Generating XML reports...
2022-05-18T04:31:55.5843508Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043150.xml
2022-05-18T04:31:56.9980725Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:56.9995254Z 
2022-05-18T04:31:56.9995502Z Running tests...
2022-05-18T04:31:56.9996209Z ----------------------------------------------------------------------
2022-05-18T04:31:58.5785485Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:58.5934945Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.594s)
2022-05-18T04:31:58.5935517Z 
2022-05-18T04:31:58.5935796Z ----------------------------------------------------------------------
2022-05-18T04:31:58.5936133Z Ran 1 test in 1.594s
2022-05-18T04:31:58.5936298Z 
2022-05-18T04:31:58.5936410Z OK (skipped=1)
2022-05-18T04:31:58.5936569Z 
2022-05-18T04:31:58.5936695Z Generating XML reports...
2022-05-18T04:31:58.5968169Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043156.xml
2022-05-18T04:31:59.9267424Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:31:59.9283393Z 
2022-05-18T04:31:59.9283822Z Running tests...
2022-05-18T04:31:59.9284337Z ----------------------------------------------------------------------
2022-05-18T04:32:01.5205350Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:01.5611943Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33260
2022-05-18T04:32:01.5719542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33261
2022-05-18T04:32:02.6910248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:02.7070777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:02.7071600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:02.7112583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:02.7118761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:02.8086712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:03.9730315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0543ea35
2022-05-18T04:32:03.9731013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0543ea35/_remote_module_non_scriptable.py
2022-05-18T04:32:04.0624950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstpu1sie
2022-05-18T04:32:04.0626331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstpu1sie/_remote_module_non_scriptable.py
2022-05-18T04:32:04.4381930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:04.4382493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:05.4817965Z ok (5.553s)
2022-05-18T04:32:05.4818156Z 
2022-05-18T04:32:05.4818573Z ----------------------------------------------------------------------
2022-05-18T04:32:05.4818924Z Ran 1 test in 5.553s
2022-05-18T04:32:05.4819090Z 
2022-05-18T04:32:05.4819190Z OK
2022-05-18T04:32:05.4819326Z 
2022-05-18T04:32:05.4819444Z Generating XML reports...
2022-05-18T04:32:05.4860055Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043159.xml
2022-05-18T04:32:06.9069367Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:06.9083624Z 
2022-05-18T04:32:06.9083842Z Running tests...
2022-05-18T04:32:06.9084289Z ----------------------------------------------------------------------
2022-05-18T04:32:08.4748028Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:08.5145651Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33384
2022-05-18T04:32:08.5251555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33385
2022-05-18T04:32:09.6577065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:09.6841464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:09.6842292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:09.6880660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:09.6886957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:09.7856275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:10.9195993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pxmjs40
2022-05-18T04:32:10.9196603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pxmjs40/_remote_module_non_scriptable.py
2022-05-18T04:32:11.0324472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1y92cbe
2022-05-18T04:32:11.0325398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1y92cbe/_remote_module_non_scriptable.py
2022-05-18T04:32:11.4329588Z ok (4.524s)
2022-05-18T04:32:11.4329804Z 
2022-05-18T04:32:11.4330433Z ----------------------------------------------------------------------
2022-05-18T04:32:11.4331039Z Ran 1 test in 4.525s
2022-05-18T04:32:11.4331215Z 
2022-05-18T04:32:11.4331313Z OK
2022-05-18T04:32:11.4331452Z 
2022-05-18T04:32:11.4331587Z Generating XML reports...
2022-05-18T04:32:11.4379159Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043206.xml
2022-05-18T04:32:12.8527324Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:12.8541694Z 
2022-05-18T04:32:12.8541962Z Running tests...
2022-05-18T04:32:12.8542409Z ----------------------------------------------------------------------
2022-05-18T04:32:14.4267609Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:14.4662030Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33500
2022-05-18T04:32:14.4768249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33501
2022-05-18T04:32:15.5906519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:15.6026794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:15.6027671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:15.6108855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:15.6114854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:15.7041819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:16.8406185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxlh7khze
2022-05-18T04:32:16.8406804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxlh7khze/_remote_module_non_scriptable.py
2022-05-18T04:32:16.9275084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5joyfb7
2022-05-18T04:32:16.9276529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5joyfb7/_remote_module_non_scriptable.py
2022-05-18T04:32:17.2844182Z ok (4.430s)
2022-05-18T04:32:17.2844569Z 
2022-05-18T04:32:17.2845072Z ----------------------------------------------------------------------
2022-05-18T04:32:17.2845427Z Ran 1 test in 4.430s
2022-05-18T04:32:17.2845594Z 
2022-05-18T04:32:17.2845699Z OK
2022-05-18T04:32:17.2845836Z 
2022-05-18T04:32:17.2845970Z Generating XML reports...
2022-05-18T04:32:17.2886545Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043212.xml
2022-05-18T04:32:18.7026423Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:18.7041585Z 
2022-05-18T04:32:18.7041861Z Running tests...
2022-05-18T04:32:18.7042311Z ----------------------------------------------------------------------
2022-05-18T04:32:20.2997323Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:20.3396132Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33616
2022-05-18T04:32:20.3501463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33617
2022-05-18T04:32:21.5190324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:21.5435478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:21.5436301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:21.5494890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:21.5501406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:21.6450734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:22.8131717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplf8qv14q
2022-05-18T04:32:22.8135606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplf8qv14q/_remote_module_non_scriptable.py
2022-05-18T04:32:22.9234430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4orglg91
2022-05-18T04:32:22.9235284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4orglg91/_remote_module_non_scriptable.py
2022-05-18T04:32:23.1729569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:23.1730103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:23.4585977Z ok (4.754s)
2022-05-18T04:32:23.4586195Z 
2022-05-18T04:32:23.4586590Z ----------------------------------------------------------------------
2022-05-18T04:32:23.4586936Z Ran 1 test in 4.754s
2022-05-18T04:32:23.4587103Z 
2022-05-18T04:32:23.4589758Z OK
2022-05-18T04:32:23.4590175Z 
2022-05-18T04:32:23.4590443Z Generating XML reports...
2022-05-18T04:32:23.4628651Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043218.xml
2022-05-18T04:32:24.8536431Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:24.8550093Z 
2022-05-18T04:32:24.8550284Z Running tests...
2022-05-18T04:32:24.8550731Z ----------------------------------------------------------------------
2022-05-18T04:32:26.3819563Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:26.3968386Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.541s)
2022-05-18T04:32:26.3969076Z 
2022-05-18T04:32:26.3969614Z ----------------------------------------------------------------------
2022-05-18T04:32:26.3969975Z Ran 1 test in 1.542s
2022-05-18T04:32:26.3970139Z 
2022-05-18T04:32:26.3970250Z OK (skipped=1)
2022-05-18T04:32:26.3970407Z 
2022-05-18T04:32:26.3970533Z Generating XML reports...
2022-05-18T04:32:26.4001583Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043224.xml
2022-05-18T04:32:27.7590549Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:27.7608041Z 
2022-05-18T04:32:27.7608461Z Running tests...
2022-05-18T04:32:27.7608975Z ----------------------------------------------------------------------
2022-05-18T04:32:29.3376604Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:29.3771065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33770
2022-05-18T04:32:29.3876032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33771
2022-05-18T04:32:30.5129139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:30.5570863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:30.5572212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:30.5635534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:30.5641879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:30.6586386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:31.8214577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnbe6vyb
2022-05-18T04:32:31.8215186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnbe6vyb/_remote_module_non_scriptable.py
2022-05-18T04:32:31.9093809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgslunjr
2022-05-18T04:32:31.9095405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgslunjr/_remote_module_non_scriptable.py
2022-05-18T04:32:32.1757545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:32.1758315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:32.1980966Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:32.1981488Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:32.2073813Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpgnbe6vyb'>
2022-05-18T04:32:32.2074310Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:32:32.4959826Z ok (4.735s)
2022-05-18T04:32:32.4960121Z 
2022-05-18T04:32:32.4960494Z ----------------------------------------------------------------------
2022-05-18T04:32:32.4960834Z Ran 1 test in 4.735s
2022-05-18T04:32:32.4961002Z 
2022-05-18T04:32:32.4961106Z OK
2022-05-18T04:32:32.4961243Z 
2022-05-18T04:32:32.4961378Z Generating XML reports...
2022-05-18T04:32:32.5002056Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043227.xml
2022-05-18T04:32:33.9128662Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:33.9143387Z 
2022-05-18T04:32:33.9143687Z Running tests...
2022-05-18T04:32:33.9144118Z ----------------------------------------------------------------------
2022-05-18T04:32:35.5014186Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:35.5410329Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33890
2022-05-18T04:32:35.5516323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33891
2022-05-18T04:32:36.7196021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:36.7207817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:36.7208609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:36.7297399Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:36.7303414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:36.8221452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:37.9631760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqn5q5oew
2022-05-18T04:32:37.9633015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqn5q5oew/_remote_module_non_scriptable.py
2022-05-18T04:32:38.0358278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8k4uctk
2022-05-18T04:32:38.0359527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8k4uctk/_remote_module_non_scriptable.py
2022-05-18T04:32:38.5596468Z ok (4.645s)
2022-05-18T04:32:38.5596676Z 
2022-05-18T04:32:38.5597039Z ----------------------------------------------------------------------
2022-05-18T04:32:38.5597383Z Ran 1 test in 4.645s
2022-05-18T04:32:38.5597790Z 
2022-05-18T04:32:38.5597893Z OK
2022-05-18T04:32:38.5598031Z 
2022-05-18T04:32:38.5598444Z Generating XML reports...
2022-05-18T04:32:38.5637950Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043233.xml
2022-05-18T04:32:39.9879494Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:39.9895088Z 
2022-05-18T04:32:39.9895361Z Running tests...
2022-05-18T04:32:39.9895812Z ----------------------------------------------------------------------
2022-05-18T04:32:41.5963834Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:41.6359004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34010
2022-05-18T04:32:41.6463958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34011
2022-05-18T04:32:42.7821589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:42.8294666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:42.8295467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:42.8328330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:42.8335883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:42.8339013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:32:42.9305969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:42.9309190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:32:42.9309893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:32:42.9358002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:32:43.2520241Z ok (3.262s)
2022-05-18T04:32:43.2520474Z 
2022-05-18T04:32:43.2521140Z ----------------------------------------------------------------------
2022-05-18T04:32:43.2521517Z Ran 1 test in 3.262s
2022-05-18T04:32:43.2521684Z 
2022-05-18T04:32:43.2521782Z OK
2022-05-18T04:32:43.2521919Z 
2022-05-18T04:32:43.2522064Z Generating XML reports...
2022-05-18T04:32:43.2563486Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043239.xml
2022-05-18T04:32:44.6484685Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:44.6498401Z 
2022-05-18T04:32:44.6498685Z Running tests...
2022-05-18T04:32:44.6499120Z ----------------------------------------------------------------------
2022-05-18T04:32:46.1804638Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:46.2188739Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34121
2022-05-18T04:32:46.2293128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34122
2022-05-18T04:32:47.3536595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:47.3663577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:47.3664801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:47.3741189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:47.3747302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:47.3750573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:32:47.4675202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:47.4678815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:32:47.4679820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:32:47.4770667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:32:47.7345035Z ok (3.084s)
2022-05-18T04:32:47.7345261Z 
2022-05-18T04:32:47.7345650Z ----------------------------------------------------------------------
2022-05-18T04:32:47.7346003Z Ran 1 test in 3.085s
2022-05-18T04:32:47.7346174Z 
2022-05-18T04:32:47.7346295Z OK
2022-05-18T04:32:47.7347413Z 
2022-05-18T04:32:47.7349992Z Generating XML reports...
2022-05-18T04:32:47.7389890Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043244.xml
2022-05-18T04:32:49.1505272Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:49.1520124Z 
2022-05-18T04:32:49.1520424Z Running tests...
2022-05-18T04:32:49.1520875Z ----------------------------------------------------------------------
2022-05-18T04:32:50.7255485Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:50.7641357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34232
2022-05-18T04:32:50.7746339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34233
2022-05-18T04:32:51.9167405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:51.9368926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:51.9369759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:51.9370672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:51.9377414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:51.9378052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:53.2117939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnimjxpyf
2022-05-18T04:32:53.2118903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnimjxpyf/_remote_module_non_scriptable.py
2022-05-18T04:32:53.2524233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkmujel4
2022-05-18T04:32:53.2526790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkmujel4/_remote_module_non_scriptable.py
2022-05-18T04:32:53.5078440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:53.5079279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:53.5142669Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:32:53.5144249Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:32:53.7826692Z ok (4.630s)
2022-05-18T04:32:53.7827044Z 
2022-05-18T04:32:53.7827791Z ----------------------------------------------------------------------
2022-05-18T04:32:53.7828450Z Ran 1 test in 4.631s
2022-05-18T04:32:53.7828626Z 
2022-05-18T04:32:53.7828725Z OK
2022-05-18T04:32:53.7828865Z 
2022-05-18T04:32:53.7829003Z Generating XML reports...
2022-05-18T04:32:53.7868895Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043249.xml
2022-05-18T04:32:55.1993129Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:32:55.2007242Z 
2022-05-18T04:32:55.2007737Z Running tests...
2022-05-18T04:32:55.2008695Z ----------------------------------------------------------------------
2022-05-18T04:32:56.7405972Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:56.7793481Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34352
2022-05-18T04:32:56.7900618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34353
2022-05-18T04:32:57.9175439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:57.9431085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:57.9431948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:57.9479928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:57.9486791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:58.0445580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:59.2160628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajr0bwqz
2022-05-18T04:32:59.2162159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajr0bwqz/_remote_module_non_scriptable.py
2022-05-18T04:32:59.3079463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp40g01abl
2022-05-18T04:32:59.3081799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp40g01abl/_remote_module_non_scriptable.py
2022-05-18T04:32:59.5471159Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:32:59.5705106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:59.5705642Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:59.8981415Z ok (4.697s)
2022-05-18T04:32:59.8981723Z 
2022-05-18T04:32:59.8982133Z ----------------------------------------------------------------------
2022-05-18T04:32:59.8982472Z Ran 1 test in 4.697s
2022-05-18T04:32:59.8982642Z 
2022-05-18T04:32:59.8982744Z OK
2022-05-18T04:32:59.8982880Z 
2022-05-18T04:32:59.8983315Z Generating XML reports...
2022-05-18T04:32:59.9023178Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043255.xml
2022-05-18T04:33:01.3169393Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:01.3184475Z 
2022-05-18T04:33:01.3185127Z Running tests...
2022-05-18T04:33:01.3185740Z ----------------------------------------------------------------------
2022-05-18T04:33:02.8950582Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:02.9345699Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34472
2022-05-18T04:33:02.9451073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34473
2022-05-18T04:33:04.0572820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:04.0894609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:04.0895726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:04.0979147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:04.0985487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:04.1906264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:04.3501933Z ok (3.031s)
2022-05-18T04:33:04.3502168Z 
2022-05-18T04:33:04.3502548Z ----------------------------------------------------------------------
2022-05-18T04:33:04.3503139Z Ran 1 test in 3.032s
2022-05-18T04:33:04.3503377Z 
2022-05-18T04:33:04.3503479Z OK
2022-05-18T04:33:04.3503620Z 
2022-05-18T04:33:04.3503761Z Generating XML reports...
2022-05-18T04:33:04.3544832Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043301.xml
2022-05-18T04:33:05.7548046Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:05.7563084Z 
2022-05-18T04:33:05.7563239Z Running tests...
2022-05-18T04:33:05.7564362Z ----------------------------------------------------------------------
2022-05-18T04:33:05.7583107Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:33:05.7583420Z 
2022-05-18T04:33:05.7583713Z ----------------------------------------------------------------------
2022-05-18T04:33:05.7584059Z Ran 1 test in 0.002s
2022-05-18T04:33:05.7584225Z 
2022-05-18T04:33:05.7584340Z OK (skipped=1)
2022-05-18T04:33:05.7584500Z 
2022-05-18T04:33:05.7584609Z Generating XML reports...
2022-05-18T04:33:05.7621452Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043305.xml
2022-05-18T04:33:07.0073073Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:07.0089781Z 
2022-05-18T04:33:07.0090146Z Running tests...
2022-05-18T04:33:07.0090575Z ----------------------------------------------------------------------
2022-05-18T04:33:07.0117622Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.003s)
2022-05-18T04:33:07.0118456Z 
2022-05-18T04:33:07.0118758Z ----------------------------------------------------------------------
2022-05-18T04:33:07.0119107Z Ran 1 test in 0.003s
2022-05-18T04:33:07.0119254Z 
2022-05-18T04:33:07.0119367Z OK (skipped=1)
2022-05-18T04:33:07.0119525Z 
2022-05-18T04:33:07.0119653Z Generating XML reports...
2022-05-18T04:33:07.0155748Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043306.xml
2022-05-18T04:33:08.2435064Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:08.2449308Z 
2022-05-18T04:33:08.2449660Z Running tests...
2022-05-18T04:33:08.2450097Z ----------------------------------------------------------------------
2022-05-18T04:33:09.7828412Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:09.8214709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34645
2022-05-18T04:33:09.8320799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34646
2022-05-18T04:33:10.9698751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:10.9914048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:10.9914862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:11.0005257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:11.0012153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:11.0928508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:13.1406116Z ok (4.895s)
2022-05-18T04:33:13.1406433Z 
2022-05-18T04:33:13.1407068Z ----------------------------------------------------------------------
2022-05-18T04:33:13.1407452Z Ran 1 test in 4.896s
2022-05-18T04:33:13.1407634Z 
2022-05-18T04:33:13.1407731Z OK
2022-05-18T04:33:13.1407850Z 
2022-05-18T04:33:13.1407988Z Generating XML reports...
2022-05-18T04:33:13.1448412Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043308.xml
2022-05-18T04:33:14.5367903Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:14.5381555Z 
2022-05-18T04:33:14.5381877Z Running tests...
2022-05-18T04:33:14.5382330Z ----------------------------------------------------------------------
2022-05-18T04:33:14.5400303Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:33:14.5401059Z 
2022-05-18T04:33:14.5401663Z ----------------------------------------------------------------------
2022-05-18T04:33:14.5402042Z Ran 1 test in 0.002s
2022-05-18T04:33:14.5402211Z 
2022-05-18T04:33:14.5402325Z OK (skipped=1)
2022-05-18T04:33:14.5402463Z 
2022-05-18T04:33:14.5402591Z Generating XML reports...
2022-05-18T04:33:14.5435687Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043314.xml
2022-05-18T04:33:15.7857275Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:15.7871449Z 
2022-05-18T04:33:15.7871605Z Running tests...
2022-05-18T04:33:15.7872276Z ----------------------------------------------------------------------
2022-05-18T04:33:15.7892207Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:33:15.7892523Z 
2022-05-18T04:33:15.7892817Z ----------------------------------------------------------------------
2022-05-18T04:33:15.7893162Z Ran 1 test in 0.002s
2022-05-18T04:33:15.7893332Z 
2022-05-18T04:33:15.7893444Z OK (skipped=1)
2022-05-18T04:33:15.7893604Z 
2022-05-18T04:33:15.7894435Z Generating XML reports...
2022-05-18T04:33:15.7930459Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043315.xml
2022-05-18T04:33:17.0406633Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:17.0421323Z 
2022-05-18T04:33:17.0421684Z Running tests...
2022-05-18T04:33:17.0422126Z ----------------------------------------------------------------------
2022-05-18T04:33:18.6218059Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:18.6611674Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34831
2022-05-18T04:33:18.6717749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34832
2022-05-18T04:33:19.8492745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:19.8525195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:19.8526035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:19.8594055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:19.8600498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:19.9540178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:22.6815863Z ok (5.639s)
2022-05-18T04:33:22.6816097Z 
2022-05-18T04:33:22.6816500Z ----------------------------------------------------------------------
2022-05-18T04:33:22.6816824Z Ran 1 test in 5.639s
2022-05-18T04:33:22.6816994Z 
2022-05-18T04:33:22.6817109Z OK
2022-05-18T04:33:22.6817258Z 
2022-05-18T04:33:22.6817398Z Generating XML reports...
2022-05-18T04:33:22.6857250Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043317.xml
2022-05-18T04:33:24.1032693Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:24.1048013Z 
2022-05-18T04:33:24.1048464Z Running tests...
2022-05-18T04:33:24.1048957Z ----------------------------------------------------------------------
2022-05-18T04:33:25.6890007Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:25.7283974Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34949
2022-05-18T04:33:25.7389516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34950
2022-05-18T04:33:26.8958501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:26.9208114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:26.9208951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:26.9261623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:26.9267499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:27.0219711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:27.0387816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:27.0388600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:27.0389336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:33:27.0390018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:33:29.4887288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:33:29.4888147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:33:29.4888939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:33:29.4889651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:33:29.5293511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:33:29.5294550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:33:29.5295329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:33:29.5296026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:33:29.9490389Z ok (5.844s)
2022-05-18T04:33:29.9490590Z 
2022-05-18T04:33:29.9490983Z ----------------------------------------------------------------------
2022-05-18T04:33:29.9491328Z Ran 1 test in 5.844s
2022-05-18T04:33:29.9491497Z 
2022-05-18T04:33:29.9491597Z OK
2022-05-18T04:33:29.9492839Z 
2022-05-18T04:33:29.9493293Z Generating XML reports...
2022-05-18T04:33:29.9532151Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043324.xml
2022-05-18T04:33:31.3757885Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:31.3772874Z 
2022-05-18T04:33:31.3773177Z Running tests...
2022-05-18T04:33:31.3773925Z ----------------------------------------------------------------------
2022-05-18T04:33:32.9390381Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:32.9776896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35106
2022-05-18T04:33:32.9885013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35107
2022-05-18T04:33:34.1490745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:34.1766221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:34.1767064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:34.1794520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:34.1800318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:34.1803401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:34.2779112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:34.2782794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:34.2783506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:33:34.2821911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:33:34.5940520Z ok (3.216s)
2022-05-18T04:33:34.5940736Z 
2022-05-18T04:33:34.5941123Z ----------------------------------------------------------------------
2022-05-18T04:33:34.5941470Z Ran 1 test in 3.217s
2022-05-18T04:33:34.5941635Z 
2022-05-18T04:33:34.5941737Z OK
2022-05-18T04:33:34.5941872Z 
2022-05-18T04:33:34.5942008Z Generating XML reports...
2022-05-18T04:33:34.5990470Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043331.xml
2022-05-18T04:33:35.9943249Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:35.9956876Z 
2022-05-18T04:33:35.9957286Z Running tests...
2022-05-18T04:33:35.9957792Z ----------------------------------------------------------------------
2022-05-18T04:33:37.5374088Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:37.5759572Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35217
2022-05-18T04:33:37.5868144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35218
2022-05-18T04:33:38.7705472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:38.7776847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:38.7777639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:38.7806900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:38.7812679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:38.8791463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:40.4943415Z ok (4.498s)
2022-05-18T04:33:40.4943632Z 
2022-05-18T04:33:40.4944030Z ----------------------------------------------------------------------
2022-05-18T04:33:40.4944384Z Ran 1 test in 4.499s
2022-05-18T04:33:40.4944555Z 
2022-05-18T04:33:40.4944655Z OK
2022-05-18T04:33:40.4944810Z 
2022-05-18T04:33:40.4944951Z Generating XML reports...
2022-05-18T04:33:40.4985649Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043335.xml
2022-05-18T04:33:41.9240555Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:41.9255623Z 
2022-05-18T04:33:41.9255946Z Running tests...
2022-05-18T04:33:41.9256378Z ----------------------------------------------------------------------
2022-05-18T04:33:43.4930149Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:43.5327395Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35333
2022-05-18T04:33:43.5433366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35334
2022-05-18T04:33:44.6696120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:44.6804878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:44.6805682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:44.6898314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:44.6904559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:44.7816423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:45.1487939Z ok (3.223s)
2022-05-18T04:33:45.1488164Z 
2022-05-18T04:33:45.1488561Z ----------------------------------------------------------------------
2022-05-18T04:33:45.1488910Z Ran 1 test in 3.223s
2022-05-18T04:33:45.1489081Z 
2022-05-18T04:33:45.1489158Z OK
2022-05-18T04:33:45.1489294Z 
2022-05-18T04:33:45.1489428Z Generating XML reports...
2022-05-18T04:33:45.1531564Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043341.xml
2022-05-18T04:33:46.5667263Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:46.5682131Z 
2022-05-18T04:33:46.5682494Z Running tests...
2022-05-18T04:33:46.5682989Z ----------------------------------------------------------------------
2022-05-18T04:33:48.1467608Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:48.1861555Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35440
2022-05-18T04:33:48.1967057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35441
2022-05-18T04:33:49.3477869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:49.3748749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:49.3749561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:49.3781913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:49.3788503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:49.3791396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:49.4760640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:49.4765228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:49.4765931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:33:49.4810205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:33:49.7020016Z ok (3.133s)
2022-05-18T04:33:49.7020234Z 
2022-05-18T04:33:49.7020623Z ----------------------------------------------------------------------
2022-05-18T04:33:49.7020944Z Ran 1 test in 3.134s
2022-05-18T04:33:49.7021120Z 
2022-05-18T04:33:49.7021215Z OK
2022-05-18T04:33:49.7021350Z 
2022-05-18T04:33:49.7021483Z Generating XML reports...
2022-05-18T04:33:49.7062772Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043346.xml
2022-05-18T04:33:51.1005788Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:51.1020300Z 
2022-05-18T04:33:51.1020575Z Running tests...
2022-05-18T04:33:51.1021032Z ----------------------------------------------------------------------
2022-05-18T04:33:52.6587685Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:52.6983635Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35551
2022-05-18T04:33:52.7089023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35552
2022-05-18T04:33:53.8526979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:53.8847565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:53.8848389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:53.8932171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:53.8938505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:53.8941634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:53.9858704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:53.9862186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:53.9862924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:33:53.9960232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:33:54.2141666Z ok (3.112s)
2022-05-18T04:33:54.2142203Z 
2022-05-18T04:33:54.2142630Z ----------------------------------------------------------------------
2022-05-18T04:33:54.2142978Z Ran 1 test in 3.112s
2022-05-18T04:33:54.2143159Z 
2022-05-18T04:33:54.2143260Z OK
2022-05-18T04:33:54.2143400Z 
2022-05-18T04:33:54.2143535Z Generating XML reports...
2022-05-18T04:33:54.2184133Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043351.xml
2022-05-18T04:33:55.6194589Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:33:55.6209541Z 
2022-05-18T04:33:55.6209851Z Running tests...
2022-05-18T04:33:55.6210310Z ----------------------------------------------------------------------
2022-05-18T04:33:57.1939660Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:57.2327033Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35662
2022-05-18T04:33:57.2432481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35663
2022-05-18T04:33:58.4045435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:58.4310372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:58.4311198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:58.4349446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:58.4355511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:58.5325389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:59.6676453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8faqpdr
2022-05-18T04:33:59.6677127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8faqpdr/_remote_module_non_scriptable.py
2022-05-18T04:33:59.8019317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ecjj4r5
2022-05-18T04:33:59.8020630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ecjj4r5/_remote_module_non_scriptable.py
2022-05-18T04:34:00.3512818Z ok (4.730s)
2022-05-18T04:34:00.3513042Z 
2022-05-18T04:34:00.3513435Z ----------------------------------------------------------------------
2022-05-18T04:34:00.3513794Z Ran 1 test in 4.730s
2022-05-18T04:34:00.3513964Z 
2022-05-18T04:34:00.3514041Z OK
2022-05-18T04:34:00.3514178Z 
2022-05-18T04:34:00.3514316Z Generating XML reports...
2022-05-18T04:34:00.3555840Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043355.xml
2022-05-18T04:34:01.7903877Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:01.7918407Z 
2022-05-18T04:34:01.7918930Z Running tests...
2022-05-18T04:34:01.7919472Z ----------------------------------------------------------------------
2022-05-18T04:34:01.7945126Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2022-05-18T04:34:01.7945431Z 
2022-05-18T04:34:01.7945738Z ----------------------------------------------------------------------
2022-05-18T04:34:01.7946086Z Ran 1 test in 0.003s
2022-05-18T04:34:01.7946256Z 
2022-05-18T04:34:01.7946370Z OK (skipped=1)
2022-05-18T04:34:01.7947594Z 
2022-05-18T04:34:01.7948050Z Generating XML reports...
2022-05-18T04:34:01.7983196Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043401.xml
2022-05-18T04:34:03.0468342Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:03.0482528Z 
2022-05-18T04:34:03.0483022Z Running tests...
2022-05-18T04:34:03.0483506Z ----------------------------------------------------------------------
2022-05-18T04:34:03.0501051Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-05-18T04:34:03.0501364Z 
2022-05-18T04:34:03.0501664Z ----------------------------------------------------------------------
2022-05-18T04:34:03.0502016Z Ran 1 test in 0.002s
2022-05-18T04:34:03.0502165Z 
2022-05-18T04:34:03.0502279Z OK (skipped=1)
2022-05-18T04:34:03.0502439Z 
2022-05-18T04:34:03.0503343Z Generating XML reports...
2022-05-18T04:34:03.0536725Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043403.xml
2022-05-18T04:34:04.3030278Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:04.3045625Z 
2022-05-18T04:34:04.3046092Z Running tests...
2022-05-18T04:34:04.3046591Z ----------------------------------------------------------------------
2022-05-18T04:34:04.3064736Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-05-18T04:34:04.3065083Z 
2022-05-18T04:34:04.3065511Z ----------------------------------------------------------------------
2022-05-18T04:34:04.3065982Z Ran 1 test in 0.002s
2022-05-18T04:34:04.3066150Z 
2022-05-18T04:34:04.3066265Z OK (skipped=1)
2022-05-18T04:34:04.3066438Z 
2022-05-18T04:34:04.3066565Z Generating XML reports...
2022-05-18T04:34:04.3102349Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043404.xml
2022-05-18T04:34:05.5758481Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:05.5772793Z 
2022-05-18T04:34:05.5773277Z Running tests...
2022-05-18T04:34:05.5773786Z ----------------------------------------------------------------------
2022-05-18T04:34:05.5793089Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-05-18T04:34:05.5793539Z 
2022-05-18T04:34:05.5794066Z ----------------------------------------------------------------------
2022-05-18T04:34:05.5794442Z Ran 1 test in 0.002s
2022-05-18T04:34:05.5794610Z 
2022-05-18T04:34:05.5794722Z OK (skipped=1)
2022-05-18T04:34:05.5794880Z 
2022-05-18T04:34:05.5795009Z Generating XML reports...
2022-05-18T04:34:05.5830490Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043405.xml
2022-05-18T04:34:06.8458801Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:06.8472992Z 
2022-05-18T04:34:06.8473203Z Running tests...
2022-05-18T04:34:06.8473687Z ----------------------------------------------------------------------
2022-05-18T04:34:06.8494026Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T04:34:06.8494418Z 
2022-05-18T04:34:06.8494769Z ----------------------------------------------------------------------
2022-05-18T04:34:06.8495092Z Ran 1 test in 0.002s
2022-05-18T04:34:06.8495259Z 
2022-05-18T04:34:06.8495374Z OK (skipped=1)
2022-05-18T04:34:06.8495536Z 
2022-05-18T04:34:06.8495664Z Generating XML reports...
2022-05-18T04:34:06.8531001Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043406.xml
2022-05-18T04:34:08.1141265Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:08.1156022Z 
2022-05-18T04:34:08.1156341Z Running tests...
2022-05-18T04:34:08.1156778Z ----------------------------------------------------------------------
2022-05-18T04:34:08.1177087Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T04:34:08.1177475Z 
2022-05-18T04:34:08.1177762Z ----------------------------------------------------------------------
2022-05-18T04:34:08.1178405Z Ran 1 test in 0.002s
2022-05-18T04:34:08.1178574Z 
2022-05-18T04:34:08.1178668Z OK (skipped=1)
2022-05-18T04:34:08.1178828Z 
2022-05-18T04:34:08.1178955Z Generating XML reports...
2022-05-18T04:34:08.1213915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043408.xml
2022-05-18T04:34:09.3809258Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:09.3823884Z 
2022-05-18T04:34:09.3824206Z Running tests...
2022-05-18T04:34:09.3824666Z ----------------------------------------------------------------------
2022-05-18T04:34:09.3848359Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:34:09.3848713Z 
2022-05-18T04:34:09.3848986Z ----------------------------------------------------------------------
2022-05-18T04:34:09.3849344Z Ran 1 test in 0.002s
2022-05-18T04:34:09.3849510Z 
2022-05-18T04:34:09.3849622Z OK (skipped=1)
2022-05-18T04:34:09.3849760Z 
2022-05-18T04:34:09.3849889Z Generating XML reports...
2022-05-18T04:34:09.3885387Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043409.xml
2022-05-18T04:34:10.6526074Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:10.6540295Z 
2022-05-18T04:34:10.6540620Z Running tests...
2022-05-18T04:34:10.6541046Z ----------------------------------------------------------------------
2022-05-18T04:34:10.6568740Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.003s)
2022-05-18T04:34:10.6569071Z 
2022-05-18T04:34:10.6569345Z ----------------------------------------------------------------------
2022-05-18T04:34:10.6569679Z Ran 1 test in 0.003s
2022-05-18T04:34:10.6569825Z 
2022-05-18T04:34:10.6569953Z OK (skipped=1)
2022-05-18T04:34:10.6570113Z 
2022-05-18T04:34:10.6570240Z Generating XML reports...
2022-05-18T04:34:10.6605631Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043410.xml
2022-05-18T04:34:11.9260903Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:11.9275682Z 
2022-05-18T04:34:11.9276004Z Running tests...
2022-05-18T04:34:11.9276439Z ----------------------------------------------------------------------
2022-05-18T04:34:11.9298076Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:34:11.9298425Z 
2022-05-18T04:34:11.9298696Z ----------------------------------------------------------------------
2022-05-18T04:34:11.9299035Z Ran 1 test in 0.002s
2022-05-18T04:34:11.9299182Z 
2022-05-18T04:34:11.9299297Z OK (skipped=1)
2022-05-18T04:34:11.9299473Z 
2022-05-18T04:34:11.9299601Z Generating XML reports...
2022-05-18T04:34:11.9336258Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043411.xml
2022-05-18T04:34:13.1689700Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:13.1702911Z 
2022-05-18T04:34:13.1703518Z Running tests...
2022-05-18T04:34:13.1704433Z ----------------------------------------------------------------------
2022-05-18T04:34:13.1725620Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:34:13.1726384Z 
2022-05-18T04:34:13.1726862Z ----------------------------------------------------------------------
2022-05-18T04:34:13.1727190Z Ran 1 test in 0.002s
2022-05-18T04:34:13.1727357Z 
2022-05-18T04:34:13.1727474Z OK (skipped=1)
2022-05-18T04:34:13.1727634Z 
2022-05-18T04:34:13.1727764Z Generating XML reports...
2022-05-18T04:34:13.1761636Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043413.xml
2022-05-18T04:34:14.3945658Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:14.3960733Z 
2022-05-18T04:34:14.3960972Z Running tests...
2022-05-18T04:34:14.3961457Z ----------------------------------------------------------------------
2022-05-18T04:34:14.3984283Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:34:14.3984864Z 
2022-05-18T04:34:14.3985180Z ----------------------------------------------------------------------
2022-05-18T04:34:14.3985526Z Ran 1 test in 0.002s
2022-05-18T04:34:14.3985701Z 
2022-05-18T04:34:14.3985817Z OK (skipped=1)
2022-05-18T04:34:14.3985978Z 
2022-05-18T04:34:14.3986087Z Generating XML reports...
2022-05-18T04:34:14.4022059Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043414.xml
2022-05-18T04:34:15.6659827Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:15.6674961Z 
2022-05-18T04:34:15.6675275Z Running tests...
2022-05-18T04:34:15.6675728Z ----------------------------------------------------------------------
2022-05-18T04:34:17.2691770Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:17.3081695Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36145
2022-05-18T04:34:17.3187364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36146
2022-05-18T04:34:18.5394453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:18.5402230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:18.5403033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:18.5495906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:18.5502343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:18.6416861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:20.2266665Z ok (4.559s)
2022-05-18T04:34:20.2266890Z 
2022-05-18T04:34:20.2267307Z ----------------------------------------------------------------------
2022-05-18T04:34:20.2267653Z Ran 1 test in 4.559s
2022-05-18T04:34:20.2267803Z 
2022-05-18T04:34:20.2267904Z OK
2022-05-18T04:34:20.2268042Z 
2022-05-18T04:34:20.2268187Z Generating XML reports...
2022-05-18T04:34:20.2309680Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043415.xml
2022-05-18T04:34:21.6579011Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:21.6593059Z 
2022-05-18T04:34:21.6593577Z Running tests...
2022-05-18T04:34:21.6594088Z ----------------------------------------------------------------------
2022-05-18T04:34:23.2337683Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:23.2726669Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36261
2022-05-18T04:34:23.2831953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36262
2022-05-18T04:34:24.4010663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:24.4163200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:24.4164010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:24.4213249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:24.4219274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:24.5177610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:26.0907636Z ok (4.431s)
2022-05-18T04:34:26.0907849Z 
2022-05-18T04:34:26.0908259Z ----------------------------------------------------------------------
2022-05-18T04:34:26.0908607Z Ran 1 test in 4.431s
2022-05-18T04:34:26.0908776Z 
2022-05-18T04:34:26.0908874Z OK
2022-05-18T04:34:26.0908995Z 
2022-05-18T04:34:26.0909134Z Generating XML reports...
2022-05-18T04:34:26.0950757Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043421.xml
2022-05-18T04:34:27.5081213Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:27.5095946Z 
2022-05-18T04:34:27.5096159Z Running tests...
2022-05-18T04:34:27.5096617Z ----------------------------------------------------------------------
2022-05-18T04:34:29.0905762Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:29.1304909Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36377
2022-05-18T04:34:29.1410708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36378
2022-05-18T04:34:30.3112412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:30.3246299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:30.3247125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:30.3315066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:30.3322226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:30.4261531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:31.9487635Z ok (4.439s)
2022-05-18T04:34:31.9487858Z 
2022-05-18T04:34:31.9488224Z ----------------------------------------------------------------------
2022-05-18T04:34:31.9488571Z Ran 1 test in 4.439s
2022-05-18T04:34:31.9488739Z 
2022-05-18T04:34:31.9488837Z OK
2022-05-18T04:34:31.9488974Z 
2022-05-18T04:34:31.9489090Z Generating XML reports...
2022-05-18T04:34:31.9529732Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043427.xml
2022-05-18T04:34:33.3200178Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:33.3214897Z 
2022-05-18T04:34:33.3215439Z Running tests...
2022-05-18T04:34:33.3216079Z ----------------------------------------------------------------------
2022-05-18T04:34:34.8895556Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:34.9293840Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36493
2022-05-18T04:34:34.9398526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36494
2022-05-18T04:34:36.0876891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:36.1100689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:36.1101817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:36.1180560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:36.1186979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:36.2114729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:37.8478506Z ok (4.526s)
2022-05-18T04:34:37.8479026Z 
2022-05-18T04:34:37.8479626Z ----------------------------------------------------------------------
2022-05-18T04:34:37.8480008Z Ran 1 test in 4.526s
2022-05-18T04:34:37.8480177Z 
2022-05-18T04:34:37.8480272Z OK
2022-05-18T04:34:37.8480412Z 
2022-05-18T04:34:37.8480549Z Generating XML reports...
2022-05-18T04:34:37.8519868Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043433.xml
2022-05-18T04:34:39.2681829Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:39.2696445Z 
2022-05-18T04:34:39.2696932Z Running tests...
2022-05-18T04:34:39.2697442Z ----------------------------------------------------------------------
2022-05-18T04:34:40.8382826Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:40.8780613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36609
2022-05-18T04:34:40.8887307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36610
2022-05-18T04:34:42.0113373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:42.0461819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:42.0462649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:42.0518199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:42.0524460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:42.1472474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:44.2974024Z ok (5.027s)
2022-05-18T04:34:44.2974252Z 
2022-05-18T04:34:44.2975108Z ----------------------------------------------------------------------
2022-05-18T04:34:44.2975469Z Ran 1 test in 5.028s
2022-05-18T04:34:44.2975632Z 
2022-05-18T04:34:44.2975728Z OK
2022-05-18T04:34:44.2975862Z 
2022-05-18T04:34:44.2976001Z Generating XML reports...
2022-05-18T04:34:44.3015834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043439.xml
2022-05-18T04:34:45.7122877Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:45.7136837Z 
2022-05-18T04:34:45.7137115Z Running tests...
2022-05-18T04:34:45.7137553Z ----------------------------------------------------------------------
2022-05-18T04:34:45.7160413Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:34:45.7160966Z 
2022-05-18T04:34:45.7161389Z ----------------------------------------------------------------------
2022-05-18T04:34:45.7161763Z Ran 1 test in 0.002s
2022-05-18T04:34:45.7161937Z 
2022-05-18T04:34:45.7162050Z OK (skipped=1)
2022-05-18T04:34:45.7162206Z 
2022-05-18T04:34:45.7162334Z Generating XML reports...
2022-05-18T04:34:45.7197023Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043445.xml
2022-05-18T04:34:46.9757738Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:46.9772825Z 
2022-05-18T04:34:46.9773103Z Running tests...
2022-05-18T04:34:46.9773552Z ----------------------------------------------------------------------
2022-05-18T04:34:46.9799615Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:34:46.9800152Z 
2022-05-18T04:34:46.9800461Z ----------------------------------------------------------------------
2022-05-18T04:34:46.9800777Z Ran 1 test in 0.003s
2022-05-18T04:34:46.9800940Z 
2022-05-18T04:34:46.9801050Z OK (skipped=1)
2022-05-18T04:34:46.9801215Z 
2022-05-18T04:34:46.9801344Z Generating XML reports...
2022-05-18T04:34:46.9836701Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043446.xml
2022-05-18T04:34:48.2460493Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:48.2474571Z 
2022-05-18T04:34:48.2475042Z Running tests...
2022-05-18T04:34:48.2475814Z ----------------------------------------------------------------------
2022-05-18T04:34:48.2498888Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:34:48.2499285Z 
2022-05-18T04:34:48.2499660Z ----------------------------------------------------------------------
2022-05-18T04:34:48.2500209Z Ran 1 test in 0.002s
2022-05-18T04:34:48.2500359Z 
2022-05-18T04:34:48.2500475Z OK (skipped=1)
2022-05-18T04:34:48.2500631Z 
2022-05-18T04:34:48.2500771Z Generating XML reports...
2022-05-18T04:34:48.2536164Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043448.xml
2022-05-18T04:34:49.5288834Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:49.5303839Z 
2022-05-18T04:34:49.5304056Z Running tests...
2022-05-18T04:34:49.5304506Z ----------------------------------------------------------------------
2022-05-18T04:34:51.1218740Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:51.1616962Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36835
2022-05-18T04:34:51.1722588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36836
2022-05-18T04:34:52.3263568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:52.3522009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:52.3522847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:52.3567090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:52.3573529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:52.4532580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:52.6774674Z ok (3.147s)
2022-05-18T04:34:52.6774918Z 
2022-05-18T04:34:52.6775315Z ----------------------------------------------------------------------
2022-05-18T04:34:52.6775657Z Ran 1 test in 3.147s
2022-05-18T04:34:52.6775824Z 
2022-05-18T04:34:52.6775903Z OK
2022-05-18T04:34:52.6777120Z 
2022-05-18T04:34:52.6777579Z Generating XML reports...
2022-05-18T04:34:52.6816400Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043449.xml
2022-05-18T04:34:54.0796190Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:54.0810656Z 
2022-05-18T04:34:54.0811121Z Running tests...
2022-05-18T04:34:54.0811602Z ----------------------------------------------------------------------
2022-05-18T04:34:55.6639291Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:55.7027957Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36942
2022-05-18T04:34:55.7133180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36943
2022-05-18T04:34:56.8837598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:56.8851765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:56.8852587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:56.8938691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:34:56.8944642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:56.9862362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:57.2184495Z ok (3.137s)
2022-05-18T04:34:57.2184726Z 
2022-05-18T04:34:57.2185127Z ----------------------------------------------------------------------
2022-05-18T04:34:57.2185741Z Ran 1 test in 3.137s
2022-05-18T04:34:57.2185947Z 
2022-05-18T04:34:57.2186077Z OK
2022-05-18T04:34:57.2186248Z 
2022-05-18T04:34:57.2186371Z Generating XML reports...
2022-05-18T04:34:57.2226357Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043454.xml
2022-05-18T04:34:58.5781977Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:58.5796068Z 
2022-05-18T04:34:58.5796491Z Running tests...
2022-05-18T04:34:58.5797004Z ----------------------------------------------------------------------
2022-05-18T04:34:58.5817738Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:34:58.5818271Z 
2022-05-18T04:34:58.5818573Z ----------------------------------------------------------------------
2022-05-18T04:34:58.5818921Z Ran 1 test in 0.002s
2022-05-18T04:34:58.5819088Z 
2022-05-18T04:34:58.5819200Z OK (skipped=1)
2022-05-18T04:34:58.5819358Z 
2022-05-18T04:34:58.5819485Z Generating XML reports...
2022-05-18T04:34:58.5854795Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043458.xml
2022-05-18T04:34:59.8148093Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:34:59.8161602Z 
2022-05-18T04:34:59.8161879Z Running tests...
2022-05-18T04:34:59.8162314Z ----------------------------------------------------------------------
2022-05-18T04:34:59.8181408Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:34:59.8181761Z 
2022-05-18T04:34:59.8182050Z ----------------------------------------------------------------------
2022-05-18T04:34:59.8182380Z Ran 1 test in 0.002s
2022-05-18T04:34:59.8182548Z 
2022-05-18T04:34:59.8182660Z OK (skipped=1)
2022-05-18T04:34:59.8182817Z 
2022-05-18T04:34:59.8182946Z Generating XML reports...
2022-05-18T04:34:59.8217345Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043459.xml
2022-05-18T04:35:01.0663455Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:01.0677977Z 
2022-05-18T04:35:01.0678265Z Running tests...
2022-05-18T04:35:01.0678938Z ----------------------------------------------------------------------
2022-05-18T04:35:02.6495138Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:02.6889117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37115
2022-05-18T04:35:02.6994168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37116
2022-05-18T04:35:03.8543177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:03.8850151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:03.8850972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:03.8948911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:03.8955445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:03.9864767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:05.1731656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptx4n752z
2022-05-18T04:35:05.1732536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptx4n752z/_remote_module_non_scriptable.py
2022-05-18T04:35:05.2350689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgktpt3wj
2022-05-18T04:35:05.2351959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgktpt3wj/_remote_module_non_scriptable.py
2022-05-18T04:35:05.8073723Z ok (4.739s)
2022-05-18T04:35:05.8073960Z 
2022-05-18T04:35:05.8074596Z ----------------------------------------------------------------------
2022-05-18T04:35:05.8074960Z Ran 1 test in 4.740s
2022-05-18T04:35:05.8075128Z 
2022-05-18T04:35:05.8075224Z OK
2022-05-18T04:35:05.8075358Z 
2022-05-18T04:35:05.8075496Z Generating XML reports...
2022-05-18T04:35:05.8115494Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043501.xml
2022-05-18T04:35:07.2286653Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:07.2300514Z 
2022-05-18T04:35:07.2301000Z Running tests...
2022-05-18T04:35:07.2301661Z ----------------------------------------------------------------------
2022-05-18T04:35:08.7954554Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:08.8343425Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37235
2022-05-18T04:35:08.8450867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37236
2022-05-18T04:35:09.9763395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:09.9849541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:09.9850345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:09.9864537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:09.9870597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:10.0864759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:11.2377303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1qwxg2k
2022-05-18T04:35:11.2377934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1qwxg2k/_remote_module_non_scriptable.py
2022-05-18T04:35:11.3392186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxi8euuc
2022-05-18T04:35:11.3393907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxi8euuc/_remote_module_non_scriptable.py
2022-05-18T04:35:11.9531104Z ok (4.723s)
2022-05-18T04:35:11.9531333Z 
2022-05-18T04:35:11.9531724Z ----------------------------------------------------------------------
2022-05-18T04:35:11.9532049Z Ran 1 test in 4.723s
2022-05-18T04:35:11.9532214Z 
2022-05-18T04:35:11.9532315Z OK
2022-05-18T04:35:11.9532449Z 
2022-05-18T04:35:11.9532581Z Generating XML reports...
2022-05-18T04:35:11.9574342Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043507.xml
2022-05-18T04:35:13.3515426Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:13.3529968Z 
2022-05-18T04:35:13.3530495Z Running tests...
2022-05-18T04:35:13.3530987Z ----------------------------------------------------------------------
2022-05-18T04:35:14.8993196Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:14.9382201Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37355
2022-05-18T04:35:14.9483739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37356
2022-05-18T04:35:16.0945254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:16.1204744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:16.1205589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:16.1250688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:16.1257267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:16.2218864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:18.8584228Z ok (5.505s)
2022-05-18T04:35:18.8584445Z 
2022-05-18T04:35:18.8584852Z ----------------------------------------------------------------------
2022-05-18T04:35:18.8585175Z Ran 1 test in 5.505s
2022-05-18T04:35:18.8585342Z 
2022-05-18T04:35:18.8585439Z OK
2022-05-18T04:35:18.8585574Z 
2022-05-18T04:35:18.8585709Z Generating XML reports...
2022-05-18T04:35:18.8627414Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043513.xml
2022-05-18T04:35:20.2739663Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:20.2753980Z 
2022-05-18T04:35:20.2754126Z Running tests...
2022-05-18T04:35:20.2755292Z ----------------------------------------------------------------------
2022-05-18T04:35:21.8371194Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:21.8760384Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37472
2022-05-18T04:35:21.8864234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37473
2022-05-18T04:35:23.0291981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:23.0534358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:23.0535411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:23.0596392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:23.0603681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:23.1550153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:25.7958768Z ok (5.520s)
2022-05-18T04:35:25.7958991Z 
2022-05-18T04:35:25.7959647Z ----------------------------------------------------------------------
2022-05-18T04:35:25.7960002Z Ran 1 test in 5.520s
2022-05-18T04:35:25.7960170Z 
2022-05-18T04:35:25.7960273Z OK
2022-05-18T04:35:25.7960391Z 
2022-05-18T04:35:25.7960525Z Generating XML reports...
2022-05-18T04:35:25.8001534Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043520.xml
2022-05-18T04:35:27.1910988Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:27.1925214Z 
2022-05-18T04:35:27.1925543Z Running tests...
2022-05-18T04:35:27.1925982Z ----------------------------------------------------------------------
2022-05-18T04:35:28.7381481Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:28.7531785Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.560s)
2022-05-18T04:35:28.7532376Z 
2022-05-18T04:35:28.7532664Z ----------------------------------------------------------------------
2022-05-18T04:35:28.7532984Z Ran 1 test in 1.561s
2022-05-18T04:35:28.7533162Z 
2022-05-18T04:35:28.7533272Z OK (skipped=1)
2022-05-18T04:35:28.7533428Z 
2022-05-18T04:35:28.7533553Z Generating XML reports...
2022-05-18T04:35:28.7565615Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043527.xml
2022-05-18T04:35:30.1201021Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:30.1216440Z 
2022-05-18T04:35:30.1216719Z Running tests...
2022-05-18T04:35:30.1217166Z ----------------------------------------------------------------------
2022-05-18T04:35:31.6826356Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:31.6983001Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.576s)
2022-05-18T04:35:31.6983595Z 
2022-05-18T04:35:31.6983874Z ----------------------------------------------------------------------
2022-05-18T04:35:31.6984206Z Ran 1 test in 1.577s
2022-05-18T04:35:31.6984370Z 
2022-05-18T04:35:31.6984481Z OK (skipped=1)
2022-05-18T04:35:31.6986566Z 
2022-05-18T04:35:31.6987203Z Generating XML reports...
2022-05-18T04:35:31.7018211Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043530.xml
2022-05-18T04:35:33.0970005Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:33.0985801Z 
2022-05-18T04:35:33.0986100Z Running tests...
2022-05-18T04:35:33.0986794Z ----------------------------------------------------------------------
2022-05-18T04:35:34.6637446Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:34.7025700Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37657
2022-05-18T04:35:34.7129309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37658
2022-05-18T04:35:35.8789560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:35.8835718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:35.8836531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:35.8890627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:35.8896596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:35.9848917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:36.1179443Z skip: Need at least 4 CUDA devices (3.019s)
2022-05-18T04:35:36.1179708Z 
2022-05-18T04:35:36.1180103Z ----------------------------------------------------------------------
2022-05-18T04:35:36.1180455Z Ran 1 test in 3.019s
2022-05-18T04:35:36.1180620Z 
2022-05-18T04:35:36.1180733Z OK (skipped=1)
2022-05-18T04:35:36.1180892Z 
2022-05-18T04:35:36.1181017Z Generating XML reports...
2022-05-18T04:35:36.1222093Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043533.xml
2022-05-18T04:35:37.5257794Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:37.5272737Z 
2022-05-18T04:35:37.5273072Z Running tests...
2022-05-18T04:35:37.5273517Z ----------------------------------------------------------------------
2022-05-18T04:35:39.0935611Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:39.1324218Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37764
2022-05-18T04:35:39.1428856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37765
2022-05-18T04:35:40.2731657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:40.2818794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:40.2819601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:40.2832895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:40.2839162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:40.3833799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:40.5477378Z skip: Need at least 4 CUDA devices (3.020s)
2022-05-18T04:35:40.5477630Z 
2022-05-18T04:35:40.5478007Z ----------------------------------------------------------------------
2022-05-18T04:35:40.5478699Z Ran 1 test in 3.020s
2022-05-18T04:35:40.5478866Z 
2022-05-18T04:35:40.5478984Z OK (skipped=1)
2022-05-18T04:35:40.5479345Z 
2022-05-18T04:35:40.5479490Z Generating XML reports...
2022-05-18T04:35:40.5521135Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043537.xml
2022-05-18T04:35:41.9252726Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:41.9267058Z 
2022-05-18T04:35:41.9267454Z Running tests...
2022-05-18T04:35:41.9267969Z ----------------------------------------------------------------------
2022-05-18T04:35:41.9287376Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:35:41.9288439Z 
2022-05-18T04:35:41.9288899Z ----------------------------------------------------------------------
2022-05-18T04:35:41.9289236Z Ran 1 test in 0.002s
2022-05-18T04:35:41.9289403Z 
2022-05-18T04:35:41.9289517Z OK (skipped=1)
2022-05-18T04:35:41.9289675Z 
2022-05-18T04:35:41.9289802Z Generating XML reports...
2022-05-18T04:35:41.9324952Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043541.xml
2022-05-18T04:35:43.1773314Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:43.1788485Z 
2022-05-18T04:35:43.1789057Z Running tests...
2022-05-18T04:35:43.1789537Z ----------------------------------------------------------------------
2022-05-18T04:35:43.1809793Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:35:43.1810683Z 
2022-05-18T04:35:43.1810991Z ----------------------------------------------------------------------
2022-05-18T04:35:43.1811646Z Ran 1 test in 0.002s
2022-05-18T04:35:43.1811815Z 
2022-05-18T04:35:43.1811909Z OK (skipped=1)
2022-05-18T04:35:43.1812064Z 
2022-05-18T04:35:43.1812190Z Generating XML reports...
2022-05-18T04:35:43.1847874Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043543.xml
2022-05-18T04:35:44.4310452Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:44.4325492Z 
2022-05-18T04:35:44.4325952Z Running tests...
2022-05-18T04:35:44.4326469Z ----------------------------------------------------------------------
2022-05-18T04:35:44.4348511Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:35:44.4348834Z 
2022-05-18T04:35:44.4349158Z ----------------------------------------------------------------------
2022-05-18T04:35:44.4349499Z Ran 1 test in 0.002s
2022-05-18T04:35:44.4349697Z 
2022-05-18T04:35:44.4349808Z OK (skipped=1)
2022-05-18T04:35:44.4349967Z 
2022-05-18T04:35:44.4350076Z Generating XML reports...
2022-05-18T04:35:44.4386221Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043544.xml
2022-05-18T04:35:45.6853087Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:45.6867160Z 
2022-05-18T04:35:45.6867485Z Running tests...
2022-05-18T04:35:45.6867921Z ----------------------------------------------------------------------
2022-05-18T04:35:45.6889499Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:35:45.6889817Z 
2022-05-18T04:35:45.6890106Z ----------------------------------------------------------------------
2022-05-18T04:35:45.6890446Z Ran 1 test in 0.002s
2022-05-18T04:35:45.6890611Z 
2022-05-18T04:35:45.6890714Z OK (skipped=1)
2022-05-18T04:35:45.6890884Z 
2022-05-18T04:35:45.6891008Z Generating XML reports...
2022-05-18T04:35:45.6926326Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043545.xml
2022-05-18T04:35:46.9516172Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:46.9531014Z 
2022-05-18T04:35:46.9531273Z Running tests...
2022-05-18T04:35:46.9531892Z ----------------------------------------------------------------------
2022-05-18T04:35:46.9551882Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:35:46.9552202Z 
2022-05-18T04:35:46.9552646Z ----------------------------------------------------------------------
2022-05-18T04:35:46.9553076Z Ran 1 test in 0.002s
2022-05-18T04:35:46.9553241Z 
2022-05-18T04:35:46.9553352Z OK (skipped=1)
2022-05-18T04:35:46.9553507Z 
2022-05-18T04:35:46.9553632Z Generating XML reports...
2022-05-18T04:35:46.9588367Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043546.xml
2022-05-18T04:35:48.1843768Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:48.1858066Z 
2022-05-18T04:35:48.1858595Z Running tests...
2022-05-18T04:35:48.1859238Z ----------------------------------------------------------------------
2022-05-18T04:35:48.1879027Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:35:48.1879358Z 
2022-05-18T04:35:48.1879653Z ----------------------------------------------------------------------
2022-05-18T04:35:48.1880241Z Ran 1 test in 0.002s
2022-05-18T04:35:48.1880411Z 
2022-05-18T04:35:48.1880527Z OK (skipped=1)
2022-05-18T04:35:48.1880683Z 
2022-05-18T04:35:48.1880791Z Generating XML reports...
2022-05-18T04:35:48.1915481Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043548.xml
2022-05-18T04:35:49.4087662Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:49.4101989Z 
2022-05-18T04:35:49.4102522Z Running tests...
2022-05-18T04:35:49.4103096Z ----------------------------------------------------------------------
2022-05-18T04:35:49.4124643Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:35:49.4125212Z 
2022-05-18T04:35:49.4125509Z ----------------------------------------------------------------------
2022-05-18T04:35:49.4125845Z Ran 1 test in 0.002s
2022-05-18T04:35:49.4126038Z 
2022-05-18T04:35:49.4126243Z OK (skipped=1)
2022-05-18T04:35:49.4126518Z 
2022-05-18T04:35:49.4126632Z Generating XML reports...
2022-05-18T04:35:49.4161627Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043549.xml
2022-05-18T04:35:50.6661063Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:50.6675231Z 
2022-05-18T04:35:50.6675404Z Running tests...
2022-05-18T04:35:50.6676104Z ----------------------------------------------------------------------
2022-05-18T04:35:50.6697911Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:35:50.6698225Z 
2022-05-18T04:35:50.6698683Z ----------------------------------------------------------------------
2022-05-18T04:35:50.6699134Z Ran 1 test in 0.002s
2022-05-18T04:35:50.6699283Z 
2022-05-18T04:35:50.6699394Z OK (skipped=1)
2022-05-18T04:35:50.6699549Z 
2022-05-18T04:35:50.6699674Z Generating XML reports...
2022-05-18T04:35:50.6735631Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043550.xml
2022-05-18T04:35:51.9161560Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:51.9175920Z 
2022-05-18T04:35:51.9176142Z Running tests...
2022-05-18T04:35:51.9176751Z ----------------------------------------------------------------------
2022-05-18T04:35:51.9196673Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:35:51.9197000Z 
2022-05-18T04:35:51.9197616Z ----------------------------------------------------------------------
2022-05-18T04:35:51.9197959Z Ran 1 test in 0.002s
2022-05-18T04:35:51.9198448Z 
2022-05-18T04:35:51.9198564Z OK (skipped=1)
2022-05-18T04:35:51.9198726Z 
2022-05-18T04:35:51.9198855Z Generating XML reports...
2022-05-18T04:35:51.9233209Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043551.xml
2022-05-18T04:35:53.1442826Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:53.1456684Z 
2022-05-18T04:35:53.1456852Z Running tests...
2022-05-18T04:35:53.1457330Z ----------------------------------------------------------------------
2022-05-18T04:35:53.1477252Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:35:53.1477569Z 
2022-05-18T04:35:53.1477860Z ----------------------------------------------------------------------
2022-05-18T04:35:53.1478453Z Ran 1 test in 0.002s
2022-05-18T04:35:53.1478621Z 
2022-05-18T04:35:53.1478732Z OK (skipped=1)
2022-05-18T04:35:53.1478898Z 
2022-05-18T04:35:53.1479024Z Generating XML reports...
2022-05-18T04:35:53.1512787Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043553.xml
2022-05-18T04:35:54.4043914Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:35:54.4058241Z 
2022-05-18T04:35:54.4058680Z Running tests...
2022-05-18T04:35:54.4059197Z ----------------------------------------------------------------------
2022-05-18T04:35:55.9932922Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:56.0328491Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38201
2022-05-18T04:35:56.0435029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38202
2022-05-18T04:35:57.1703711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:57.2026612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:57.2027717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:57.2109017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:35:57.2115359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:57.3040985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:59.2517335Z ok (4.845s)
2022-05-18T04:35:59.2517556Z 
2022-05-18T04:35:59.2517948Z ----------------------------------------------------------------------
2022-05-18T04:35:59.2518535Z Ran 1 test in 4.846s
2022-05-18T04:35:59.2518712Z 
2022-05-18T04:35:59.2518827Z OK
2022-05-18T04:35:59.2518948Z 
2022-05-18T04:35:59.2519082Z Generating XML reports...
2022-05-18T04:35:59.2559100Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043554.xml
2022-05-18T04:36:00.6526798Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:00.6540353Z 
2022-05-18T04:36:00.6540622Z Running tests...
2022-05-18T04:36:00.6541047Z ----------------------------------------------------------------------
2022-05-18T04:36:00.6562418Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:36:00.6562755Z 
2022-05-18T04:36:00.6563042Z ----------------------------------------------------------------------
2022-05-18T04:36:00.6563373Z Ran 1 test in 0.002s
2022-05-18T04:36:00.6563520Z 
2022-05-18T04:36:00.6563634Z OK (skipped=1)
2022-05-18T04:36:00.6563789Z 
2022-05-18T04:36:00.6564148Z Generating XML reports...
2022-05-18T04:36:00.6597568Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043600.xml
2022-05-18T04:36:01.9041987Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:01.9056300Z 
2022-05-18T04:36:01.9056721Z Running tests...
2022-05-18T04:36:01.9057246Z ----------------------------------------------------------------------
2022-05-18T04:36:01.9078300Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:36:01.9078613Z 
2022-05-18T04:36:01.9079258Z ----------------------------------------------------------------------
2022-05-18T04:36:01.9079595Z Ran 1 test in 0.002s
2022-05-18T04:36:01.9079761Z 
2022-05-18T04:36:01.9079871Z OK (skipped=1)
2022-05-18T04:36:01.9080029Z 
2022-05-18T04:36:01.9080154Z Generating XML reports...
2022-05-18T04:36:01.9115062Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043601.xml
2022-05-18T04:36:03.1321119Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:03.1337303Z 
2022-05-18T04:36:03.1337773Z Running tests...
2022-05-18T04:36:03.1338306Z ----------------------------------------------------------------------
2022-05-18T04:36:04.7196718Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:04.7595926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38387
2022-05-18T04:36:04.7703103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38388
2022-05-18T04:36:05.9036025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:05.9647291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:05.9648121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:05.9747833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:05.9754709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:06.0662414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:08.0788717Z ok (4.945s)
2022-05-18T04:36:08.0788991Z 
2022-05-18T04:36:08.0789394Z ----------------------------------------------------------------------
2022-05-18T04:36:08.0789762Z Ran 1 test in 4.945s
2022-05-18T04:36:08.0789934Z 
2022-05-18T04:36:08.0790035Z OK
2022-05-18T04:36:08.0790174Z 
2022-05-18T04:36:08.0790294Z Generating XML reports...
2022-05-18T04:36:08.0832170Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043603.xml
2022-05-18T04:36:09.4803344Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:09.4816781Z 
2022-05-18T04:36:09.4817072Z Running tests...
2022-05-18T04:36:09.4817512Z ----------------------------------------------------------------------
2022-05-18T04:36:11.0080020Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:11.0467998Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38507
2022-05-18T04:36:11.0571944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38508
2022-05-18T04:36:12.1831207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:12.1953682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:12.1954745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:12.2033900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:12.2039855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:12.2968038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:14.3656220Z ok (4.884s)
2022-05-18T04:36:14.3656449Z 
2022-05-18T04:36:14.3656853Z ----------------------------------------------------------------------
2022-05-18T04:36:14.3657210Z Ran 1 test in 4.884s
2022-05-18T04:36:14.3657398Z 
2022-05-18T04:36:14.3657495Z OK
2022-05-18T04:36:14.3657617Z 
2022-05-18T04:36:14.3657754Z Generating XML reports...
2022-05-18T04:36:14.3698293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043609.xml
2022-05-18T04:36:15.7823111Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:15.7837444Z 
2022-05-18T04:36:15.7837653Z Running tests...
2022-05-18T04:36:15.7838406Z ----------------------------------------------------------------------
2022-05-18T04:36:15.7859607Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:36:15.7859921Z 
2022-05-18T04:36:15.7860210Z ----------------------------------------------------------------------
2022-05-18T04:36:15.7860545Z Ran 1 test in 0.002s
2022-05-18T04:36:15.7860713Z 
2022-05-18T04:36:15.7861157Z OK (skipped=1)
2022-05-18T04:36:15.7861381Z 
2022-05-18T04:36:15.7861850Z Generating XML reports...
2022-05-18T04:36:15.7897079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043615.xml
2022-05-18T04:36:17.0011213Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:17.0025401Z 
2022-05-18T04:36:17.0025867Z Running tests...
2022-05-18T04:36:17.0026348Z ----------------------------------------------------------------------
2022-05-18T04:36:17.0045712Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:36:17.0046026Z 
2022-05-18T04:36:17.0046315Z ----------------------------------------------------------------------
2022-05-18T04:36:17.0046656Z Ran 1 test in 0.002s
2022-05-18T04:36:17.0046804Z 
2022-05-18T04:36:17.0046917Z OK (skipped=1)
2022-05-18T04:36:17.0047074Z 
2022-05-18T04:36:17.0047204Z Generating XML reports...
2022-05-18T04:36:17.0082741Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043616.xml
2022-05-18T04:36:18.2662594Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:18.2676333Z 
2022-05-18T04:36:18.2676611Z Running tests...
2022-05-18T04:36:18.2677073Z ----------------------------------------------------------------------
2022-05-18T04:36:18.2702689Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:36:18.2703003Z 
2022-05-18T04:36:18.2703286Z ----------------------------------------------------------------------
2022-05-18T04:36:18.2703596Z Ran 1 test in 0.003s
2022-05-18T04:36:18.2703759Z 
2022-05-18T04:36:18.2703874Z OK (skipped=1)
2022-05-18T04:36:18.2704031Z 
2022-05-18T04:36:18.2704157Z Generating XML reports...
2022-05-18T04:36:18.2739976Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043618.xml
2022-05-18T04:36:19.5269979Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:19.5284781Z 
2022-05-18T04:36:19.5284994Z Running tests...
2022-05-18T04:36:19.5285447Z ----------------------------------------------------------------------
2022-05-18T04:36:19.5304984Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:36:19.5305351Z 
2022-05-18T04:36:19.5305669Z ----------------------------------------------------------------------
2022-05-18T04:36:19.5306004Z Ran 1 test in 0.002s
2022-05-18T04:36:19.5306177Z 
2022-05-18T04:36:19.5306292Z OK (skipped=1)
2022-05-18T04:36:19.5306457Z 
2022-05-18T04:36:19.5306589Z Generating XML reports...
2022-05-18T04:36:19.5341700Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043619.xml
2022-05-18T04:36:20.7968916Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:20.7983704Z 
2022-05-18T04:36:20.7984132Z Running tests...
2022-05-18T04:36:20.7984614Z ----------------------------------------------------------------------
2022-05-18T04:36:22.3999181Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:22.4397930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38759
2022-05-18T04:36:22.4505336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38760
2022-05-18T04:36:23.6083014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:23.6378953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:23.6379809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:23.6386857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:23.6392996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:23.7393488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:25.8592972Z ok (5.060s)
2022-05-18T04:36:25.8593199Z 
2022-05-18T04:36:25.8593591Z ----------------------------------------------------------------------
2022-05-18T04:36:25.8593938Z Ran 1 test in 5.061s
2022-05-18T04:36:25.8594110Z 
2022-05-18T04:36:25.8594195Z OK
2022-05-18T04:36:25.8594341Z 
2022-05-18T04:36:25.8594479Z Generating XML reports...
2022-05-18T04:36:25.8635914Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043620.xml
2022-05-18T04:36:27.2960224Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:27.2975389Z 
2022-05-18T04:36:27.2975661Z Running tests...
2022-05-18T04:36:27.2976088Z ----------------------------------------------------------------------
2022-05-18T04:36:28.8671560Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:28.9061076Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38879
2022-05-18T04:36:28.9168340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38880
2022-05-18T04:36:30.0333996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:30.0663568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:30.0664395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:30.0739117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:30.0746249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:30.1679268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:32.1251989Z ok (4.827s)
2022-05-18T04:36:32.1252229Z 
2022-05-18T04:36:32.1252874Z ----------------------------------------------------------------------
2022-05-18T04:36:32.1253261Z Ran 1 test in 4.828s
2022-05-18T04:36:32.1253436Z 
2022-05-18T04:36:32.1253533Z OK
2022-05-18T04:36:32.1253681Z 
2022-05-18T04:36:32.1253819Z Generating XML reports...
2022-05-18T04:36:32.1294122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043627.xml
2022-05-18T04:36:33.5246410Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:33.5260053Z 
2022-05-18T04:36:33.5260469Z Running tests...
2022-05-18T04:36:33.5260944Z ----------------------------------------------------------------------
2022-05-18T04:36:33.5279079Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:36:33.5279395Z 
2022-05-18T04:36:33.5279682Z ----------------------------------------------------------------------
2022-05-18T04:36:33.5280031Z Ran 1 test in 0.002s
2022-05-18T04:36:33.5280203Z 
2022-05-18T04:36:33.5280298Z OK (skipped=1)
2022-05-18T04:36:33.5280456Z 
2022-05-18T04:36:33.5280586Z Generating XML reports...
2022-05-18T04:36:33.5315233Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043633.xml
2022-05-18T04:36:34.7782826Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:34.7796936Z 
2022-05-18T04:36:34.7797155Z Running tests...
2022-05-18T04:36:34.7797589Z ----------------------------------------------------------------------
2022-05-18T04:36:34.7817377Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-05-18T04:36:34.7818000Z 
2022-05-18T04:36:34.7818330Z ----------------------------------------------------------------------
2022-05-18T04:36:34.7818669Z Ran 1 test in 0.002s
2022-05-18T04:36:34.7818837Z 
2022-05-18T04:36:34.7818929Z OK (skipped=1)
2022-05-18T04:36:34.7819106Z 
2022-05-18T04:36:34.7819234Z Generating XML reports...
2022-05-18T04:36:34.7854468Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043634.xml
2022-05-18T04:36:36.0244067Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:36.0258652Z 
2022-05-18T04:36:36.0258921Z Running tests...
2022-05-18T04:36:36.0259365Z ----------------------------------------------------------------------
2022-05-18T04:36:36.0284340Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-05-18T04:36:36.0284687Z 
2022-05-18T04:36:36.0285526Z ----------------------------------------------------------------------
2022-05-18T04:36:36.0286202Z Ran 1 test in 0.003s
2022-05-18T04:36:36.0286512Z 
2022-05-18T04:36:36.0286737Z OK (skipped=1)
2022-05-18T04:36:36.0287040Z 
2022-05-18T04:36:36.0287299Z Generating XML reports...
2022-05-18T04:36:36.0323064Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043636.xml
2022-05-18T04:36:37.2908969Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:37.2923680Z 
2022-05-18T04:36:37.2923830Z Running tests...
2022-05-18T04:36:37.2924564Z ----------------------------------------------------------------------
2022-05-18T04:36:37.2943330Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:36:37.2943654Z 
2022-05-18T04:36:37.2943952Z ----------------------------------------------------------------------
2022-05-18T04:36:37.2944317Z Ran 1 test in 0.002s
2022-05-18T04:36:37.2944484Z 
2022-05-18T04:36:37.2944599Z OK (skipped=1)
2022-05-18T04:36:37.2944761Z 
2022-05-18T04:36:37.2944871Z Generating XML reports...
2022-05-18T04:36:37.2981688Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043637.xml
2022-05-18T04:36:38.5596363Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:38.5611143Z 
2022-05-18T04:36:38.5611632Z Running tests...
2022-05-18T04:36:38.5612144Z ----------------------------------------------------------------------
2022-05-18T04:36:38.5630351Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-05-18T04:36:38.5630694Z 
2022-05-18T04:36:38.5630984Z ----------------------------------------------------------------------
2022-05-18T04:36:38.5631336Z Ran 1 test in 0.002s
2022-05-18T04:36:38.5631504Z 
2022-05-18T04:36:38.5631596Z OK (skipped=1)
2022-05-18T04:36:38.5631755Z 
2022-05-18T04:36:38.5631881Z Generating XML reports...
2022-05-18T04:36:38.5668094Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043638.xml
2022-05-18T04:36:39.8176804Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:39.8191100Z 
2022-05-18T04:36:39.8191662Z Running tests...
2022-05-18T04:36:39.8192183Z ----------------------------------------------------------------------
2022-05-18T04:36:39.8211213Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-05-18T04:36:39.8211596Z 
2022-05-18T04:36:39.8211891Z ----------------------------------------------------------------------
2022-05-18T04:36:39.8212229Z Ran 1 test in 0.002s
2022-05-18T04:36:39.8212718Z 
2022-05-18T04:36:39.8212832Z OK (skipped=1)
2022-05-18T04:36:39.8212990Z 
2022-05-18T04:36:39.8213099Z Generating XML reports...
2022-05-18T04:36:39.8247972Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043639.xml
2022-05-18T04:36:41.0819040Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:41.0833912Z 
2022-05-18T04:36:41.0834228Z Running tests...
2022-05-18T04:36:41.0834665Z ----------------------------------------------------------------------
2022-05-18T04:36:41.0855735Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-05-18T04:36:41.0856201Z 
2022-05-18T04:36:41.0856501Z ----------------------------------------------------------------------
2022-05-18T04:36:41.0856850Z Ran 1 test in 0.002s
2022-05-18T04:36:41.0856999Z 
2022-05-18T04:36:41.0857116Z OK (skipped=1)
2022-05-18T04:36:41.0857289Z 
2022-05-18T04:36:41.0857422Z Generating XML reports...
2022-05-18T04:36:41.0894034Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043641.xml
2022-05-18T04:36:42.3422834Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:42.3437372Z 
2022-05-18T04:36:42.3437607Z Running tests...
2022-05-18T04:36:42.3438363Z ----------------------------------------------------------------------
2022-05-18T04:36:42.3457965Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:36:42.3458760Z 
2022-05-18T04:36:42.3459085Z ----------------------------------------------------------------------
2022-05-18T04:36:42.3459417Z Ran 1 test in 0.002s
2022-05-18T04:36:42.3459588Z 
2022-05-18T04:36:42.3459703Z OK (skipped=1)
2022-05-18T04:36:42.3459864Z 
2022-05-18T04:36:42.3459993Z Generating XML reports...
2022-05-18T04:36:42.3496079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043642.xml
2022-05-18T04:36:43.6197824Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:43.6212755Z 
2022-05-18T04:36:43.6213336Z Running tests...
2022-05-18T04:36:43.6213802Z ----------------------------------------------------------------------
2022-05-18T04:36:45.2302708Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:45.2702403Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39263
2022-05-18T04:36:45.2807745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39264
2022-05-18T04:36:46.3910381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:46.4098793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:46.4099644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:46.4113135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:46.4119032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:46.5113641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:48.1899293Z ok (4.568s)
2022-05-18T04:36:48.1899533Z 
2022-05-18T04:36:48.1899934Z ----------------------------------------------------------------------
2022-05-18T04:36:48.1900286Z Ran 1 test in 4.569s
2022-05-18T04:36:48.1900565Z 
2022-05-18T04:36:48.1900729Z OK
2022-05-18T04:36:48.1900960Z 
2022-05-18T04:36:48.1903608Z Generating XML reports...
2022-05-18T04:36:48.1941951Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043643.xml
2022-05-18T04:36:49.6249094Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:49.6263428Z 
2022-05-18T04:36:49.6263588Z Running tests...
2022-05-18T04:36:49.6264641Z ----------------------------------------------------------------------
2022-05-18T04:36:51.2074950Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:51.2462600Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39379
2022-05-18T04:36:51.2569115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39380
2022-05-18T04:36:52.4151023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:52.4533958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:52.4534824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:52.4556242Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:52.4563678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:52.5547989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:54.5655441Z ok (4.939s)
2022-05-18T04:36:54.5655666Z 
2022-05-18T04:36:54.5656060Z ----------------------------------------------------------------------
2022-05-18T04:36:54.5656406Z Ran 1 test in 4.939s
2022-05-18T04:36:54.5656579Z 
2022-05-18T04:36:54.5656657Z OK
2022-05-18T04:36:54.5656794Z 
2022-05-18T04:36:54.5656935Z Generating XML reports...
2022-05-18T04:36:54.5698378Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043649.xml
2022-05-18T04:36:55.9601432Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:36:55.9615410Z 
2022-05-18T04:36:55.9615803Z Running tests...
2022-05-18T04:36:55.9616280Z ----------------------------------------------------------------------
2022-05-18T04:36:57.5065926Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:57.5453816Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39499
2022-05-18T04:36:57.5554091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39500
2022-05-18T04:36:58.7220919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:58.7563124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:58.7563938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:58.7626539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:58.7632681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:58.8577323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:01.8654256Z ok (5.903s)
2022-05-18T04:37:01.8654491Z 
2022-05-18T04:37:01.8654901Z ----------------------------------------------------------------------
2022-05-18T04:37:01.8655252Z Ran 1 test in 5.904s
2022-05-18T04:37:01.8655423Z 
2022-05-18T04:37:01.8655501Z OK
2022-05-18T04:37:01.8655639Z 
2022-05-18T04:37:01.8655774Z Generating XML reports...
2022-05-18T04:37:01.8696295Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043655.xml
2022-05-18T04:37:03.3091467Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:03.3106203Z 
2022-05-18T04:37:03.3106587Z Running tests...
2022-05-18T04:37:03.3107087Z ----------------------------------------------------------------------
2022-05-18T04:37:03.3127644Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:37:03.3128006Z 
2022-05-18T04:37:03.3128343Z ----------------------------------------------------------------------
2022-05-18T04:37:03.3128695Z Ran 1 test in 0.002s
2022-05-18T04:37:03.3128844Z 
2022-05-18T04:37:03.3128961Z OK (skipped=1)
2022-05-18T04:37:03.3129120Z 
2022-05-18T04:37:03.3129252Z Generating XML reports...
2022-05-18T04:37:03.3165415Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043703.xml
2022-05-18T04:37:04.5867620Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:04.5882422Z 
2022-05-18T04:37:04.5882930Z Running tests...
2022-05-18T04:37:04.5883441Z ----------------------------------------------------------------------
2022-05-18T04:37:04.5902019Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:37:04.5902397Z 
2022-05-18T04:37:04.5902677Z ----------------------------------------------------------------------
2022-05-18T04:37:04.5903024Z Ran 1 test in 0.002s
2022-05-18T04:37:04.5903195Z 
2022-05-18T04:37:04.5903311Z OK (skipped=1)
2022-05-18T04:37:04.5903471Z 
2022-05-18T04:37:04.5903599Z Generating XML reports...
2022-05-18T04:37:04.5938907Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043704.xml
2022-05-18T04:37:05.8136367Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:05.8150760Z 
2022-05-18T04:37:05.8150943Z Running tests...
2022-05-18T04:37:05.8151802Z ----------------------------------------------------------------------
2022-05-18T04:37:05.8170711Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:37:05.8171073Z 
2022-05-18T04:37:05.8171708Z ----------------------------------------------------------------------
2022-05-18T04:37:05.8172051Z Ran 1 test in 0.002s
2022-05-18T04:37:05.8172222Z 
2022-05-18T04:37:05.8172336Z OK (skipped=1)
2022-05-18T04:37:05.8172498Z 
2022-05-18T04:37:05.8172627Z Generating XML reports...
2022-05-18T04:37:05.8207994Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043705.xml
2022-05-18T04:37:07.0730316Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:07.0745414Z 
2022-05-18T04:37:07.0745580Z Running tests...
2022-05-18T04:37:07.0746307Z ----------------------------------------------------------------------
2022-05-18T04:37:07.0766873Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-05-18T04:37:07.0767231Z 
2022-05-18T04:37:07.0767533Z ----------------------------------------------------------------------
2022-05-18T04:37:07.0767865Z Ran 1 test in 0.002s
2022-05-18T04:37:07.0768035Z 
2022-05-18T04:37:07.0768148Z OK (skipped=1)
2022-05-18T04:37:07.0768307Z 
2022-05-18T04:37:07.0768438Z Generating XML reports...
2022-05-18T04:37:07.0804628Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043707.xml
2022-05-18T04:37:08.3409458Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:08.3423877Z 
2022-05-18T04:37:08.3424182Z Running tests...
2022-05-18T04:37:08.3424630Z ----------------------------------------------------------------------
2022-05-18T04:37:08.3442771Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-05-18T04:37:08.3443210Z 
2022-05-18T04:37:08.3443628Z ----------------------------------------------------------------------
2022-05-18T04:37:08.3444018Z Ran 1 test in 0.002s
2022-05-18T04:37:08.3444203Z 
2022-05-18T04:37:08.3444323Z OK (skipped=1)
2022-05-18T04:37:08.3444486Z 
2022-05-18T04:37:08.3444616Z Generating XML reports...
2022-05-18T04:37:08.3480172Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043708.xml
2022-05-18T04:37:09.6173972Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:09.6188179Z 
2022-05-18T04:37:09.6188499Z Running tests...
2022-05-18T04:37:09.6188924Z ----------------------------------------------------------------------
2022-05-18T04:37:09.6207935Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-05-18T04:37:09.6208301Z 
2022-05-18T04:37:09.6208587Z ----------------------------------------------------------------------
2022-05-18T04:37:09.6208929Z Ran 1 test in 0.002s
2022-05-18T04:37:09.6209094Z 
2022-05-18T04:37:09.6209187Z OK (skipped=1)
2022-05-18T04:37:09.6209353Z 
2022-05-18T04:37:09.6209480Z Generating XML reports...
2022-05-18T04:37:09.6245512Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043709.xml
2022-05-18T04:37:10.8639064Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:10.8653384Z 
2022-05-18T04:37:10.8653828Z Running tests...
2022-05-18T04:37:10.8654313Z ----------------------------------------------------------------------
2022-05-18T04:37:12.4492395Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:12.4887250Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39817
2022-05-18T04:37:12.4994382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39818
2022-05-18T04:37:13.6098026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:13.6284749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:13.6285594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:13.6300146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:13.6305998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:13.7299939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:15.5074489Z ok (4.642s)
2022-05-18T04:37:15.5074726Z 
2022-05-18T04:37:15.5075109Z ----------------------------------------------------------------------
2022-05-18T04:37:15.5075442Z Ran 1 test in 4.642s
2022-05-18T04:37:15.5075612Z 
2022-05-18T04:37:15.5075710Z OK
2022-05-18T04:37:15.5075849Z 
2022-05-18T04:37:15.5075986Z Generating XML reports...
2022-05-18T04:37:15.5117515Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043710.xml
2022-05-18T04:37:16.9282442Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:16.9296829Z 
2022-05-18T04:37:16.9297185Z Running tests...
2022-05-18T04:37:16.9298096Z ----------------------------------------------------------------------
2022-05-18T04:37:16.9321012Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-05-18T04:37:16.9321648Z 
2022-05-18T04:37:16.9322148Z ----------------------------------------------------------------------
2022-05-18T04:37:16.9322893Z Ran 1 test in 0.002s
2022-05-18T04:37:16.9323064Z 
2022-05-18T04:37:16.9323177Z OK (skipped=1)
2022-05-18T04:37:16.9323337Z 
2022-05-18T04:37:16.9323474Z Generating XML reports...
2022-05-18T04:37:16.9358409Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043716.xml
2022-05-18T04:37:18.1938141Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:18.1951837Z 
2022-05-18T04:37:18.1952125Z Running tests...
2022-05-18T04:37:18.1952835Z ----------------------------------------------------------------------
2022-05-18T04:37:19.7730338Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:19.8125359Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39970
2022-05-18T04:37:19.8232862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39971
2022-05-18T04:37:20.9662385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:21.0124528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:21.0125363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:21.0168787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:21.0175326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:21.1138979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:22.2969661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsolde8l
2022-05-18T04:37:22.2970750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsolde8l/_remote_module_non_scriptable.py
2022-05-18T04:37:22.3653243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqp5tbc_0
2022-05-18T04:37:22.3654090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqp5tbc_0/_remote_module_non_scriptable.py
2022-05-18T04:37:22.7310175Z ok (4.535s)
2022-05-18T04:37:22.7310450Z 
2022-05-18T04:37:22.7310846Z ----------------------------------------------------------------------
2022-05-18T04:37:22.7311192Z Ran 1 test in 4.536s
2022-05-18T04:37:22.7311361Z 
2022-05-18T04:37:22.7311439Z OK
2022-05-18T04:37:22.7311588Z 
2022-05-18T04:37:22.7311724Z Generating XML reports...
2022-05-18T04:37:22.7352790Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043718.xml
2022-05-18T04:37:24.1303431Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:24.1317538Z 
2022-05-18T04:37:24.1317707Z Running tests...
2022-05-18T04:37:24.1318451Z ----------------------------------------------------------------------
2022-05-18T04:37:25.6619550Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:25.7006083Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40086
2022-05-18T04:37:25.7110833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40087
2022-05-18T04:37:26.8533990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:26.9028056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:26.9029073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:26.9041733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:26.9047686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:27.0041946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:28.1514161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1gnokfe2
2022-05-18T04:37:28.1515239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1gnokfe2/_remote_module_non_scriptable.py
2022-05-18T04:37:28.2316221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9i81yrjv
2022-05-18T04:37:28.2317572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9i81yrjv/_remote_module_non_scriptable.py
2022-05-18T04:37:28.4658582Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:37:28.4676175Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:37:28.8191257Z ok (4.687s)
2022-05-18T04:37:28.8191474Z 
2022-05-18T04:37:28.8192265Z ----------------------------------------------------------------------
2022-05-18T04:37:28.8192798Z Ran 1 test in 4.687s
2022-05-18T04:37:28.8192969Z 
2022-05-18T04:37:28.8193073Z OK
2022-05-18T04:37:28.8193209Z 
2022-05-18T04:37:28.8193347Z Generating XML reports...
2022-05-18T04:37:28.8235962Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043724.xml
2022-05-18T04:37:30.2456491Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:30.2471104Z 
2022-05-18T04:37:30.2471569Z Running tests...
2022-05-18T04:37:30.2472086Z ----------------------------------------------------------------------
2022-05-18T04:37:31.8177681Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:31.8572117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40206
2022-05-18T04:37:31.8677155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40207
2022-05-18T04:37:32.9790312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:32.9797215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:32.9798354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:32.9895259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:32.9902579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:33.0808863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:33.0916406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:37:33.0916932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:37:33.0917897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:37:33.0918866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:37:33.0919688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:37:33.0920187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:37:33.0920858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:37:33.0921531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:37:34.3638841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdchew9mp
2022-05-18T04:37:34.3639855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdchew9mp/_remote_module_non_scriptable.py
2022-05-18T04:37:34.3931560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkbfkm02g
2022-05-18T04:37:34.3933888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkbfkm02g/_remote_module_non_scriptable.py
2022-05-18T04:37:40.1851275Z ok (9.938s)
2022-05-18T04:37:40.1851494Z 
2022-05-18T04:37:40.1852029Z ----------------------------------------------------------------------
2022-05-18T04:37:40.1852507Z Ran 1 test in 9.938s
2022-05-18T04:37:40.1852683Z 
2022-05-18T04:37:40.1852790Z OK
2022-05-18T04:37:40.1852927Z 
2022-05-18T04:37:40.1853061Z Generating XML reports...
2022-05-18T04:37:40.1893819Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043730.xml
2022-05-18T04:37:41.6023287Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-05-18T04:37:41.6038169Z 
2022-05-18T04:37:41.6038535Z Running tests...
2022-05-18T04:37:41.6039251Z ----------------------------------------------------------------------
2022-05-18T04:37:43.1705224Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:43.2092629Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40330
2022-05-18T04:37:43.2196414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40331
2022-05-18T04:37:44.3279460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:44.3711290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:44.3712454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:44.3785631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:44.3791709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:44.4722960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:44.4911169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:37:44.4911697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:37:44.4912390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:37:44.4913086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:37:44.4915238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:37:44.5016267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:37:44.5016948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:37:44.5017659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:37:45.7671557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2y0ccnls
2022-05-18T04:37:45.7672451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2y0ccnls/_remote_module_non_scriptable.py
2022-05-18T04:37:45.7701640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpba36vbpa
2022-05-18T04:37:45.7704673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpba36vbpa/_remote_module_non_scriptable.py
2022-05-18T04:37:51.3370742Z ok (9.733s)
2022-05-18T04:37:51.3370994Z 
2022-05-18T04:37:51.3371422Z ----------------------------------------------------------------------
2022-05-18T04:37:51.3371776Z Ran 1 test in 9.733s
2022-05-18T04:37:51.3371946Z 
2022-05-18T04:37:51.3372031Z OK
2022-05-18T04:37:51.3372169Z 
2022-05-18T04:37:51.3372307Z Generating XML reports...
2022-05-18T04:37:51.3413447Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043741.xml
2022-05-18T04:37:51.7415372Z Running distributed tests for the gloo backend with env init_method
2022-05-18T04:37:51.7418054Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:37:51.741427]
2022-05-18T04:37:52.8959831Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:37:52.9003079Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-05-18T04:37:52.9036912Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9037438Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9037857Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9038876Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9039339Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9039800Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9040289Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9040786Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9041304Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9041844Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9042407Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9042943Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9043464Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9043985Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9044465Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9044953Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9045404Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9045848Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9046282Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9046714Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9047317Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9047818Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9048231Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9048615Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9049055Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9049482Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9049886Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9050321Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9050745Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9051132Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9051516Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9051925Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9052319Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9052698Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9053105Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9053531Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9053932Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9054367Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9054931Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9055383Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9055818Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9056256Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9056690Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9057111Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9057550Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9057968Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9058418Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9058892Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9059319Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9059738Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9060148Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9060581Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9060996Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9061401Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9061816Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9062219Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9062611Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9063000Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9063410Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9063792Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9064151Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9064542Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9065016Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9065419Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9065817Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9066201Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9066589Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9066967Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9067366Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9067763Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9068168Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9068559Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9068939Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9069305Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9069695Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9070100Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9070503Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9070886Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9071275Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9071684Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9072102Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9072611Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9073058Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9073520Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9073966Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9074417Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9074864Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9075285Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9075724Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9076171Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9076630Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9077076Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9077542Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9078000Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9079095Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9079532Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9079932Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9080317Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9080669Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9081040Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9081423Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9081816Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9082209Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9082597Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9083081Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9083510Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9083915Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9084301Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9084715Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9085143Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9085561Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9085968Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9086385Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9086800Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9087218Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9087651Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9088068Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9088460Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9088818Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9089215Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9089610Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9089986Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9090477Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9090934Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9091432Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9091899Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9092320Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9092741Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9093160Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9093617Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9094087Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9094512Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9094948Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9095399Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9095814Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9096179Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9096572Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9096991Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9097390Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9097831Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9098274Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9098677Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9099147Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9099650Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9100276Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9100888Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9101496Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9102099Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9102706Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9103300Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9103902Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9104497Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9105047Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9105526Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9105980Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9106371Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9106855Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9107252Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9107657Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9108088Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9108518Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9108980Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9109454Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9109874Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9110252Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9110672Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9111116Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9111531Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9111955Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9112375Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9112792Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9113233Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9113660Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9114074Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9114469Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9114894Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9115304Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9115718Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9116183Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9116614Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9117056Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9117480Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9117933Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9119003Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9119379Z test_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9119746Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9120123Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9120491Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9120883Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9121260Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9121655Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9122031Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9122400Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9122760Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9123127Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9123528Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9123925Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9124284Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9124634Z test_isend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9125023Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9125522Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9125941Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9126403Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9126863Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9127266Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9127693Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9128132Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9128545Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9128975Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9129398Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9129826Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9130224Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9130638Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9131041Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9131428Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9131899Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9132399Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9132880Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9133316Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9133782Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9134246Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9134665Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9135168Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9135616Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9136041Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9136498Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9136983Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9137501Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9137946Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9138360Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9138771Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9139162Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9139558Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9139944Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9140341Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9140720Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9141096Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9141459Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9141818Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9142202Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9142575Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9143004Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9143394Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9143784Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9144162Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9144515Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9144899Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9145273Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9145644Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9146039Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9146421Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9146789Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9147162Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9147550Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9147958Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9148407Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9148843Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9149242Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9149633Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9150063Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9150482Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9150864Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9151282Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9151727Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9152145Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9152538Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9153005Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9153421Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9153794Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9154219Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9154674Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:52.9155100Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:37:54.0343152Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:37:54.0357236Z 
2022-05-18T04:37:54.0357579Z Running tests...
2022-05-18T04:37:54.0358025Z ----------------------------------------------------------------------
2022-05-18T04:37:55.5894816Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:55.6278608Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40487
2022-05-18T04:37:55.6380685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40488
2022-05-18T04:37:56.7801897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:56.7802448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:56.7803244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:56.7803946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:56.7810805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:56.7811277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:58.0494080Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:37:58.0495122Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:37:58.0807098Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:37:58.0807983Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:37:59.0640263Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:37:59.0642184Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:37:59.0643532Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:37:59.0645147Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:37:59.0789741Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:37:59.0791479Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:37:59.0792848Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:37:59.0794483Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:37:59.0939892Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:37:59.0941647Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:37:59.0942950Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:37:59.0944582Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:37:59.4474726Z ok (5.411s)
2022-05-18T04:37:59.4474959Z 
2022-05-18T04:37:59.4475367Z ----------------------------------------------------------------------
2022-05-18T04:37:59.4475712Z Ran 1 test in 5.412s
2022-05-18T04:37:59.4475863Z 
2022-05-18T04:37:59.4475967Z OK
2022-05-18T04:37:59.4476104Z 
2022-05-18T04:37:59.4476257Z Generating XML reports...
2022-05-18T04:37:59.4516937Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043754.xml
2022-05-18T04:38:00.8456280Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:38:00.8470306Z 
2022-05-18T04:38:00.8470437Z Running tests...
2022-05-18T04:38:00.8471355Z ----------------------------------------------------------------------
2022-05-18T04:38:00.8513287Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-05-18T04:38:00.8514246Z 
2022-05-18T04:38:00.8514861Z ----------------------------------------------------------------------
2022-05-18T04:38:00.8515222Z Ran 1 test in 0.004s
2022-05-18T04:38:00.8515395Z 
2022-05-18T04:38:00.8515524Z OK (skipped=1)
2022-05-18T04:38:00.8515683Z 
2022-05-18T04:38:00.8515812Z Generating XML reports...
2022-05-18T04:38:00.8550061Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043800.xml
2022-05-18T04:38:02.1169985Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:38:02.1184506Z 
2022-05-18T04:38:02.1184819Z Running tests...
2022-05-18T04:38:02.1185279Z ----------------------------------------------------------------------
2022-05-18T04:38:03.6922034Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:03.7314755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40632
2022-05-18T04:38:03.7419352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40633
2022-05-18T04:38:04.9259448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:04.9260043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:04.9260865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:04.9261575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:04.9267330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:04.9267809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:05.1471339Z ok (3.028s)
2022-05-18T04:38:05.1471570Z 
2022-05-18T04:38:05.1471966Z ----------------------------------------------------------------------
2022-05-18T04:38:05.1472315Z Ran 1 test in 3.029s
2022-05-18T04:38:05.1472480Z 
2022-05-18T04:38:05.1472583Z OK
2022-05-18T04:38:05.1472718Z 
2022-05-18T04:38:05.1472853Z Generating XML reports...
2022-05-18T04:38:05.1513290Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043802.xml
2022-05-18T04:38:06.5424457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:38:06.5438417Z 
2022-05-18T04:38:06.5438887Z Running tests...
2022-05-18T04:38:06.5439625Z ----------------------------------------------------------------------
2022-05-18T04:38:08.1285562Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:08.1439233Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.600s)
2022-05-18T04:38:08.1439837Z 
2022-05-18T04:38:08.1440124Z ----------------------------------------------------------------------
2022-05-18T04:38:08.1440465Z Ran 1 test in 1.600s
2022-05-18T04:38:08.1440632Z 
2022-05-18T04:38:08.1440751Z OK (skipped=1)
2022-05-18T04:38:08.1440892Z 
2022-05-18T04:38:08.1441018Z Generating XML reports...
2022-05-18T04:38:08.1472856Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043806.xml
2022-05-18T04:38:09.5161496Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:38:09.5175914Z 
2022-05-18T04:38:09.5176071Z Running tests...
2022-05-18T04:38:09.5176801Z ----------------------------------------------------------------------
2022-05-18T04:38:11.0845905Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:11.1231580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40775
2022-05-18T04:38:11.1335425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40776
2022-05-18T04:38:12.3200395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:12.3200949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:12.3201751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:12.3202449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:12.3310497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:12.3411894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwhilj6h
2022-05-18T04:38:12.3414785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwhilj6h/_remote_module_non_scriptable.py
2022-05-18T04:38:12.4211526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:12.4318240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiurhr22k
2022-05-18T04:38:12.4321234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiurhr22k/_remote_module_non_scriptable.py
2022-05-18T04:38:12.4526086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:12.4526590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:12.6388562Z ok (3.121s)
2022-05-18T04:38:12.6388784Z 
2022-05-18T04:38:12.6389190Z ----------------------------------------------------------------------
2022-05-18T04:38:12.6389531Z Ran 1 test in 3.121s
2022-05-18T04:38:12.6389707Z 
2022-05-18T04:38:12.6389811Z OK
2022-05-18T04:38:12.6389948Z 
2022-05-18T04:38:12.6390082Z Generating XML reports...
2022-05-18T04:38:12.6437841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043809.xml
2022-05-18T04:38:14.0444938Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:38:14.0459140Z 
2022-05-18T04:38:14.0459498Z Running tests...
2022-05-18T04:38:14.0459944Z ----------------------------------------------------------------------
2022-05-18T04:38:15.6177138Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:15.6570045Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40888
2022-05-18T04:38:15.6674137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40889
2022-05-18T04:38:16.8347744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:16.8348313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:16.8349124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:16.8349832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:16.8457893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:16.8560850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvc5xu38
2022-05-18T04:38:16.8563239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvc5xu38/_remote_module_non_scriptable.py
2022-05-18T04:38:16.9358945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:16.9461234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8b0k3mlv
2022-05-18T04:38:16.9464052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8b0k3mlv/_remote_module_non_scriptable.py
2022-05-18T04:38:16.9665520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:16.9666564Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:17.1725766Z ok (3.126s)
2022-05-18T04:38:17.1726246Z 
2022-05-18T04:38:17.1726733Z ----------------------------------------------------------------------
2022-05-18T04:38:17.1727126Z Ran 1 test in 3.127s
2022-05-18T04:38:17.1727308Z 
2022-05-18T04:38:17.1727406Z OK
2022-05-18T04:38:17.1727544Z 
2022-05-18T04:38:17.1727681Z Generating XML reports...
2022-05-18T04:38:17.1767667Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043814.xml
2022-05-18T04:38:18.6079492Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:38:18.6094350Z 
2022-05-18T04:38:18.6094495Z Running tests...
2022-05-18T04:38:18.6095381Z ----------------------------------------------------------------------
2022-05-18T04:38:20.1740980Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:20.2127524Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41001
2022-05-18T04:38:20.2232584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41002
2022-05-18T04:38:21.3756201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:21.3756768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:21.3757573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:21.3758489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:21.3866611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:21.4771079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:22.6389316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhbin1ui
2022-05-18T04:38:22.6390221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhbin1ui/_remote_module_non_scriptable.py
2022-05-18T04:38:22.7242680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6stoytnq
2022-05-18T04:38:22.7243981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6stoytnq/_remote_module_non_scriptable.py
2022-05-18T04:38:23.4808662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:23.4809234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:23.5061136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:23.5061685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:23.5381499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:23.5382033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:23.5627819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:23.5628325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:23.6923629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:23.6924143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:23.7173378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:23.7173902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:24.0326252Z ok (5.423s)
2022-05-18T04:38:24.0326438Z 
2022-05-18T04:38:24.0327241Z ----------------------------------------------------------------------
2022-05-18T04:38:24.0327610Z Ran 1 test in 5.423s
2022-05-18T04:38:24.0327779Z 
2022-05-18T04:38:24.0327877Z OK
2022-05-18T04:38:24.0328013Z 
2022-05-18T04:38:24.0328147Z Generating XML reports...
2022-05-18T04:38:24.0369088Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043818.xml
2022-05-18T04:38:25.4273674Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:38:25.4288757Z 
2022-05-18T04:38:25.4289259Z Running tests...
2022-05-18T04:38:27.0142209Z ----------------------------------------------------------------------
2022-05-18T04:38:27.0142820Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:27.0535610Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41116
2022-05-18T04:38:27.0641662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41117
2022-05-18T04:38:28.2137835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:28.2138387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:28.2139185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:28.2139876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:28.2146344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:28.2146993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:29.4835670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbczcf3c
2022-05-18T04:38:29.4836735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbczcf3c/_remote_module_non_scriptable.py
2022-05-18T04:38:29.4944335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ijloyx8
2022-05-18T04:38:29.4947222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ijloyx8/_remote_module_non_scriptable.py
2022-05-18T04:38:29.5185119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:29.5185626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:29.5355974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:29.5356484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:29.8717936Z ok (4.443s)
2022-05-18T04:38:29.8718416Z 
2022-05-18T04:38:29.8718818Z ----------------------------------------------------------------------
2022-05-18T04:38:29.8719169Z Ran 1 test in 4.443s
2022-05-18T04:38:29.8719339Z 
2022-05-18T04:38:29.8719438Z OK
2022-05-18T04:38:29.8719579Z 
2022-05-18T04:38:29.8719715Z Generating XML reports...
2022-05-18T04:38:29.8761071Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043825.xml
2022-05-18T04:38:31.3020143Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:38:31.3034145Z 
2022-05-18T04:38:31.3034416Z Running tests...
2022-05-18T04:38:31.3034854Z ----------------------------------------------------------------------
2022-05-18T04:38:32.8658324Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:32.9043980Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41231
2022-05-18T04:38:32.9147072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41232
2022-05-18T04:38:34.0920311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:34.0920895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:34.0921691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:34.0922393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:34.0928806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:34.0929837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:35.3560943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo828sj2j
2022-05-18T04:38:35.3561943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo828sj2j/_remote_module_non_scriptable.py
2022-05-18T04:38:35.3924465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79r6n0u6
2022-05-18T04:38:35.3926943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79r6n0u6/_remote_module_non_scriptable.py
2022-05-18T04:38:35.4187705Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:35.4188220Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:35.4388899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:35.4389389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:35.7222304Z ok (4.418s)
2022-05-18T04:38:35.7222543Z 
2022-05-18T04:38:35.7222951Z ----------------------------------------------------------------------
2022-05-18T04:38:35.7223278Z Ran 1 test in 4.419s
2022-05-18T04:38:35.7223447Z 
2022-05-18T04:38:35.7223544Z OK
2022-05-18T04:38:35.7223987Z 
2022-05-18T04:38:35.7224147Z Generating XML reports...
2022-05-18T04:38:35.7263667Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043831.xml
2022-05-18T04:38:37.1344348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:38:37.1358588Z 
2022-05-18T04:38:37.1358905Z Running tests...
2022-05-18T04:38:37.1359362Z ----------------------------------------------------------------------
2022-05-18T04:38:38.7000473Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:38.7387184Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41346
2022-05-18T04:38:38.7491807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41347
2022-05-18T04:38:39.9305215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:39.9305758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:39.9306555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:39.9307258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:39.9415132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:40.0319240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:41.1871378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7o9a54o
2022-05-18T04:38:41.1871990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7o9a54o/_remote_module_non_scriptable.py
2022-05-18T04:38:41.2672701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpditel61l
2022-05-18T04:38:41.2673793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpditel61l/_remote_module_non_scriptable.py
2022-05-18T04:38:41.2891019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:41.2891528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:41.7569775Z ok (4.621s)
2022-05-18T04:38:41.7569987Z 
2022-05-18T04:38:41.7570768Z ----------------------------------------------------------------------
2022-05-18T04:38:41.7571164Z Ran 1 test in 4.621s
2022-05-18T04:38:41.7571334Z 
2022-05-18T04:38:41.7571435Z OK
2022-05-18T04:38:41.7571572Z 
2022-05-18T04:38:41.7571708Z Generating XML reports...
2022-05-18T04:38:41.7611146Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043837.xml
2022-05-18T04:38:43.1754456Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:38:43.1768802Z 
2022-05-18T04:38:43.1769274Z Running tests...
2022-05-18T04:38:43.1769752Z ----------------------------------------------------------------------
2022-05-18T04:38:44.7474700Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:44.7869631Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41461
2022-05-18T04:38:44.7974800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41462
2022-05-18T04:38:46.0367116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:46.0367692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:46.0368498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:46.0369410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:46.0475128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:46.1381360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:47.3035234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3bjmlwa
2022-05-18T04:38:47.3036405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3bjmlwa/_remote_module_non_scriptable.py
2022-05-18T04:38:47.3965421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsfet_8hr
2022-05-18T04:38:47.3966565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsfet_8hr/_remote_module_non_scriptable.py
2022-05-18T04:38:48.1515987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:48.1516554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:48.1771375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:48.1771920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:48.5065920Z ok (5.329s)
2022-05-18T04:38:48.5066106Z 
2022-05-18T04:38:48.5066478Z ----------------------------------------------------------------------
2022-05-18T04:38:48.5066822Z Ran 1 test in 5.330s
2022-05-18T04:38:48.5066996Z 
2022-05-18T04:38:48.5067097Z OK
2022-05-18T04:38:48.5067234Z 
2022-05-18T04:38:48.5067592Z Generating XML reports...
2022-05-18T04:38:48.5107687Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043843.xml
2022-05-18T04:38:49.9173555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:38:49.9186883Z 
2022-05-18T04:38:49.9187217Z Running tests...
2022-05-18T04:38:49.9187669Z ----------------------------------------------------------------------
2022-05-18T04:38:51.4876763Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:51.5267483Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41576
2022-05-18T04:38:51.5371507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41577
2022-05-18T04:38:52.6831139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:52.6831690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:52.6832509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:52.6833227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:52.6938121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:52.7844593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:53.9295533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw191quia
2022-05-18T04:38:53.9296150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw191quia/_remote_module_non_scriptable.py
2022-05-18T04:38:54.0422844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpblw02ruc
2022-05-18T04:38:54.0424178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpblw02ruc/_remote_module_non_scriptable.py
2022-05-18T04:38:54.2399664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:54.2400227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:54.2613005Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:54.2613552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:54.5452389Z ok (4.626s)
2022-05-18T04:38:54.5452576Z 
2022-05-18T04:38:54.5452987Z ----------------------------------------------------------------------
2022-05-18T04:38:54.5453335Z Ran 1 test in 4.627s
2022-05-18T04:38:54.5453504Z 
2022-05-18T04:38:54.5453585Z OK
2022-05-18T04:38:54.5453724Z 
2022-05-18T04:38:54.5453854Z Generating XML reports...
2022-05-18T04:38:54.5495348Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043849.xml
2022-05-18T04:38:55.9501564Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:38:55.9515730Z 
2022-05-18T04:38:55.9516031Z Running tests...
2022-05-18T04:38:55.9516495Z ----------------------------------------------------------------------
2022-05-18T04:38:57.4792435Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:57.5178630Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41691
2022-05-18T04:38:57.5282797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41692
2022-05-18T04:38:58.7148114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:58.7148679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:58.7149764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:58.7150461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:58.7256564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:58.8161519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:59.9756950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2n45bqy
2022-05-18T04:38:59.9757839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2n45bqy/_remote_module_non_scriptable.py
2022-05-18T04:39:00.0861972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9tt9g6k
2022-05-18T04:39:00.0863260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9tt9g6k/_remote_module_non_scriptable.py
2022-05-18T04:39:00.1105360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:39:00.1106209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:39:00.1273298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:39:00.1274180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:39:00.4360054Z ok (4.484s)
2022-05-18T04:39:00.4360282Z 
2022-05-18T04:39:00.4360866Z ----------------------------------------------------------------------
2022-05-18T04:39:00.4361291Z Ran 1 test in 4.484s
2022-05-18T04:39:00.4361459Z 
2022-05-18T04:39:00.4361561Z OK
2022-05-18T04:39:00.4361698Z 
2022-05-18T04:39:00.4361834Z Generating XML reports...
2022-05-18T04:39:00.4403151Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043855.xml
2022-05-18T04:39:01.8538204Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:01.8552320Z 
2022-05-18T04:39:01.8552659Z Running tests...
2022-05-18T04:39:01.8553113Z ----------------------------------------------------------------------
2022-05-18T04:39:03.4441002Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:03.4595396Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.604s)
2022-05-18T04:39:03.4596329Z 
2022-05-18T04:39:03.4596648Z ----------------------------------------------------------------------
2022-05-18T04:39:03.4608095Z Ran 1 test in 1.604s
2022-05-18T04:39:03.4608283Z 
2022-05-18T04:39:03.4608411Z OK (skipped=1)
2022-05-18T04:39:03.4608587Z 
2022-05-18T04:39:03.4608720Z Generating XML reports...
2022-05-18T04:39:03.4630941Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043901.xml
2022-05-18T04:39:04.8204258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:04.8218724Z 
2022-05-18T04:39:04.8219114Z Running tests...
2022-05-18T04:39:04.8219549Z ----------------------------------------------------------------------
2022-05-18T04:39:06.3732047Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:06.4122145Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41840
2022-05-18T04:39:06.4225490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41841
2022-05-18T04:39:07.5964469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:07.5965263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:07.5966084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:07.5966791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:07.6072199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:07.6975194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:08.0282738Z ok (3.206s)
2022-05-18T04:39:08.0282909Z 
2022-05-18T04:39:08.0283289Z ----------------------------------------------------------------------
2022-05-18T04:39:08.0283645Z Ran 1 test in 3.206s
2022-05-18T04:39:08.0283814Z 
2022-05-18T04:39:08.0283913Z OK
2022-05-18T04:39:08.0284070Z 
2022-05-18T04:39:08.0284184Z Generating XML reports...
2022-05-18T04:39:08.0325748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043904.xml
2022-05-18T04:39:09.4186230Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:09.4199955Z 
2022-05-18T04:39:09.4200409Z Running tests...
2022-05-18T04:39:09.4200899Z ----------------------------------------------------------------------
2022-05-18T04:39:10.9576149Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:10.9725616Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.552s)
2022-05-18T04:39:10.9726214Z 
2022-05-18T04:39:10.9726475Z ----------------------------------------------------------------------
2022-05-18T04:39:10.9726814Z Ran 1 test in 1.553s
2022-05-18T04:39:10.9726981Z 
2022-05-18T04:39:10.9727094Z OK (skipped=1)
2022-05-18T04:39:10.9727253Z 
2022-05-18T04:39:10.9727382Z Generating XML reports...
2022-05-18T04:39:10.9758636Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043909.xml
2022-05-18T04:39:12.3271818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:12.3285863Z 
2022-05-18T04:39:12.3286016Z Running tests...
2022-05-18T04:39:12.3286852Z ----------------------------------------------------------------------
2022-05-18T04:39:13.8737873Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:13.9126846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41983
2022-05-18T04:39:13.9227303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41984
2022-05-18T04:39:15.1288674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:15.1289258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:15.1290059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:15.1290766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:15.1297298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:15.1297799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:16.7304134Z ok (4.401s)
2022-05-18T04:39:16.7304365Z 
2022-05-18T04:39:16.7304743Z ----------------------------------------------------------------------
2022-05-18T04:39:16.7305435Z Ran 1 test in 4.402s
2022-05-18T04:39:16.7305603Z 
2022-05-18T04:39:16.7305701Z OK
2022-05-18T04:39:16.7305840Z 
2022-05-18T04:39:16.7305981Z Generating XML reports...
2022-05-18T04:39:16.7346080Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043912.xml
2022-05-18T04:39:18.1434631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:18.1451444Z 
2022-05-18T04:39:18.1451991Z Running tests...
2022-05-18T04:39:18.1452497Z ----------------------------------------------------------------------
2022-05-18T04:39:18.1473431Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-05-18T04:39:18.1473833Z 
2022-05-18T04:39:18.1474876Z ----------------------------------------------------------------------
2022-05-18T04:39:18.1475554Z Ran 1 test in 0.002s
2022-05-18T04:39:18.1475906Z 
2022-05-18T04:39:18.1476138Z OK (skipped=1)
2022-05-18T04:39:18.1476438Z 
2022-05-18T04:39:18.1476686Z Generating XML reports...
2022-05-18T04:39:18.1513793Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043918.xml
2022-05-18T04:39:19.3991080Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:19.4007389Z 
2022-05-18T04:39:19.4007883Z Running tests...
2022-05-18T04:39:19.4008405Z ----------------------------------------------------------------------
2022-05-18T04:39:19.4026797Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:39:21.0027287Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:21.0423445Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42127
2022-05-18T04:39:21.0528654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42128
2022-05-18T04:39:22.2306278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:22.2306837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:22.2307895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:22.2308612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:22.2416819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:22.2533427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9uv3d27
2022-05-18T04:39:22.2536662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9uv3d27/_remote_module_non_scriptable.py
2022-05-18T04:39:22.3316595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:22.3427231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppryts5bl
2022-05-18T04:39:22.3429932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppryts5bl/_remote_module_non_scriptable.py
2022-05-18T04:39:22.5582298Z ok (3.157s)
2022-05-18T04:39:22.5582494Z 
2022-05-18T04:39:22.5582887Z ----------------------------------------------------------------------
2022-05-18T04:39:22.5583240Z Ran 1 test in 3.158s
2022-05-18T04:39:22.5583389Z 
2022-05-18T04:39:22.5583485Z OK
2022-05-18T04:39:22.5583622Z 
2022-05-18T04:39:22.5583755Z Generating XML reports...
2022-05-18T04:39:22.5624681Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043919.xml
2022-05-18T04:39:23.9203896Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:23.9219753Z 
2022-05-18T04:39:23.9220210Z Running tests...
2022-05-18T04:39:23.9221045Z ----------------------------------------------------------------------
2022-05-18T04:39:23.9243952Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:39:25.5055887Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:25.5463091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42240
2022-05-18T04:39:25.5567234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42241
2022-05-18T04:39:26.7008619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:26.7009477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:26.7010452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:26.7011238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:26.7017301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:26.7018031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:26.7124058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6t414nl
2022-05-18T04:39:26.7126509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6t414nl/_remote_module_non_scriptable.py
2022-05-18T04:39:26.7128622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0cgjcjc7
2022-05-18T04:39:26.7131520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0cgjcjc7/_remote_module_non_scriptable.py
2022-05-18T04:39:26.7380922Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:39:26.7381412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:39:26.9619563Z ok (3.040s)
2022-05-18T04:39:26.9619792Z 
2022-05-18T04:39:26.9620190Z ----------------------------------------------------------------------
2022-05-18T04:39:26.9620531Z Ran 1 test in 3.040s
2022-05-18T04:39:26.9620681Z 
2022-05-18T04:39:26.9620788Z OK
2022-05-18T04:39:26.9621188Z 
2022-05-18T04:39:26.9621341Z Generating XML reports...
2022-05-18T04:39:26.9662429Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043923.xml
2022-05-18T04:39:28.3550687Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:28.3565366Z 
2022-05-18T04:39:28.3565674Z Running tests...
2022-05-18T04:39:28.3566111Z ----------------------------------------------------------------------
2022-05-18T04:39:28.3590073Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:39:29.9288265Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:29.9692720Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42353
2022-05-18T04:39:29.9797243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42354
2022-05-18T04:39:31.1942027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:31.1942596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:31.1943388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:31.1944099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:31.1949557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:31.1950300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:31.2058420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkxdvgor
2022-05-18T04:39:31.2061492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkxdvgor/_remote_module_non_scriptable.py
2022-05-18T04:39:31.2062050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmvhee_t
2022-05-18T04:39:31.2065087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmvhee_t/_remote_module_non_scriptable.py
2022-05-18T04:39:31.2316605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:39:31.2317103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:39:31.3847864Z ok (3.028s)
2022-05-18T04:39:31.3848095Z 
2022-05-18T04:39:31.3848489Z ----------------------------------------------------------------------
2022-05-18T04:39:31.3849070Z Ran 1 test in 3.028s
2022-05-18T04:39:31.3849350Z 
2022-05-18T04:39:31.3849448Z OK
2022-05-18T04:39:31.3849567Z 
2022-05-18T04:39:31.3849706Z Generating XML reports...
2022-05-18T04:39:31.3890198Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043928.xml
2022-05-18T04:39:32.7842000Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:32.7855881Z 
2022-05-18T04:39:32.7856327Z Running tests...
2022-05-18T04:39:32.7856841Z ----------------------------------------------------------------------
2022-05-18T04:39:32.7874813Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:39:34.3692191Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:34.4089478Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42466
2022-05-18T04:39:34.4196167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42467
2022-05-18T04:39:35.6051433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:35.6052812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:35.6053684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:35.6054418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:35.6159546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:35.6267882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn22qjl6j
2022-05-18T04:39:35.6271011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn22qjl6j/_remote_module_non_scriptable.py
2022-05-18T04:39:35.7061285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:35.7175144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_ncg3_d
2022-05-18T04:39:35.7177856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_ncg3_d/_remote_module_non_scriptable.py
2022-05-18T04:39:35.9251117Z ok (3.139s)
2022-05-18T04:39:35.9251338Z 
2022-05-18T04:39:35.9251793Z ----------------------------------------------------------------------
2022-05-18T04:39:35.9252123Z Ran 1 test in 3.139s
2022-05-18T04:39:35.9252296Z 
2022-05-18T04:39:35.9252395Z OK
2022-05-18T04:39:35.9252532Z 
2022-05-18T04:39:35.9252668Z Generating XML reports...
2022-05-18T04:39:35.9293048Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043932.xml
2022-05-18T04:39:37.3215838Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:37.3231030Z 
2022-05-18T04:39:37.3231417Z Running tests...
2022-05-18T04:39:37.3231943Z ----------------------------------------------------------------------
2022-05-18T04:39:38.8893977Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:38.9291987Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42579
2022-05-18T04:39:38.9397433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42580
2022-05-18T04:39:40.0741528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:40.0742092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:40.0742863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:40.0743562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:40.0848958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:40.1753354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:40.3449375Z ok (3.021s)
2022-05-18T04:39:40.3449619Z 
2022-05-18T04:39:40.3450025Z ----------------------------------------------------------------------
2022-05-18T04:39:40.3450355Z Ran 1 test in 3.022s
2022-05-18T04:39:40.3450521Z 
2022-05-18T04:39:40.3450620Z OK
2022-05-18T04:39:40.3450755Z 
2022-05-18T04:39:40.3450891Z Generating XML reports...
2022-05-18T04:39:40.3491984Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043937.xml
2022-05-18T04:39:41.7491316Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:41.7506337Z 
2022-05-18T04:39:41.7506472Z Running tests...
2022-05-18T04:39:41.7506956Z ----------------------------------------------------------------------
2022-05-18T04:39:43.3267369Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:43.3655106Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42692
2022-05-18T04:39:43.3759361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42693
2022-05-18T04:39:44.5532420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:44.5532995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:44.5533807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:44.5534505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:44.5640208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:44.6547676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:44.8813418Z ok (3.130s)
2022-05-18T04:39:44.8813643Z 
2022-05-18T04:39:44.8814065Z ----------------------------------------------------------------------
2022-05-18T04:39:44.8814394Z Ran 1 test in 3.131s
2022-05-18T04:39:44.8814564Z 
2022-05-18T04:39:44.8817393Z OK
2022-05-18T04:39:44.8817925Z 
2022-05-18T04:39:44.8818363Z Generating XML reports...
2022-05-18T04:39:44.8854924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043941.xml
2022-05-18T04:39:46.2945966Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:46.2961447Z 
2022-05-18T04:39:46.2961774Z Running tests...
2022-05-18T04:39:46.2962199Z ----------------------------------------------------------------------
2022-05-18T04:39:47.8728177Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:47.9125689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42805
2022-05-18T04:39:47.9230386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42806
2022-05-18T04:39:49.1136041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:49.1137198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:49.1137983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:49.1138685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:49.1243770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:49.2147236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:49.2354546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:39:49.2355687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:39:49.2356490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:39:49.2357188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:39:49.4281483Z ok (3.132s)
2022-05-18T04:39:49.4281855Z 
2022-05-18T04:39:49.4282417Z ----------------------------------------------------------------------
2022-05-18T04:39:49.4282769Z Ran 1 test in 3.132s
2022-05-18T04:39:49.4282935Z 
2022-05-18T04:39:49.4283048Z OK
2022-05-18T04:39:49.4283194Z 
2022-05-18T04:39:49.4283442Z Generating XML reports...
2022-05-18T04:39:49.4324737Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043946.xml
2022-05-18T04:39:50.8465129Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:50.8480047Z 
2022-05-18T04:39:50.8480454Z Running tests...
2022-05-18T04:39:50.8480951Z ----------------------------------------------------------------------
2022-05-18T04:39:52.4460530Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:52.4848093Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42924
2022-05-18T04:39:52.4952218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42925
2022-05-18T04:39:53.7047342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:53.7048130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:53.7049163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:53.7049884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:53.7054932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:53.7055794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:53.9002785Z skip: Skipped due to small world size. (3.052s)
2022-05-18T04:39:53.9003051Z 
2022-05-18T04:39:53.9003427Z ----------------------------------------------------------------------
2022-05-18T04:39:53.9003767Z Ran 1 test in 3.052s
2022-05-18T04:39:53.9003915Z 
2022-05-18T04:39:53.9004026Z OK (skipped=1)
2022-05-18T04:39:53.9004470Z 
2022-05-18T04:39:53.9004600Z Generating XML reports...
2022-05-18T04:39:53.9045618Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043950.xml
2022-05-18T04:39:55.3167233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:55.3182555Z 
2022-05-18T04:39:55.3182839Z Running tests...
2022-05-18T04:39:55.3183290Z ----------------------------------------------------------------------
2022-05-18T04:39:56.8909463Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:56.9296407Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43033
2022-05-18T04:39:56.9400783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43034
2022-05-18T04:39:58.0955148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:39:58.0955722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:39:58.0956537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:58.0957256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:39:58.1062768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:58.1965982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:58.4452804Z ok (3.127s)
2022-05-18T04:39:58.4453032Z 
2022-05-18T04:39:58.4453434Z ----------------------------------------------------------------------
2022-05-18T04:39:58.4453778Z Ran 1 test in 3.127s
2022-05-18T04:39:58.4453949Z 
2022-05-18T04:39:58.4454038Z OK
2022-05-18T04:39:58.4454180Z 
2022-05-18T04:39:58.4454323Z Generating XML reports...
2022-05-18T04:39:58.4494208Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043955.xml
2022-05-18T04:39:59.8789057Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:39:59.8804496Z 
2022-05-18T04:39:59.8805167Z Running tests...
2022-05-18T04:39:59.8805667Z ----------------------------------------------------------------------
2022-05-18T04:40:01.4498896Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:01.4904636Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43146
2022-05-18T04:40:01.5016256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43147
2022-05-18T04:40:02.6911587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:02.6912186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:02.6913015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:02.6913725Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:02.6919358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:02.6920151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:02.9067151Z ok (3.026s)
2022-05-18T04:40:02.9067374Z 
2022-05-18T04:40:02.9067791Z ----------------------------------------------------------------------
2022-05-18T04:40:02.9068138Z Ran 1 test in 3.026s
2022-05-18T04:40:02.9068306Z 
2022-05-18T04:40:02.9068402Z OK
2022-05-18T04:40:02.9068538Z 
2022-05-18T04:40:02.9068652Z Generating XML reports...
2022-05-18T04:40:02.9109719Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043959.xml
2022-05-18T04:40:04.3105779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:04.3120331Z 
2022-05-18T04:40:04.3120648Z Running tests...
2022-05-18T04:40:04.3121115Z ----------------------------------------------------------------------
2022-05-18T04:40:05.8810000Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:05.9196769Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43259
2022-05-18T04:40:05.9301949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43260
2022-05-18T04:40:07.1059870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:07.1060437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:07.1061232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:07.1061946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:07.1067933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:07.1068425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:07.3353121Z ok (3.023s)
2022-05-18T04:40:07.3353318Z 
2022-05-18T04:40:07.3353709Z ----------------------------------------------------------------------
2022-05-18T04:40:07.3354047Z Ran 1 test in 3.023s
2022-05-18T04:40:07.3354218Z 
2022-05-18T04:40:07.3354314Z OK
2022-05-18T04:40:07.3354450Z 
2022-05-18T04:40:07.3354568Z Generating XML reports...
2022-05-18T04:40:07.3395878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044004.xml
2022-05-18T04:40:08.7387154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:08.7402212Z 
2022-05-18T04:40:08.7402484Z Running tests...
2022-05-18T04:40:08.7402922Z ----------------------------------------------------------------------
2022-05-18T04:40:08.7427447Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-05-18T04:40:08.7427793Z 
2022-05-18T04:40:08.7428082Z ----------------------------------------------------------------------
2022-05-18T04:40:08.7428452Z Ran 1 test in 0.003s
2022-05-18T04:40:08.7428624Z 
2022-05-18T04:40:08.7428737Z OK (skipped=1)
2022-05-18T04:40:08.7428903Z 
2022-05-18T04:40:08.7429035Z Generating XML reports...
2022-05-18T04:40:08.7471749Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044008.xml
2022-05-18T04:40:10.0010689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:10.0025951Z 
2022-05-18T04:40:10.0026515Z Running tests...
2022-05-18T04:40:10.0027013Z ----------------------------------------------------------------------
2022-05-18T04:40:10.0051901Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-05-18T04:40:10.0052282Z 
2022-05-18T04:40:10.0052804Z ----------------------------------------------------------------------
2022-05-18T04:40:10.0053150Z Ran 1 test in 0.003s
2022-05-18T04:40:10.0053327Z 
2022-05-18T04:40:10.0053439Z OK (skipped=1)
2022-05-18T04:40:10.0053600Z 
2022-05-18T04:40:10.0053726Z Generating XML reports...
2022-05-18T04:40:10.0097231Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044009.xml
2022-05-18T04:40:11.2677997Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:11.2693210Z 
2022-05-18T04:40:11.2693710Z Running tests...
2022-05-18T04:40:11.2694161Z ----------------------------------------------------------------------
2022-05-18T04:40:12.8411046Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:12.8797147Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43438
2022-05-18T04:40:12.8899898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43439
2022-05-18T04:40:14.0309654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:14.0310193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:14.0310987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:14.0311686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:14.0420067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:14.1320747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:14.1433333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:40:14.1433843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:40:14.1434536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:14.1435413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:14.3950790Z ok (3.125s)
2022-05-18T04:40:14.3951016Z 
2022-05-18T04:40:14.3951396Z ----------------------------------------------------------------------
2022-05-18T04:40:14.3951755Z Ran 1 test in 3.126s
2022-05-18T04:40:14.3951902Z 
2022-05-18T04:40:14.3952003Z OK
2022-05-18T04:40:14.3952138Z 
2022-05-18T04:40:14.3952272Z Generating XML reports...
2022-05-18T04:40:14.3992959Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044011.xml
2022-05-18T04:40:15.7976699Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:15.7991380Z 
2022-05-18T04:40:15.7991752Z Running tests...
2022-05-18T04:40:15.7992201Z ----------------------------------------------------------------------
2022-05-18T04:40:17.3749917Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:17.4146255Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43557
2022-05-18T04:40:17.4251457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43558
2022-05-18T04:40:18.6398353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:18.6398959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:18.6399755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:18.6400459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:18.6406644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:18.6407419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:18.8302049Z skip: Skipped due to small world size. (3.031s)
2022-05-18T04:40:18.8302303Z 
2022-05-18T04:40:18.8302700Z ----------------------------------------------------------------------
2022-05-18T04:40:18.8303025Z Ran 1 test in 3.031s
2022-05-18T04:40:18.8303455Z 
2022-05-18T04:40:18.8303568Z OK (skipped=1)
2022-05-18T04:40:18.8303723Z 
2022-05-18T04:40:18.8303853Z Generating XML reports...
2022-05-18T04:40:18.8344403Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044015.xml
2022-05-18T04:40:20.2357115Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:20.2371672Z 
2022-05-18T04:40:20.2372050Z Running tests...
2022-05-18T04:40:20.2372787Z ----------------------------------------------------------------------
2022-05-18T04:40:20.2393179Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-05-18T04:40:20.2393529Z 
2022-05-18T04:40:20.2393845Z ----------------------------------------------------------------------
2022-05-18T04:40:20.2394402Z Ran 1 test in 0.002s
2022-05-18T04:40:20.2394570Z 
2022-05-18T04:40:20.2394680Z OK (skipped=1)
2022-05-18T04:40:20.2394850Z 
2022-05-18T04:40:20.2394976Z Generating XML reports...
2022-05-18T04:40:20.2430134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044020.xml
2022-05-18T04:40:21.5047995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:21.5062283Z 
2022-05-18T04:40:21.5062523Z Running tests...
2022-05-18T04:40:21.5062975Z ----------------------------------------------------------------------
2022-05-18T04:40:21.5084084Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-05-18T04:40:21.5084439Z 
2022-05-18T04:40:21.5084729Z ----------------------------------------------------------------------
2022-05-18T04:40:21.5085060Z Ran 1 test in 0.002s
2022-05-18T04:40:21.5085226Z 
2022-05-18T04:40:21.5085343Z OK (skipped=1)
2022-05-18T04:40:21.5085509Z 
2022-05-18T04:40:21.5085619Z Generating XML reports...
2022-05-18T04:40:21.5121860Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044021.xml
2022-05-18T04:40:22.7707503Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:22.7721866Z 
2022-05-18T04:40:22.7722709Z Running tests...
2022-05-18T04:40:22.7723232Z ----------------------------------------------------------------------
2022-05-18T04:40:24.3424413Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:24.3817160Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43732
2022-05-18T04:40:24.3922173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43733
2022-05-18T04:40:25.5785838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:25.5786428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:25.5787243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:25.5787953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:25.5895383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:25.6799026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:25.8976193Z ok (3.125s)
2022-05-18T04:40:25.8976417Z 
2022-05-18T04:40:25.8976819Z ----------------------------------------------------------------------
2022-05-18T04:40:25.8977143Z Ran 1 test in 3.125s
2022-05-18T04:40:25.8977313Z 
2022-05-18T04:40:25.8977408Z OK
2022-05-18T04:40:25.8977545Z 
2022-05-18T04:40:25.8977680Z Generating XML reports...
2022-05-18T04:40:25.9018706Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044022.xml
2022-05-18T04:40:27.3028397Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:27.3043754Z 
2022-05-18T04:40:27.3044158Z Running tests...
2022-05-18T04:40:27.3044648Z ----------------------------------------------------------------------
2022-05-18T04:40:28.8733176Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:28.9119085Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43841
2022-05-18T04:40:28.9224894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43842
2022-05-18T04:40:30.0925651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:30.0926203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:30.0927000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:30.0927702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:30.0933857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:30.0934332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:30.1149753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:40:30.1150280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:40:30.1150970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:30.1151653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:30.1294145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:40:30.1294673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:40:30.1295587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:40:30.1296295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:40:30.1517242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:40:30.1518647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:40:30.1519467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:40:30.1619997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:40:30.4278627Z ok (3.123s)
2022-05-18T04:40:30.4278885Z 
2022-05-18T04:40:30.4279274Z ----------------------------------------------------------------------
2022-05-18T04:40:30.4279622Z Ran 1 test in 3.123s
2022-05-18T04:40:30.4279790Z 
2022-05-18T04:40:30.4279890Z OK
2022-05-18T04:40:30.4280028Z 
2022-05-18T04:40:30.4280165Z Generating XML reports...
2022-05-18T04:40:30.4322415Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044027.xml
2022-05-18T04:40:31.8549367Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:31.8568726Z 
2022-05-18T04:40:31.8569315Z Running tests...
2022-05-18T04:40:31.8569789Z ----------------------------------------------------------------------
2022-05-18T04:40:33.4462079Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:33.4861397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43974
2022-05-18T04:40:33.4968349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43975
2022-05-18T04:40:34.6866511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:34.6867206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:34.6868004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:34.6868712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:34.6975836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:34.7877303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:34.7991076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:40:34.7991840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:40:34.7992624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:34.7993333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:35.0020851Z ok (3.145s)
2022-05-18T04:40:35.0021050Z 
2022-05-18T04:40:35.0021496Z ----------------------------------------------------------------------
2022-05-18T04:40:35.0022074Z Ran 1 test in 3.145s
2022-05-18T04:40:35.0022246Z 
2022-05-18T04:40:35.0022344Z OK
2022-05-18T04:40:35.0022479Z 
2022-05-18T04:40:35.0022632Z Generating XML reports...
2022-05-18T04:40:35.0064817Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044031.xml
2022-05-18T04:40:36.3890326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:36.3903676Z 
2022-05-18T04:40:36.3904410Z Running tests...
2022-05-18T04:40:36.3904918Z ----------------------------------------------------------------------
2022-05-18T04:40:37.9420181Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:37.9811953Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44093
2022-05-18T04:40:37.9914484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44094
2022-05-18T04:40:39.1554995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:39.1555539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:39.1556326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:39.1557034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:39.1664703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:39.2566059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:39.2775637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:40:39.2776165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:40:39.2776859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:39.2777817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:39.4967228Z ok (3.106s)
2022-05-18T04:40:39.4967693Z 
2022-05-18T04:40:39.4968299Z ----------------------------------------------------------------------
2022-05-18T04:40:39.4968661Z Ran 1 test in 3.106s
2022-05-18T04:40:39.4968833Z 
2022-05-18T04:40:39.4968933Z OK
2022-05-18T04:40:39.4969076Z 
2022-05-18T04:40:39.4969188Z Generating XML reports...
2022-05-18T04:40:39.5009293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044036.xml
2022-05-18T04:40:40.9181745Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:40.9197120Z 
2022-05-18T04:40:40.9197524Z Running tests...
2022-05-18T04:40:40.9198018Z ----------------------------------------------------------------------
2022-05-18T04:40:42.5035341Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:42.5432472Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44212
2022-05-18T04:40:42.5538485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44213
2022-05-18T04:40:43.7355174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:43.7355762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:43.7356549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:43.7357251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:43.7364940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:43.7365523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:43.7573051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:40:43.7573790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:40:43.7574488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:43.7575227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:43.9590366Z ok (3.039s)
2022-05-18T04:40:43.9590592Z 
2022-05-18T04:40:43.9591179Z ----------------------------------------------------------------------
2022-05-18T04:40:43.9591513Z Ran 1 test in 3.039s
2022-05-18T04:40:43.9591683Z 
2022-05-18T04:40:43.9591789Z OK
2022-05-18T04:40:43.9591931Z 
2022-05-18T04:40:43.9592095Z Generating XML reports...
2022-05-18T04:40:43.9633077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044040.xml
2022-05-18T04:40:45.3659443Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:45.3674012Z 
2022-05-18T04:40:45.3674291Z Running tests...
2022-05-18T04:40:45.3674746Z ----------------------------------------------------------------------
2022-05-18T04:40:46.9412468Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:46.9808591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44331
2022-05-18T04:40:46.9915450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44332
2022-05-18T04:40:48.1767906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:48.1768537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:48.1769604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:48.1770317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:48.1775528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:48.1776680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:48.1884412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:40:48.1885196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:40:48.1885894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:48.1886586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:40:48.3966076Z ok (3.029s)
2022-05-18T04:40:48.3966308Z 
2022-05-18T04:40:48.3966692Z ----------------------------------------------------------------------
2022-05-18T04:40:48.3967064Z Ran 1 test in 3.029s
2022-05-18T04:40:48.3967233Z 
2022-05-18T04:40:48.3967333Z OK
2022-05-18T04:40:48.3967477Z 
2022-05-18T04:40:48.3967616Z Generating XML reports...
2022-05-18T04:40:48.4008229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044045.xml
2022-05-18T04:40:49.7805072Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:49.7818564Z 
2022-05-18T04:40:49.7819079Z Running tests...
2022-05-18T04:40:49.7819699Z ----------------------------------------------------------------------
2022-05-18T04:40:51.3190116Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:51.3577692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44450
2022-05-18T04:40:51.3684974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44451
2022-05-18T04:40:52.5225531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:52.5226141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:52.5226929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:52.5227616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:52.5334839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:52.6239353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:52.7735832Z skip: Skipped due to small world size. (2.991s)
2022-05-18T04:40:52.7736321Z 
2022-05-18T04:40:52.7737055Z ----------------------------------------------------------------------
2022-05-18T04:40:52.7737620Z Ran 1 test in 2.992s
2022-05-18T04:40:52.7737791Z 
2022-05-18T04:40:52.7737903Z OK (skipped=1)
2022-05-18T04:40:52.7738065Z 
2022-05-18T04:40:52.7738196Z Generating XML reports...
2022-05-18T04:40:52.7778027Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044049.xml
2022-05-18T04:40:54.1657130Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:54.1670515Z 
2022-05-18T04:40:54.1670779Z Running tests...
2022-05-18T04:40:54.1671238Z ----------------------------------------------------------------------
2022-05-18T04:40:55.7066502Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:55.7454379Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44559
2022-05-18T04:40:55.7561482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44560
2022-05-18T04:40:56.9109855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:40:56.9110432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:40:56.9111229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:56.9111919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:40:56.9219304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:57.0123849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:57.1612193Z skip: Skipped due to small world size. (2.994s)
2022-05-18T04:40:57.1612481Z 
2022-05-18T04:40:57.1613118Z ----------------------------------------------------------------------
2022-05-18T04:40:57.1613474Z Ran 1 test in 2.994s
2022-05-18T04:40:57.1613660Z 
2022-05-18T04:40:57.1613773Z OK (skipped=1)
2022-05-18T04:40:57.1613942Z 
2022-05-18T04:40:57.1614054Z Generating XML reports...
2022-05-18T04:40:57.1654114Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044054.xml
2022-05-18T04:40:58.5685263Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:40:58.5699742Z 
2022-05-18T04:40:58.5699967Z Running tests...
2022-05-18T04:40:58.5700413Z ----------------------------------------------------------------------
2022-05-18T04:41:00.1465157Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:00.1867612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44668
2022-05-18T04:41:00.1974056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44669
2022-05-18T04:41:01.3652839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:01.3653410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:01.3654239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:01.3654937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:01.3660435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:01.3660915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:01.6024747Z skip: Skipped due to small world size. (3.032s)
2022-05-18T04:41:01.6025015Z 
2022-05-18T04:41:01.6025420Z ----------------------------------------------------------------------
2022-05-18T04:41:01.6025747Z Ran 1 test in 3.032s
2022-05-18T04:41:01.6025927Z 
2022-05-18T04:41:01.6026045Z OK (skipped=1)
2022-05-18T04:41:01.6026201Z 
2022-05-18T04:41:01.6026329Z Generating XML reports...
2022-05-18T04:41:01.6066173Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044058.xml
2022-05-18T04:41:02.9881469Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:41:02.9895205Z 
2022-05-18T04:41:02.9895505Z Running tests...
2022-05-18T04:41:02.9895961Z ----------------------------------------------------------------------
2022-05-18T04:41:04.5300752Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:04.5689492Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44777
2022-05-18T04:41:04.5797276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44778
2022-05-18T04:41:05.7280226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:05.7280814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:05.7281604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:05.7282313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:05.7288384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:05.7289081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:05.8848420Z skip: Skipped due to small world size. (2.895s)
2022-05-18T04:41:05.8848736Z 
2022-05-18T04:41:05.8849119Z ----------------------------------------------------------------------
2022-05-18T04:41:05.8849458Z Ran 1 test in 2.895s
2022-05-18T04:41:05.8849624Z 
2022-05-18T04:41:05.8849761Z OK (skipped=1)
2022-05-18T04:41:05.8849917Z 
2022-05-18T04:41:05.8850027Z Generating XML reports...
2022-05-18T04:41:05.8891473Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044102.xml
2022-05-18T04:41:07.3049859Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:41:07.3064497Z 
2022-05-18T04:41:07.3064947Z Running tests...
2022-05-18T04:41:07.3065461Z ----------------------------------------------------------------------
2022-05-18T04:41:08.8909155Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:08.9308527Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44886
2022-05-18T04:41:08.9415935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44887
2022-05-18T04:41:10.0960123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:10.0960710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:10.0961518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:10.0962221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:10.0967551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:10.0968541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:10.3464199Z ok (3.040s)
2022-05-18T04:41:10.3464521Z 
2022-05-18T04:41:10.3464919Z ----------------------------------------------------------------------
2022-05-18T04:41:10.3465276Z Ran 1 test in 3.040s
2022-05-18T04:41:10.3465444Z 
2022-05-18T04:41:10.3465525Z OK
2022-05-18T04:41:10.3465663Z 
2022-05-18T04:41:10.3465817Z Generating XML reports...
2022-05-18T04:41:10.3506923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044107.xml
2022-05-18T04:41:11.7677917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:41:11.7693146Z 
2022-05-18T04:41:11.7693446Z Running tests...
2022-05-18T04:41:11.7693876Z ----------------------------------------------------------------------
2022-05-18T04:41:13.3476376Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:13.3874188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44999
2022-05-18T04:41:13.3981868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45000
2022-05-18T04:41:14.5626747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:14.5627324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:14.5628128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:14.5628809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:14.5635064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:14.5635966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:14.8033166Z ok (3.034s)
2022-05-18T04:41:14.8033389Z 
2022-05-18T04:41:14.8033756Z ----------------------------------------------------------------------
2022-05-18T04:41:14.8034103Z Ran 1 test in 3.034s
2022-05-18T04:41:14.8034268Z 
2022-05-18T04:41:14.8034366Z OK
2022-05-18T04:41:14.8034502Z 
2022-05-18T04:41:14.8034637Z Generating XML reports...
2022-05-18T04:41:14.8076220Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044111.xml
2022-05-18T04:41:16.1771057Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:41:16.1787523Z 
2022-05-18T04:41:16.1787764Z Running tests...
2022-05-18T04:41:16.1788434Z ----------------------------------------------------------------------
2022-05-18T04:41:17.7497098Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:17.7896732Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45108
2022-05-18T04:41:17.8002260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45109
2022-05-18T04:41:18.9775082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:18.9775849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:18.9776688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:18.9777392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:18.9882634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:19.0785589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:19.3056887Z ok (3.126s)
2022-05-18T04:41:19.3057127Z 
2022-05-18T04:41:19.3057546Z ----------------------------------------------------------------------
2022-05-18T04:41:19.3057890Z Ran 1 test in 3.127s
2022-05-18T04:41:19.3058057Z 
2022-05-18T04:41:19.3058159Z OK
2022-05-18T04:41:19.3058276Z 
2022-05-18T04:41:19.3058412Z Generating XML reports...
2022-05-18T04:41:19.3099673Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044116.xml
2022-05-18T04:41:20.7091355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:41:20.7105704Z 
2022-05-18T04:41:20.7106219Z Running tests...
2022-05-18T04:41:20.7106712Z ----------------------------------------------------------------------
2022-05-18T04:41:22.3051292Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:22.3450061Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45221
2022-05-18T04:41:22.3557423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45222
2022-05-18T04:41:23.5283992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:23.5284535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:23.5285366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:23.5286072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:23.5393993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:23.6295699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:23.8610622Z ok (3.150s)
2022-05-18T04:41:23.8610858Z 
2022-05-18T04:41:23.8611254Z ----------------------------------------------------------------------
2022-05-18T04:41:23.8611619Z Ran 1 test in 3.150s
2022-05-18T04:41:23.8611787Z 
2022-05-18T04:41:23.8611892Z OK
2022-05-18T04:41:23.8612012Z 
2022-05-18T04:41:23.8612150Z Generating XML reports...
2022-05-18T04:41:23.8653240Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044120.xml
2022-05-18T04:41:25.2490668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:41:25.2504043Z 
2022-05-18T04:41:25.2504577Z Running tests...
2022-05-18T04:41:25.2505102Z ----------------------------------------------------------------------
2022-05-18T04:41:26.7855384Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:26.8242524Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45334
2022-05-18T04:41:26.8349591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45335
2022-05-18T04:41:28.0182678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:28.0183219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:28.0184243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:28.0185023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:28.0291033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:28.1197037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:28.3399877Z ok (3.089s)
2022-05-18T04:41:28.3400102Z 
2022-05-18T04:41:28.3400488Z ----------------------------------------------------------------------
2022-05-18T04:41:28.3400831Z Ran 1 test in 3.090s
2022-05-18T04:41:28.3401030Z 
2022-05-18T04:41:28.3401127Z OK
2022-05-18T04:41:28.3401245Z 
2022-05-18T04:41:28.3401383Z Generating XML reports...
2022-05-18T04:41:28.3441396Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044125.xml
2022-05-18T04:41:29.7500174Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:41:29.7514978Z 
2022-05-18T04:41:29.7515388Z Running tests...
2022-05-18T04:41:29.7515884Z ----------------------------------------------------------------------
2022-05-18T04:41:31.3174723Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:31.3564962Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45447
2022-05-18T04:41:31.3670810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45448
2022-05-18T04:41:32.5534556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:32.5535361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:32.5536173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:32.5536858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:32.5643012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:32.6545595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:32.8723503Z ok (3.120s)
2022-05-18T04:41:32.8723735Z 
2022-05-18T04:41:32.8724108Z ----------------------------------------------------------------------
2022-05-18T04:41:32.8724454Z Ran 1 test in 3.121s
2022-05-18T04:41:32.8724625Z 
2022-05-18T04:41:32.8724745Z OK
2022-05-18T04:41:32.8724885Z 
2022-05-18T04:41:32.8725018Z Generating XML reports...
2022-05-18T04:41:32.8765594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044129.xml
2022-05-18T04:41:34.2806669Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:41:34.2820675Z 
2022-05-18T04:41:34.2821094Z Running tests...
2022-05-18T04:41:34.2821592Z ----------------------------------------------------------------------
2022-05-18T04:41:35.8469610Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:35.8869029Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45556
2022-05-18T04:41:35.8974500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45557
2022-05-18T04:41:37.1037627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:37.1038480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:37.1039283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:37.1040189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:37.1147109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:37.2047815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:37.2162060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:41:37.2162587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:41:37.2163267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:41:37.2163984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:41:37.4027333Z ok (3.120s)
2022-05-18T04:41:37.4027697Z 
2022-05-18T04:41:37.4028437Z ----------------------------------------------------------------------
2022-05-18T04:41:37.4029122Z Ran 1 test in 3.121s
2022-05-18T04:41:37.4029361Z 
2022-05-18T04:41:37.4029461Z OK
2022-05-18T04:41:37.4029597Z 
2022-05-18T04:41:37.4029741Z Generating XML reports...
2022-05-18T04:41:37.4070966Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044134.xml
2022-05-18T04:41:38.8192794Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:41:38.8209921Z 
2022-05-18T04:41:38.8210600Z Running tests...
2022-05-18T04:41:38.8211440Z ----------------------------------------------------------------------
2022-05-18T04:41:40.3918622Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:40.4317321Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45675
2022-05-18T04:41:40.4425885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45676
2022-05-18T04:41:41.5873211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:41.5873769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:41.5874578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:41.5875279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:41.5880856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:41.5881375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:41.6090454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:41:41.6090980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:41:41.6091656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:41:41.6092350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:41:41.8476001Z ok (3.026s)
2022-05-18T04:41:41.8476250Z 
2022-05-18T04:41:41.8476661Z ----------------------------------------------------------------------
2022-05-18T04:41:41.8476987Z Ran 1 test in 3.027s
2022-05-18T04:41:41.8477156Z 
2022-05-18T04:41:41.8477251Z OK
2022-05-18T04:41:41.8477413Z 
2022-05-18T04:41:41.8477552Z Generating XML reports...
2022-05-18T04:41:41.8520079Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044138.xml
2022-05-18T04:41:43.2563678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:41:43.2578131Z 
2022-05-18T04:41:43.2578313Z Running tests...
2022-05-18T04:41:43.2578794Z ----------------------------------------------------------------------
2022-05-18T04:41:44.8290254Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:44.8686345Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45794
2022-05-18T04:41:44.8791139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45795
2022-05-18T04:41:46.0759344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:46.0759973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:46.0760783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:46.0761487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:46.0866818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:46.1770765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:46.1882339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:41:46.1883145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:41:46.1883836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:41:46.1884790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:41:46.3843980Z ok (3.126s)
2022-05-18T04:41:46.3844211Z 
2022-05-18T04:41:46.3844840Z ----------------------------------------------------------------------
2022-05-18T04:41:46.3845192Z Ran 1 test in 3.127s
2022-05-18T04:41:46.3845343Z 
2022-05-18T04:41:46.3845441Z OK
2022-05-18T04:41:46.3845576Z 
2022-05-18T04:41:46.3845717Z Generating XML reports...
2022-05-18T04:41:46.3886405Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044143.xml
2022-05-18T04:41:47.7911388Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:41:47.7925850Z 
2022-05-18T04:41:47.7926230Z Running tests...
2022-05-18T04:41:47.7926706Z ----------------------------------------------------------------------
2022-05-18T04:41:49.3813239Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:49.4210226Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45913
2022-05-18T04:41:49.4315669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45914
2022-05-18T04:41:50.6085266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:50.6085823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:50.6086625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:50.6087297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:50.6195937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:50.7098655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:50.7211679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:41:50.7212428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:41:50.7213193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:41:50.7213875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:41:50.9371890Z ok (3.144s)
2022-05-18T04:41:50.9372142Z 
2022-05-18T04:41:50.9372527Z ----------------------------------------------------------------------
2022-05-18T04:41:50.9372879Z Ran 1 test in 3.145s
2022-05-18T04:41:50.9373028Z 
2022-05-18T04:41:50.9373129Z OK
2022-05-18T04:41:50.9373266Z 
2022-05-18T04:41:50.9373426Z Generating XML reports...
2022-05-18T04:41:50.9413178Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044147.xml
2022-05-18T04:41:52.3443877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:41:52.3457925Z 
2022-05-18T04:41:52.3458252Z Running tests...
2022-05-18T04:41:52.3458694Z ----------------------------------------------------------------------
2022-05-18T04:41:53.9235536Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:53.9631059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46032
2022-05-18T04:41:53.9737501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46033
2022-05-18T04:41:55.1615304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:55.1615880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:55.1616940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:55.1617653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:55.1623218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:55.1624086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:55.3788847Z skip: Skipped due to small world size. (3.033s)
2022-05-18T04:41:55.3789150Z 
2022-05-18T04:41:55.3789747Z ----------------------------------------------------------------------
2022-05-18T04:41:55.3790112Z Ran 1 test in 3.033s
2022-05-18T04:41:55.3790281Z 
2022-05-18T04:41:55.3790394Z OK (skipped=1)
2022-05-18T04:41:55.3790562Z 
2022-05-18T04:41:55.3790694Z Generating XML reports...
2022-05-18T04:41:55.3831779Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044152.xml
2022-05-18T04:41:56.7769269Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:41:56.7783858Z 
2022-05-18T04:41:56.7784183Z Running tests...
2022-05-18T04:41:56.7784636Z ----------------------------------------------------------------------
2022-05-18T04:41:58.3400167Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:58.3795612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46141
2022-05-18T04:41:58.3902101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46142
2022-05-18T04:41:59.6071869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:41:59.6072788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:41:59.6073584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:59.6074483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:41:59.6079722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:59.6080804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:59.7954114Z skip: Skipped due to small world size. (3.017s)
2022-05-18T04:41:59.7954419Z 
2022-05-18T04:41:59.7954826Z ----------------------------------------------------------------------
2022-05-18T04:41:59.7955169Z Ran 1 test in 3.017s
2022-05-18T04:41:59.7955338Z 
2022-05-18T04:41:59.7955452Z OK (skipped=1)
2022-05-18T04:41:59.7955592Z 
2022-05-18T04:41:59.7955723Z Generating XML reports...
2022-05-18T04:41:59.7997617Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044156.xml
2022-05-18T04:42:01.1850198Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:42:01.1863961Z 
2022-05-18T04:42:01.1864431Z Running tests...
2022-05-18T04:42:01.1864964Z ----------------------------------------------------------------------
2022-05-18T04:42:02.7424720Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:02.7813580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46250
2022-05-18T04:42:02.7918813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46251
2022-05-18T04:42:03.9950348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:03.9950949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:03.9952009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:03.9952711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:03.9957988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:03.9959146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:04.1970455Z skip: Skipped due to small world size. (3.010s)
2022-05-18T04:42:04.1970714Z 
2022-05-18T04:42:04.1971127Z ----------------------------------------------------------------------
2022-05-18T04:42:04.1971471Z Ran 1 test in 3.011s
2022-05-18T04:42:04.1971638Z 
2022-05-18T04:42:04.1971733Z OK (skipped=1)
2022-05-18T04:42:04.1971888Z 
2022-05-18T04:42:04.1972018Z Generating XML reports...
2022-05-18T04:42:04.2012238Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044201.xml
2022-05-18T04:42:05.6121342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:42:05.6135637Z 
2022-05-18T04:42:05.6136102Z Running tests...
2022-05-18T04:42:05.6136614Z ----------------------------------------------------------------------
2022-05-18T04:42:07.1811068Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:07.2198613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46359
2022-05-18T04:42:07.2303552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46360
2022-05-18T04:42:08.4115261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:08.4115824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:08.4116665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:08.4117368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:08.4123537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:08.4124532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:08.6354525Z skip: Skipped due to small world size. (3.021s)
2022-05-18T04:42:08.6354801Z 
2022-05-18T04:42:08.6355191Z ----------------------------------------------------------------------
2022-05-18T04:42:08.6355513Z Ran 1 test in 3.022s
2022-05-18T04:42:08.6355680Z 
2022-05-18T04:42:08.6355793Z OK (skipped=1)
2022-05-18T04:42:08.6355949Z 
2022-05-18T04:42:08.6356076Z Generating XML reports...
2022-05-18T04:42:08.6396696Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044205.xml
2022-05-18T04:42:10.0394252Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:42:10.0408739Z 
2022-05-18T04:42:10.0409020Z Running tests...
2022-05-18T04:42:10.0409484Z ----------------------------------------------------------------------
2022-05-18T04:42:11.6007260Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:11.6394882Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46468
2022-05-18T04:42:11.6501665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46469
2022-05-18T04:42:12.8032411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:12.8032981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:12.8034044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:12.8034732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:12.8140390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:12.9042767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:13.1552219Z ok (3.114s)
2022-05-18T04:42:13.1552419Z 
2022-05-18T04:42:13.1552810Z ----------------------------------------------------------------------
2022-05-18T04:42:13.1553155Z Ran 1 test in 3.114s
2022-05-18T04:42:13.1553330Z 
2022-05-18T04:42:13.1553429Z OK
2022-05-18T04:42:13.1553570Z 
2022-05-18T04:42:13.1553688Z Generating XML reports...
2022-05-18T04:42:13.1593913Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044210.xml
2022-05-18T04:42:14.5631336Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:42:14.5645386Z 
2022-05-18T04:42:14.5645817Z Running tests...
2022-05-18T04:42:14.5646313Z ----------------------------------------------------------------------
2022-05-18T04:42:16.0958566Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:16.1347728Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46581
2022-05-18T04:42:16.1453456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46582
2022-05-18T04:42:17.3035685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:17.3036490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:17.3037294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:17.3038027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:17.3044036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:17.3045364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:17.5505474Z ok (2.986s)
2022-05-18T04:42:17.5505685Z 
2022-05-18T04:42:17.5506398Z ----------------------------------------------------------------------
2022-05-18T04:42:17.5506800Z Ran 1 test in 2.986s
2022-05-18T04:42:17.5506982Z 
2022-05-18T04:42:17.5507074Z OK
2022-05-18T04:42:17.5507219Z 
2022-05-18T04:42:17.5507363Z Generating XML reports...
2022-05-18T04:42:17.5550432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044214.xml
2022-05-18T04:42:18.9418723Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:42:18.9432486Z 
2022-05-18T04:42:18.9432914Z Running tests...
2022-05-18T04:42:18.9433410Z ----------------------------------------------------------------------
2022-05-18T04:42:20.4812444Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:20.5206629Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46694
2022-05-18T04:42:20.5314630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46695
2022-05-18T04:42:21.7388684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:21.7389500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:21.7390309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:21.7391282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:21.7397684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:21.7398479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:23.6396529Z ok (4.696s)
2022-05-18T04:42:23.6396764Z 
2022-05-18T04:42:23.6397412Z ----------------------------------------------------------------------
2022-05-18T04:42:23.6397828Z Ran 1 test in 4.696s
2022-05-18T04:42:23.6397997Z 
2022-05-18T04:42:23.6398343Z OK
2022-05-18T04:42:23.6398493Z 
2022-05-18T04:42:23.6398636Z Generating XML reports...
2022-05-18T04:42:23.6438878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044218.xml
2022-05-18T04:42:25.0682833Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:42:25.0698881Z 
2022-05-18T04:42:25.0699384Z Running tests...
2022-05-18T04:42:25.0700262Z ----------------------------------------------------------------------
2022-05-18T04:42:26.6374812Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:26.6771967Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46809
2022-05-18T04:42:26.6879090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46810
2022-05-18T04:42:27.8495389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:27.8495949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:27.8496767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:27.8497467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:27.8603104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:27.9509076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:29.7961380Z ok (4.726s)
2022-05-18T04:42:29.7961900Z 
2022-05-18T04:42:29.7962320Z ----------------------------------------------------------------------
2022-05-18T04:42:29.7962652Z Ran 1 test in 4.726s
2022-05-18T04:42:29.7962824Z 
2022-05-18T04:42:29.7962921Z OK
2022-05-18T04:42:29.7963059Z 
2022-05-18T04:42:29.7963202Z Generating XML reports...
2022-05-18T04:42:29.8003527Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044225.xml
2022-05-18T04:42:31.2216805Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:42:31.2230795Z 
2022-05-18T04:42:31.2230924Z Running tests...
2022-05-18T04:42:31.2231645Z ----------------------------------------------------------------------
2022-05-18T04:42:32.7978543Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:32.8378750Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46924
2022-05-18T04:42:32.8484660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46925
2022-05-18T04:42:34.0378874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:34.0379429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:34.0380232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:34.0380921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:34.0386629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:34.0387590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:34.3537215Z ok (3.130s)
2022-05-18T04:42:34.3537427Z 
2022-05-18T04:42:34.3537835Z ----------------------------------------------------------------------
2022-05-18T04:42:34.3538187Z Ran 1 test in 3.131s
2022-05-18T04:42:34.3538354Z 
2022-05-18T04:42:34.3538451Z OK
2022-05-18T04:42:34.3538586Z 
2022-05-18T04:42:34.3538704Z Generating XML reports...
2022-05-18T04:42:34.3579081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044231.xml
2022-05-18T04:42:35.7602135Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:42:35.7616976Z 
2022-05-18T04:42:35.7617459Z Running tests...
2022-05-18T04:42:35.7617958Z ----------------------------------------------------------------------
2022-05-18T04:42:37.3477037Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:37.3874799Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47037
2022-05-18T04:42:37.3981451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47038
2022-05-18T04:42:38.5959644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:38.5960196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:38.5960995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:38.5961676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:38.5968295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:38.5968808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:40.3060325Z ok (4.544s)
2022-05-18T04:42:40.3060542Z 
2022-05-18T04:42:40.3060947Z ----------------------------------------------------------------------
2022-05-18T04:42:40.3061549Z Ran 1 test in 4.544s
2022-05-18T04:42:40.3061734Z 
2022-05-18T04:42:40.3061830Z OK
2022-05-18T04:42:40.3061973Z 
2022-05-18T04:42:40.3062093Z Generating XML reports...
2022-05-18T04:42:40.3102645Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044235.xml
2022-05-18T04:42:41.7275067Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:42:41.7289516Z 
2022-05-18T04:42:41.7289861Z Running tests...
2022-05-18T04:42:41.7290318Z ----------------------------------------------------------------------
2022-05-18T04:42:43.3173263Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:43.3569094Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47148
2022-05-18T04:42:43.3674284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47149
2022-05-18T04:42:44.5377590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:44.5378149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:44.5378935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:44.5379642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:44.5487943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:44.6388704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:44.8725694Z ok (3.143s)
2022-05-18T04:42:44.8725918Z 
2022-05-18T04:42:44.8726310Z ----------------------------------------------------------------------
2022-05-18T04:42:44.8726657Z Ran 1 test in 3.144s
2022-05-18T04:42:44.8726831Z 
2022-05-18T04:42:44.8726926Z OK
2022-05-18T04:42:44.8727066Z 
2022-05-18T04:42:44.8727209Z Generating XML reports...
2022-05-18T04:42:44.8768085Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044241.xml
2022-05-18T04:42:46.2781264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:42:46.2795727Z 
2022-05-18T04:42:46.2796208Z Running tests...
2022-05-18T04:42:46.2796663Z ----------------------------------------------------------------------
2022-05-18T04:42:47.8394991Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:47.8794645Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47261
2022-05-18T04:42:47.8900916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47262
2022-05-18T04:42:49.0521373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:49.0521916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:49.0522714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:49.0523418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:49.0632223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:49.1533001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:49.3954147Z ok (3.115s)
2022-05-18T04:42:49.3954365Z 
2022-05-18T04:42:49.3954762Z ----------------------------------------------------------------------
2022-05-18T04:42:49.3955109Z Ran 1 test in 3.116s
2022-05-18T04:42:49.3955274Z 
2022-05-18T04:42:49.3955372Z OK
2022-05-18T04:42:49.3955508Z 
2022-05-18T04:42:49.3955905Z Generating XML reports...
2022-05-18T04:42:49.3996692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044246.xml
2022-05-18T04:42:50.8188505Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:42:50.8203407Z 
2022-05-18T04:42:50.8203683Z Running tests...
2022-05-18T04:42:50.8204115Z ----------------------------------------------------------------------
2022-05-18T04:42:52.4087881Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:52.4483738Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47374
2022-05-18T04:42:52.4590066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47375
2022-05-18T04:42:53.6355344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:53.6355982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:53.6356795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:53.6357491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:53.6463073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:53.7366077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:53.9643848Z ok (3.144s)
2022-05-18T04:42:53.9644426Z 
2022-05-18T04:42:53.9644825Z ----------------------------------------------------------------------
2022-05-18T04:42:53.9645177Z Ran 1 test in 3.144s
2022-05-18T04:42:53.9645327Z 
2022-05-18T04:42:53.9645422Z OK
2022-05-18T04:42:53.9645557Z 
2022-05-18T04:42:53.9645690Z Generating XML reports...
2022-05-18T04:42:53.9685978Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044250.xml
2022-05-18T04:42:55.3705350Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:42:55.3719591Z 
2022-05-18T04:42:55.3719945Z Running tests...
2022-05-18T04:42:55.3720399Z ----------------------------------------------------------------------
2022-05-18T04:42:56.9519365Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:56.9917086Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47487
2022-05-18T04:42:57.0024068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47488
2022-05-18T04:42:58.1962508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:42:58.1963061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:42:58.1963867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:58.1964572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:42:58.1971093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:58.1971582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:02.1141687Z ok (6.742s)
2022-05-18T04:43:02.1141932Z 
2022-05-18T04:43:02.1142340Z ----------------------------------------------------------------------
2022-05-18T04:43:02.1142728Z Ran 1 test in 6.742s
2022-05-18T04:43:02.1142893Z 
2022-05-18T04:43:02.1142989Z OK
2022-05-18T04:43:02.1143106Z 
2022-05-18T04:43:02.1143247Z Generating XML reports...
2022-05-18T04:43:02.1183801Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044255.xml
2022-05-18T04:43:03.5371576Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:03.5385081Z 
2022-05-18T04:43:03.5385375Z Running tests...
2022-05-18T04:43:03.5385821Z ----------------------------------------------------------------------
2022-05-18T04:43:05.0779504Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:05.1168366Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47604
2022-05-18T04:43:05.1276544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47605
2022-05-18T04:43:06.3052209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:43:06.3052773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:43:06.3053577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:06.3054282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:06.3059958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:06.3060456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:10.2391998Z ok (6.700s)
2022-05-18T04:43:10.2392225Z 
2022-05-18T04:43:10.2392619Z ----------------------------------------------------------------------
2022-05-18T04:43:10.2392962Z Ran 1 test in 6.701s
2022-05-18T04:43:10.2393385Z 
2022-05-18T04:43:10.2395950Z OK
2022-05-18T04:43:10.2396147Z 
2022-05-18T04:43:10.2396294Z Generating XML reports...
2022-05-18T04:43:10.2436421Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044303.xml
2022-05-18T04:43:11.6968063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:11.6982532Z 
2022-05-18T04:43:11.6982846Z Running tests...
2022-05-18T04:43:11.6983317Z ----------------------------------------------------------------------
2022-05-18T04:43:13.2875009Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:13.3276582Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47721
2022-05-18T04:43:13.3384250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47722
2022-05-18T04:43:14.5514452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:43:14.5515027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:43:14.5515822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:14.5516538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:14.5624700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:14.6528905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:18.5504101Z ok (6.852s)
2022-05-18T04:43:18.5504415Z 
2022-05-18T04:43:18.5504984Z ----------------------------------------------------------------------
2022-05-18T04:43:18.5505341Z Ran 1 test in 6.852s
2022-05-18T04:43:18.5505489Z 
2022-05-18T04:43:18.5506785Z OK
2022-05-18T04:43:18.5508164Z 
2022-05-18T04:43:18.5508446Z Generating XML reports...
2022-05-18T04:43:18.5546756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044311.xml
2022-05-18T04:43:19.9747558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:19.9761682Z 
2022-05-18T04:43:19.9762024Z Running tests...
2022-05-18T04:43:19.9762479Z ----------------------------------------------------------------------
2022-05-18T04:43:19.9780798Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:43:19.9781105Z 
2022-05-18T04:43:19.9781547Z ----------------------------------------------------------------------
2022-05-18T04:43:19.9782117Z Ran 1 test in 0.002s
2022-05-18T04:43:19.9782291Z 
2022-05-18T04:43:19.9782402Z OK (skipped=1)
2022-05-18T04:43:19.9782569Z 
2022-05-18T04:43:19.9782696Z Generating XML reports...
2022-05-18T04:43:19.9818030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044319.xml
2022-05-18T04:43:21.2292723Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:21.2307284Z 
2022-05-18T04:43:21.2307688Z Running tests...
2022-05-18T04:43:21.2308164Z ----------------------------------------------------------------------
2022-05-18T04:43:21.2327000Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:43:21.2327804Z 
2022-05-18T04:43:21.2328118Z ----------------------------------------------------------------------
2022-05-18T04:43:21.2328470Z Ran 1 test in 0.002s
2022-05-18T04:43:21.2328620Z 
2022-05-18T04:43:21.2328731Z OK (skipped=1)
2022-05-18T04:43:21.2328887Z 
2022-05-18T04:43:21.2329014Z Generating XML reports...
2022-05-18T04:43:21.2364080Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044321.xml
2022-05-18T04:43:22.4967161Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:22.4981014Z 
2022-05-18T04:43:22.4981475Z Running tests...
2022-05-18T04:43:22.4981955Z ----------------------------------------------------------------------
2022-05-18T04:43:22.5002039Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T04:43:22.5002350Z 
2022-05-18T04:43:22.5002639Z ----------------------------------------------------------------------
2022-05-18T04:43:22.5002955Z Ran 1 test in 0.002s
2022-05-18T04:43:22.5003122Z 
2022-05-18T04:43:22.5003240Z OK (skipped=1)
2022-05-18T04:43:22.5003398Z 
2022-05-18T04:43:22.5003525Z Generating XML reports...
2022-05-18T04:43:22.5038553Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044322.xml
2022-05-18T04:43:23.7381723Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:23.7396046Z 
2022-05-18T04:43:23.7396437Z Running tests...
2022-05-18T04:43:23.7396966Z ----------------------------------------------------------------------
2022-05-18T04:43:23.7418397Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T04:43:23.7418728Z 
2022-05-18T04:43:23.7419022Z ----------------------------------------------------------------------
2022-05-18T04:43:23.7419340Z Ran 1 test in 0.002s
2022-05-18T04:43:23.7419505Z 
2022-05-18T04:43:23.7419615Z OK (skipped=1)
2022-05-18T04:43:23.7419772Z 
2022-05-18T04:43:23.7419902Z Generating XML reports...
2022-05-18T04:43:23.7453868Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044323.xml
2022-05-18T04:43:25.0084926Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:25.0099352Z 
2022-05-18T04:43:25.0099634Z Running tests...
2022-05-18T04:43:25.0100088Z ----------------------------------------------------------------------
2022-05-18T04:43:25.0118739Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:43:25.0119355Z 
2022-05-18T04:43:25.0119936Z ----------------------------------------------------------------------
2022-05-18T04:43:25.0120300Z Ran 1 test in 0.002s
2022-05-18T04:43:25.0120466Z 
2022-05-18T04:43:25.0120576Z OK (skipped=1)
2022-05-18T04:43:25.0120733Z 
2022-05-18T04:43:25.0120859Z Generating XML reports...
2022-05-18T04:43:25.0155612Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044325.xml
2022-05-18T04:43:26.2810633Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:26.2824821Z 
2022-05-18T04:43:26.2825250Z Running tests...
2022-05-18T04:43:26.2825767Z ----------------------------------------------------------------------
2022-05-18T04:43:26.2845807Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T04:43:26.2846145Z 
2022-05-18T04:43:26.2846448Z ----------------------------------------------------------------------
2022-05-18T04:43:26.2846784Z Ran 1 test in 0.002s
2022-05-18T04:43:26.2846951Z 
2022-05-18T04:43:26.2847070Z OK (skipped=1)
2022-05-18T04:43:26.2847226Z 
2022-05-18T04:43:26.2847335Z Generating XML reports...
2022-05-18T04:43:26.2882982Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044326.xml
2022-05-18T04:43:27.5504289Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:27.5518254Z 
2022-05-18T04:43:27.5518527Z Running tests...
2022-05-18T04:43:27.5519232Z ----------------------------------------------------------------------
2022-05-18T04:43:27.5538148Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T04:43:27.5538457Z 
2022-05-18T04:43:27.5538747Z ----------------------------------------------------------------------
2022-05-18T04:43:27.5539077Z Ran 1 test in 0.002s
2022-05-18T04:43:27.5539244Z 
2022-05-18T04:43:27.5539355Z OK (skipped=1)
2022-05-18T04:43:27.5539514Z 
2022-05-18T04:43:27.5539639Z Generating XML reports...
2022-05-18T04:43:27.5575675Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044327.xml
2022-05-18T04:43:28.8087077Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:28.8101005Z 
2022-05-18T04:43:28.8101398Z Running tests...
2022-05-18T04:43:28.8101890Z ----------------------------------------------------------------------
2022-05-18T04:43:28.8121506Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:43:28.8121872Z 
2022-05-18T04:43:28.8122161Z ----------------------------------------------------------------------
2022-05-18T04:43:28.8122494Z Ran 1 test in 0.002s
2022-05-18T04:43:28.8122659Z 
2022-05-18T04:43:28.8122768Z OK (skipped=1)
2022-05-18T04:43:28.8122923Z 
2022-05-18T04:43:28.8123043Z Generating XML reports...
2022-05-18T04:43:28.8158390Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044328.xml
2022-05-18T04:43:30.0757973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:30.0773265Z 
2022-05-18T04:43:30.0773808Z Running tests...
2022-05-18T04:43:30.0774321Z ----------------------------------------------------------------------
2022-05-18T04:43:30.0792968Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:43:30.0793349Z 
2022-05-18T04:43:30.0793661Z ----------------------------------------------------------------------
2022-05-18T04:43:30.0794004Z Ran 1 test in 0.002s
2022-05-18T04:43:30.0794170Z 
2022-05-18T04:43:30.0794282Z OK (skipped=1)
2022-05-18T04:43:30.0794439Z 
2022-05-18T04:43:30.0794565Z Generating XML reports...
2022-05-18T04:43:30.0830941Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044330.xml
2022-05-18T04:43:31.3419971Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:31.3434176Z 
2022-05-18T04:43:31.3434449Z Running tests...
2022-05-18T04:43:31.3434901Z ----------------------------------------------------------------------
2022-05-18T04:43:31.3454302Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:43:31.3454688Z 
2022-05-18T04:43:31.3455011Z ----------------------------------------------------------------------
2022-05-18T04:43:31.3455368Z Ran 1 test in 0.002s
2022-05-18T04:43:31.3455533Z 
2022-05-18T04:43:31.3455644Z OK (skipped=1)
2022-05-18T04:43:31.3455785Z 
2022-05-18T04:43:31.3455919Z Generating XML reports...
2022-05-18T04:43:31.3491247Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044331.xml
2022-05-18T04:43:32.6069682Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:32.6084279Z 
2022-05-18T04:43:32.6084813Z Running tests...
2022-05-18T04:43:32.6085354Z ----------------------------------------------------------------------
2022-05-18T04:43:32.6105060Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:43:32.6105560Z 
2022-05-18T04:43:32.6106066Z ----------------------------------------------------------------------
2022-05-18T04:43:32.6119610Z Ran 1 test in 0.002s
2022-05-18T04:43:32.6119791Z 
2022-05-18T04:43:32.6119910Z OK (skipped=1)
2022-05-18T04:43:32.6120068Z 
2022-05-18T04:43:32.6120201Z Generating XML reports...
2022-05-18T04:43:32.6142374Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044332.xml
2022-05-18T04:43:33.8777922Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:33.8792530Z 
2022-05-18T04:43:33.8792938Z Running tests...
2022-05-18T04:43:33.8793431Z ----------------------------------------------------------------------
2022-05-18T04:43:33.8814481Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:43:33.8814861Z 
2022-05-18T04:43:33.8815191Z ----------------------------------------------------------------------
2022-05-18T04:43:33.8815531Z Ran 1 test in 0.002s
2022-05-18T04:43:33.8815715Z 
2022-05-18T04:43:33.8815834Z OK (skipped=1)
2022-05-18T04:43:33.8815990Z 
2022-05-18T04:43:33.8816099Z Generating XML reports...
2022-05-18T04:43:33.8851582Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044333.xml
2022-05-18T04:43:35.1388940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:35.1403557Z 
2022-05-18T04:43:35.1403839Z Running tests...
2022-05-18T04:43:35.1404289Z ----------------------------------------------------------------------
2022-05-18T04:43:35.1423023Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:43:35.1423377Z 
2022-05-18T04:43:35.1423666Z ----------------------------------------------------------------------
2022-05-18T04:43:35.1424001Z Ran 1 test in 0.002s
2022-05-18T04:43:35.1424166Z 
2022-05-18T04:43:35.1424278Z OK (skipped=1)
2022-05-18T04:43:35.1424440Z 
2022-05-18T04:43:35.1424566Z Generating XML reports...
2022-05-18T04:43:35.1459718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044335.xml
2022-05-18T04:43:36.4046331Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:36.4060933Z 
2022-05-18T04:43:36.4061111Z Running tests...
2022-05-18T04:43:36.4061548Z ----------------------------------------------------------------------
2022-05-18T04:43:36.4081873Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:43:36.4082234Z 
2022-05-18T04:43:36.4082517Z ----------------------------------------------------------------------
2022-05-18T04:43:36.4082853Z Ran 1 test in 0.002s
2022-05-18T04:43:36.4083018Z 
2022-05-18T04:43:36.4083112Z OK (skipped=1)
2022-05-18T04:43:36.4083267Z 
2022-05-18T04:43:36.4083407Z Generating XML reports...
2022-05-18T04:43:36.4118691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044336.xml
2022-05-18T04:43:37.6567125Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:37.6581054Z 
2022-05-18T04:43:37.6581500Z Running tests...
2022-05-18T04:43:37.6581984Z ----------------------------------------------------------------------
2022-05-18T04:43:37.6601547Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:43:37.6602227Z 
2022-05-18T04:43:37.6602838Z ----------------------------------------------------------------------
2022-05-18T04:43:37.6603309Z Ran 1 test in 0.002s
2022-05-18T04:43:37.6603458Z 
2022-05-18T04:43:37.6603569Z OK (skipped=1)
2022-05-18T04:43:37.6603725Z 
2022-05-18T04:43:37.6603856Z Generating XML reports...
2022-05-18T04:43:37.6639034Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044337.xml
2022-05-18T04:43:38.9345057Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:38.9359205Z 
2022-05-18T04:43:38.9359576Z Running tests...
2022-05-18T04:43:38.9360122Z ----------------------------------------------------------------------
2022-05-18T04:43:38.9380986Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:43:38.9381566Z 
2022-05-18T04:43:38.9381968Z ----------------------------------------------------------------------
2022-05-18T04:43:38.9382290Z Ran 1 test in 0.002s
2022-05-18T04:43:38.9382453Z 
2022-05-18T04:43:38.9382565Z OK (skipped=1)
2022-05-18T04:43:38.9382722Z 
2022-05-18T04:43:38.9382848Z Generating XML reports...
2022-05-18T04:43:38.9418402Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044338.xml
2022-05-18T04:43:40.1998375Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:40.2012322Z 
2022-05-18T04:43:40.2012751Z Running tests...
2022-05-18T04:43:40.2013244Z ----------------------------------------------------------------------
2022-05-18T04:43:40.2031696Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:43:40.2032079Z 
2022-05-18T04:43:40.2032574Z ----------------------------------------------------------------------
2022-05-18T04:43:40.2032903Z Ran 1 test in 0.002s
2022-05-18T04:43:40.2033068Z 
2022-05-18T04:43:40.2033181Z OK (skipped=1)
2022-05-18T04:43:40.2033337Z 
2022-05-18T04:43:40.2033462Z Generating XML reports...
2022-05-18T04:43:40.2068651Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044340.xml
2022-05-18T04:43:41.4660679Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:41.4674868Z 
2022-05-18T04:43:41.4675283Z Running tests...
2022-05-18T04:43:41.4675768Z ----------------------------------------------------------------------
2022-05-18T04:43:41.4695293Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:43:41.4695929Z 
2022-05-18T04:43:41.4696215Z ----------------------------------------------------------------------
2022-05-18T04:43:41.4696547Z Ran 1 test in 0.002s
2022-05-18T04:43:41.4696714Z 
2022-05-18T04:43:41.4696825Z OK (skipped=1)
2022-05-18T04:43:41.4696983Z 
2022-05-18T04:43:41.4697108Z Generating XML reports...
2022-05-18T04:43:41.4732721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044341.xml
2022-05-18T04:43:42.7236727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:42.7251376Z 
2022-05-18T04:43:42.7251680Z Running tests...
2022-05-18T04:43:42.7252126Z ----------------------------------------------------------------------
2022-05-18T04:43:42.7271626Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:43:42.7271970Z 
2022-05-18T04:43:42.7272258Z ----------------------------------------------------------------------
2022-05-18T04:43:42.7272574Z Ran 1 test in 0.002s
2022-05-18T04:43:42.7272739Z 
2022-05-18T04:43:42.7272858Z OK (skipped=1)
2022-05-18T04:43:42.7273012Z 
2022-05-18T04:43:42.7273138Z Generating XML reports...
2022-05-18T04:43:42.7309070Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044342.xml
2022-05-18T04:43:43.9773011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:43.9786914Z 
2022-05-18T04:43:43.9787162Z Running tests...
2022-05-18T04:43:43.9787623Z ----------------------------------------------------------------------
2022-05-18T04:43:43.9808024Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:43:43.9808406Z 
2022-05-18T04:43:43.9808723Z ----------------------------------------------------------------------
2022-05-18T04:43:43.9809042Z Ran 1 test in 0.002s
2022-05-18T04:43:43.9809208Z 
2022-05-18T04:43:43.9809316Z OK (skipped=1)
2022-05-18T04:43:43.9809470Z 
2022-05-18T04:43:43.9809596Z Generating XML reports...
2022-05-18T04:43:43.9844100Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044343.xml
2022-05-18T04:43:45.2251439Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:45.2265199Z 
2022-05-18T04:43:45.2265353Z Running tests...
2022-05-18T04:43:45.2266245Z ----------------------------------------------------------------------
2022-05-18T04:43:45.2284725Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:43:45.2285079Z 
2022-05-18T04:43:45.2285381Z ----------------------------------------------------------------------
2022-05-18T04:43:45.2285701Z Ran 1 test in 0.002s
2022-05-18T04:43:45.2285874Z 
2022-05-18T04:43:45.2285986Z OK (skipped=1)
2022-05-18T04:43:45.2286140Z 
2022-05-18T04:43:45.2286267Z Generating XML reports...
2022-05-18T04:43:45.2321030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044345.xml
2022-05-18T04:43:46.4783014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:46.4797576Z 
2022-05-18T04:43:46.4797888Z Running tests...
2022-05-18T04:43:46.4798634Z ----------------------------------------------------------------------
2022-05-18T04:43:46.4818585Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:43:46.4819164Z 
2022-05-18T04:43:46.4819462Z ----------------------------------------------------------------------
2022-05-18T04:43:46.4820062Z Ran 1 test in 0.002s
2022-05-18T04:43:46.4820256Z 
2022-05-18T04:43:46.4820368Z OK (skipped=1)
2022-05-18T04:43:46.4820526Z 
2022-05-18T04:43:46.4820653Z Generating XML reports...
2022-05-18T04:43:46.4855861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044346.xml
2022-05-18T04:43:47.7468011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:47.7482707Z 
2022-05-18T04:43:47.7483127Z Running tests...
2022-05-18T04:43:47.7483646Z ----------------------------------------------------------------------
2022-05-18T04:43:47.7502570Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:43:47.7503052Z 
2022-05-18T04:43:47.7503587Z ----------------------------------------------------------------------
2022-05-18T04:43:47.7503951Z Ran 1 test in 0.002s
2022-05-18T04:43:47.7504138Z 
2022-05-18T04:43:47.7504235Z OK (skipped=1)
2022-05-18T04:43:47.7504392Z 
2022-05-18T04:43:47.7504519Z Generating XML reports...
2022-05-18T04:43:47.7540318Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044347.xml
2022-05-18T04:43:49.0134993Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:49.0149370Z 
2022-05-18T04:43:49.0149864Z Running tests...
2022-05-18T04:43:49.0150339Z ----------------------------------------------------------------------
2022-05-18T04:43:49.0170883Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:43:49.0171446Z 
2022-05-18T04:43:49.0171749Z ----------------------------------------------------------------------
2022-05-18T04:43:49.0172070Z Ran 1 test in 0.002s
2022-05-18T04:43:49.0172233Z 
2022-05-18T04:43:49.0172356Z OK (skipped=1)
2022-05-18T04:43:49.0172514Z 
2022-05-18T04:43:49.0172640Z Generating XML reports...
2022-05-18T04:43:49.0208299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044349.xml
2022-05-18T04:43:50.2885659Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:50.2900040Z 
2022-05-18T04:43:50.2900480Z Running tests...
2022-05-18T04:43:50.2900977Z ----------------------------------------------------------------------
2022-05-18T04:43:51.9029732Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:51.9426116Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48630
2022-05-18T04:43:51.9532064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48631
2022-05-18T04:43:53.1044103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:43:53.1044764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:43:53.1045563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:53.1046251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:53.1052067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:53.1052723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:55.3556483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:43:55.3557087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:43:55.3558488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:43:55.3559528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:43:55.7626020Z ok (5.472s)
2022-05-18T04:43:55.7626246Z 
2022-05-18T04:43:55.7626635Z ----------------------------------------------------------------------
2022-05-18T04:43:55.7626978Z Ran 1 test in 5.473s
2022-05-18T04:43:55.7627154Z 
2022-05-18T04:43:55.7627233Z OK
2022-05-18T04:43:55.7627369Z 
2022-05-18T04:43:55.7627511Z Generating XML reports...
2022-05-18T04:43:55.7668204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044350.xml
2022-05-18T04:43:57.1674320Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:43:57.1688307Z 
2022-05-18T04:43:57.1688476Z Running tests...
2022-05-18T04:43:57.1689206Z ----------------------------------------------------------------------
2022-05-18T04:43:58.7100664Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:58.7489346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48753
2022-05-18T04:43:58.7597082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48754
2022-05-18T04:43:59.9107540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:43:59.9108105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:43:59.9108884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:59.9109816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:43:59.9114840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:59.9115728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:00.0644036Z skip: Need at least 3 CUDA devices (2.895s)
2022-05-18T04:44:00.0644262Z 
2022-05-18T04:44:00.0644624Z ----------------------------------------------------------------------
2022-05-18T04:44:00.0644970Z Ran 1 test in 2.896s
2022-05-18T04:44:00.0645134Z 
2022-05-18T04:44:00.0645253Z OK (skipped=1)
2022-05-18T04:44:00.0645410Z 
2022-05-18T04:44:00.0645540Z Generating XML reports...
2022-05-18T04:44:00.0687170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044357.xml
2022-05-18T04:44:01.4824179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:01.4839152Z 
2022-05-18T04:44:01.4839306Z Running tests...
2022-05-18T04:44:01.4839996Z ----------------------------------------------------------------------
2022-05-18T04:44:01.4860053Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-05-18T04:44:01.4860846Z 
2022-05-18T04:44:01.4861192Z ----------------------------------------------------------------------
2022-05-18T04:44:01.4861825Z Ran 1 test in 0.002s
2022-05-18T04:44:01.4861980Z 
2022-05-18T04:44:01.4862101Z OK (skipped=1)
2022-05-18T04:44:01.4862259Z 
2022-05-18T04:44:01.4862388Z Generating XML reports...
2022-05-18T04:44:01.4898477Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044401.xml
2022-05-18T04:44:02.7523788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:02.7537794Z 
2022-05-18T04:44:02.7538153Z Running tests...
2022-05-18T04:44:02.7538597Z ----------------------------------------------------------------------
2022-05-18T04:44:04.3323841Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:04.3710070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48895
2022-05-18T04:44:04.3814218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48896
2022-05-18T04:44:05.5737627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:05.5738195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:05.5739003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:05.5739705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:05.5845904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:05.6749501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:06.6881318Z ok (3.934s)
2022-05-18T04:44:06.6881560Z 
2022-05-18T04:44:06.6881971Z ----------------------------------------------------------------------
2022-05-18T04:44:06.6882523Z Ran 1 test in 3.934s
2022-05-18T04:44:06.6882803Z 
2022-05-18T04:44:06.6882883Z OK
2022-05-18T04:44:06.6883019Z 
2022-05-18T04:44:06.6883152Z Generating XML reports...
2022-05-18T04:44:06.6924699Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044402.xml
2022-05-18T04:44:08.1224684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:08.1238634Z 
2022-05-18T04:44:08.1239084Z Running tests...
2022-05-18T04:44:08.1239614Z ----------------------------------------------------------------------
2022-05-18T04:44:09.7094423Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:09.7492174Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49004
2022-05-18T04:44:09.7597988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49005
2022-05-18T04:44:10.9311835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:10.9312422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:10.9313213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:10.9313919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:10.9419517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:11.0326887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:13.3701395Z ok (5.246s)
2022-05-18T04:44:13.3701783Z 
2022-05-18T04:44:13.3702627Z ----------------------------------------------------------------------
2022-05-18T04:44:13.3703086Z Ran 1 test in 5.246s
2022-05-18T04:44:13.3703254Z 
2022-05-18T04:44:13.3703352Z OK
2022-05-18T04:44:13.3703495Z 
2022-05-18T04:44:13.3703631Z Generating XML reports...
2022-05-18T04:44:13.3744438Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044408.xml
2022-05-18T04:44:14.7740468Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:14.7754211Z 
2022-05-18T04:44:14.7754522Z Running tests...
2022-05-18T04:44:14.7754960Z ----------------------------------------------------------------------
2022-05-18T04:44:16.3377527Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:16.3764971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49115
2022-05-18T04:44:16.3871806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49116
2022-05-18T04:44:17.5988736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:17.5989817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:17.5991141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:17.5992561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:17.5997458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:17.5999234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:17.6204373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:44:17.6204921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:44:17.6205645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:44:17.6206348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:44:18.6934101Z ok (3.918s)
2022-05-18T04:44:18.6934330Z 
2022-05-18T04:44:18.6934953Z ----------------------------------------------------------------------
2022-05-18T04:44:18.6935394Z Ran 1 test in 3.918s
2022-05-18T04:44:18.6935551Z 
2022-05-18T04:44:18.6935918Z OK
2022-05-18T04:44:18.6936062Z 
2022-05-18T04:44:18.6936203Z Generating XML reports...
2022-05-18T04:44:18.6976495Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044414.xml
2022-05-18T04:44:20.1312164Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:20.1327411Z 
2022-05-18T04:44:20.1327775Z Running tests...
2022-05-18T04:44:20.1328234Z ----------------------------------------------------------------------
2022-05-18T04:44:21.7077519Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:21.7470634Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49230
2022-05-18T04:44:21.7576160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49231
2022-05-18T04:44:22.9248627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:22.9249725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:22.9251176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:22.9252497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:22.9256844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:22.9257821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:23.1625702Z skip: Skipped due to small world size. (3.029s)
2022-05-18T04:44:23.1625977Z 
2022-05-18T04:44:23.1626373Z ----------------------------------------------------------------------
2022-05-18T04:44:23.1626720Z Ran 1 test in 3.030s
2022-05-18T04:44:23.1626889Z 
2022-05-18T04:44:23.1627012Z OK (skipped=1)
2022-05-18T04:44:23.1627170Z 
2022-05-18T04:44:23.1627303Z Generating XML reports...
2022-05-18T04:44:23.1668525Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044420.xml
2022-05-18T04:44:24.5782198Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:24.5796747Z 
2022-05-18T04:44:24.5796918Z Running tests...
2022-05-18T04:44:24.5797352Z ----------------------------------------------------------------------
2022-05-18T04:44:26.1742591Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:26.2147861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49339
2022-05-18T04:44:26.2254841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49340
2022-05-18T04:44:27.4030329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:27.4030987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:27.4031793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:27.4032506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:27.4037780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:27.4038517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:27.6304878Z skip: Skipped due to small world size. (3.050s)
2022-05-18T04:44:27.6305118Z 
2022-05-18T04:44:27.6305687Z ----------------------------------------------------------------------
2022-05-18T04:44:27.6306070Z Ran 1 test in 3.051s
2022-05-18T04:44:27.6306245Z 
2022-05-18T04:44:27.6306362Z OK (skipped=1)
2022-05-18T04:44:27.6306519Z 
2022-05-18T04:44:27.6306870Z Generating XML reports...
2022-05-18T04:44:27.6346939Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044424.xml
2022-05-18T04:44:29.0473935Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:29.0489445Z 
2022-05-18T04:44:29.0489906Z Running tests...
2022-05-18T04:44:29.0490415Z ----------------------------------------------------------------------
2022-05-18T04:44:30.6131915Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:30.6531690Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49448
2022-05-18T04:44:30.6640713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49449
2022-05-18T04:44:31.8648994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:31.8649606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:31.8650435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:31.8651150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:31.8756873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:31.9662303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:32.1693001Z skip: Skipped due to small world size. (3.120s)
2022-05-18T04:44:32.1693268Z 
2022-05-18T04:44:32.1693652Z ----------------------------------------------------------------------
2022-05-18T04:44:32.1693978Z Ran 1 test in 3.120s
2022-05-18T04:44:32.1694145Z 
2022-05-18T04:44:32.1694267Z OK (skipped=1)
2022-05-18T04:44:32.1694429Z 
2022-05-18T04:44:32.1694560Z Generating XML reports...
2022-05-18T04:44:32.1735167Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044429.xml
2022-05-18T04:44:33.5853449Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:33.5868080Z 
2022-05-18T04:44:33.5868912Z Running tests...
2022-05-18T04:44:33.5869469Z ----------------------------------------------------------------------
2022-05-18T04:44:35.1636289Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:35.2036382Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49557
2022-05-18T04:44:35.2144005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49558
2022-05-18T04:44:36.4580483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:36.4581057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:36.4581864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:36.4582562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:36.4588691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:36.4589186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:36.4695846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:44:36.4696339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:44:36.4697098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:44:36.4698050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:44:37.7211952Z ok (4.134s)
2022-05-18T04:44:37.7212318Z 
2022-05-18T04:44:37.7213055Z ----------------------------------------------------------------------
2022-05-18T04:44:37.7213743Z Ran 1 test in 4.134s
2022-05-18T04:44:37.7213932Z 
2022-05-18T04:44:37.7214028Z OK
2022-05-18T04:44:37.7214165Z 
2022-05-18T04:44:37.7216969Z Generating XML reports...
2022-05-18T04:44:37.7253626Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044433.xml
2022-05-18T04:44:39.1561262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:39.1575854Z 
2022-05-18T04:44:39.1576300Z Running tests...
2022-05-18T04:44:39.1576811Z ----------------------------------------------------------------------
2022-05-18T04:44:39.1600573Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s)
2022-05-18T04:44:39.1601037Z 
2022-05-18T04:44:39.1601412Z ----------------------------------------------------------------------
2022-05-18T04:44:39.1601751Z Ran 1 test in 0.002s
2022-05-18T04:44:39.1601917Z 
2022-05-18T04:44:39.1602032Z OK (skipped=1)
2022-05-18T04:44:39.1602192Z 
2022-05-18T04:44:39.1602319Z Generating XML reports...
2022-05-18T04:44:39.1637090Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044439.xml
2022-05-18T04:44:40.3965225Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:40.3978716Z 
2022-05-18T04:44:40.3979172Z Running tests...
2022-05-18T04:44:40.3979669Z ----------------------------------------------------------------------
2022-05-18T04:44:41.9488053Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:41.9877950Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49705
2022-05-18T04:44:41.9981865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49706
2022-05-18T04:44:43.2257673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:43.2258277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:43.2259078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:43.2259768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:43.2265310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:43.2266926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:43.4031924Z skip: Skipped due to small world size. (3.005s)
2022-05-18T04:44:43.4032399Z 
2022-05-18T04:44:43.4032809Z ----------------------------------------------------------------------
2022-05-18T04:44:43.4033155Z Ran 1 test in 3.005s
2022-05-18T04:44:43.4033337Z 
2022-05-18T04:44:43.4033449Z OK (skipped=1)
2022-05-18T04:44:43.4033587Z 
2022-05-18T04:44:43.4033719Z Generating XML reports...
2022-05-18T04:44:43.4073281Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044440.xml
2022-05-18T04:44:44.8107877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:44.8122693Z 
2022-05-18T04:44:44.8122961Z Running tests...
2022-05-18T04:44:44.8123404Z ----------------------------------------------------------------------
2022-05-18T04:44:46.4144440Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:46.4542852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49814
2022-05-18T04:44:46.4649272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49815
2022-05-18T04:44:47.6522049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:47.6522611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:47.6523681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:47.6524658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:47.6530626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:47.6531397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:47.8702352Z ok (3.058s)
2022-05-18T04:44:47.8702621Z 
2022-05-18T04:44:47.8703199Z ----------------------------------------------------------------------
2022-05-18T04:44:47.8703551Z Ran 1 test in 3.058s
2022-05-18T04:44:47.8703716Z 
2022-05-18T04:44:47.8703801Z OK
2022-05-18T04:44:47.8703984Z 
2022-05-18T04:44:47.8704234Z Generating XML reports...
2022-05-18T04:44:47.8744440Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044444.xml
2022-05-18T04:44:49.2837442Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:49.2852692Z 
2022-05-18T04:44:49.2852983Z Running tests...
2022-05-18T04:44:49.2853410Z ----------------------------------------------------------------------
2022-05-18T04:44:50.8571240Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:50.8961066Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49923
2022-05-18T04:44:50.9066058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49924
2022-05-18T04:44:52.0913400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:44:52.0914195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:44:52.0915050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:52.0915801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:44:52.1024094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:52.1924378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:52.4120519Z ok (3.126s)
2022-05-18T04:44:52.4120867Z 
2022-05-18T04:44:52.4121556Z ----------------------------------------------------------------------
2022-05-18T04:44:52.4122200Z Ran 1 test in 3.127s
2022-05-18T04:44:52.4122514Z 
2022-05-18T04:44:52.4122687Z OK
2022-05-18T04:44:52.4122946Z 
2022-05-18T04:44:52.4123187Z Generating XML reports...
2022-05-18T04:44:52.4164569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044449.xml
2022-05-18T04:44:53.8222430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:53.8236747Z 
2022-05-18T04:44:53.8237019Z Running tests...
2022-05-18T04:44:53.8237459Z ----------------------------------------------------------------------
2022-05-18T04:44:53.8263162Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:44:53.8263493Z 
2022-05-18T04:44:53.8264101Z ----------------------------------------------------------------------
2022-05-18T04:44:53.8264469Z Ran 1 test in 0.003s
2022-05-18T04:44:53.8264642Z 
2022-05-18T04:44:53.8264755Z OK (skipped=1)
2022-05-18T04:44:53.8264924Z 
2022-05-18T04:44:53.8265061Z Generating XML reports...
2022-05-18T04:44:53.8300299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044453.xml
2022-05-18T04:44:55.0817010Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:55.0831194Z 
2022-05-18T04:44:55.0831473Z Running tests...
2022-05-18T04:44:55.0831930Z ----------------------------------------------------------------------
2022-05-18T04:44:55.0859838Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T04:44:55.0860161Z 
2022-05-18T04:44:55.0860455Z ----------------------------------------------------------------------
2022-05-18T04:44:55.0860809Z Ran 1 test in 0.003s
2022-05-18T04:44:55.0860978Z 
2022-05-18T04:44:55.0861091Z OK (skipped=1)
2022-05-18T04:44:55.0861230Z 
2022-05-18T04:44:55.0896504Z Generating XML reports...
2022-05-18T04:44:55.0897158Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044455.xml
2022-05-18T04:44:56.3450247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:56.3464863Z 
2022-05-18T04:44:56.3465143Z Running tests...
2022-05-18T04:44:56.3465846Z ----------------------------------------------------------------------
2022-05-18T04:44:56.3495318Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T04:44:56.3495683Z 
2022-05-18T04:44:56.3495974Z ----------------------------------------------------------------------
2022-05-18T04:44:56.3496378Z Ran 1 test in 0.003s
2022-05-18T04:44:56.3496688Z 
2022-05-18T04:44:56.3496841Z OK (skipped=1)
2022-05-18T04:44:56.3496983Z 
2022-05-18T04:44:56.3497111Z Generating XML reports...
2022-05-18T04:44:56.3532279Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044456.xml
2022-05-18T04:44:57.6097787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:57.6111883Z 
2022-05-18T04:44:57.6112144Z Running tests...
2022-05-18T04:44:57.6112583Z ----------------------------------------------------------------------
2022-05-18T04:44:57.6136246Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:44:57.6136569Z 
2022-05-18T04:44:57.6136856Z ----------------------------------------------------------------------
2022-05-18T04:44:57.6137192Z Ran 1 test in 0.002s
2022-05-18T04:44:57.6137340Z 
2022-05-18T04:44:57.6137449Z OK (skipped=1)
2022-05-18T04:44:57.6137604Z 
2022-05-18T04:44:57.6137746Z Generating XML reports...
2022-05-18T04:44:57.6173564Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044457.xml
2022-05-18T04:44:58.8708320Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:44:58.8722828Z 
2022-05-18T04:44:58.8723283Z Running tests...
2022-05-18T04:44:58.8723810Z ----------------------------------------------------------------------
2022-05-18T04:44:58.8744671Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:44:58.8744995Z 
2022-05-18T04:44:58.8745288Z ----------------------------------------------------------------------
2022-05-18T04:44:58.8745608Z Ran 1 test in 0.002s
2022-05-18T04:44:58.8745775Z 
2022-05-18T04:44:58.8745887Z OK (skipped=1)
2022-05-18T04:44:58.8746044Z 
2022-05-18T04:44:58.8746171Z Generating XML reports...
2022-05-18T04:44:58.8781051Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044458.xml
2022-05-18T04:45:00.1330382Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:00.1344502Z 
2022-05-18T04:45:00.1344631Z Running tests...
2022-05-18T04:45:00.1345406Z ----------------------------------------------------------------------
2022-05-18T04:45:00.1371669Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:45:00.1372011Z 
2022-05-18T04:45:00.1372476Z ----------------------------------------------------------------------
2022-05-18T04:45:00.1372920Z Ran 1 test in 0.003s
2022-05-18T04:45:00.1373086Z 
2022-05-18T04:45:00.1373200Z OK (skipped=1)
2022-05-18T04:45:00.1373357Z 
2022-05-18T04:45:00.1373487Z Generating XML reports...
2022-05-18T04:45:00.1409025Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044500.xml
2022-05-18T04:45:01.3665403Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:01.3680556Z 
2022-05-18T04:45:01.3680817Z Running tests...
2022-05-18T04:45:01.3681269Z ----------------------------------------------------------------------
2022-05-18T04:45:01.3708233Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T04:45:01.3708868Z 
2022-05-18T04:45:01.3709191Z ----------------------------------------------------------------------
2022-05-18T04:45:01.3709544Z Ran 1 test in 0.003s
2022-05-18T04:45:01.3709709Z 
2022-05-18T04:45:01.3709824Z OK (skipped=1)
2022-05-18T04:45:01.3709966Z 
2022-05-18T04:45:01.3710096Z Generating XML reports...
2022-05-18T04:45:01.3745948Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044501.xml
2022-05-18T04:45:02.6057255Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:02.6072411Z 
2022-05-18T04:45:02.6072882Z Running tests...
2022-05-18T04:45:02.6073384Z ----------------------------------------------------------------------
2022-05-18T04:45:02.6096351Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:45:02.6096899Z 
2022-05-18T04:45:02.6097225Z ----------------------------------------------------------------------
2022-05-18T04:45:02.6097749Z Ran 1 test in 0.002s
2022-05-18T04:45:02.6098044Z 
2022-05-18T04:45:02.6098165Z OK (skipped=1)
2022-05-18T04:45:02.6098324Z 
2022-05-18T04:45:02.6098433Z Generating XML reports...
2022-05-18T04:45:02.6135886Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044502.xml
2022-05-18T04:45:03.8739631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:03.8754631Z 
2022-05-18T04:45:03.8754964Z Running tests...
2022-05-18T04:45:03.8755417Z ----------------------------------------------------------------------
2022-05-18T04:45:05.4730652Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:05.5128653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50296
2022-05-18T04:45:05.5235376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50297
2022-05-18T04:45:06.7381383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:06.7381947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:06.7382757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:06.7383438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:06.7389804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:06.7390311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:06.9287115Z ok (3.053s)
2022-05-18T04:45:06.9287367Z 
2022-05-18T04:45:06.9287744Z ----------------------------------------------------------------------
2022-05-18T04:45:06.9288095Z Ran 1 test in 3.053s
2022-05-18T04:45:06.9288263Z 
2022-05-18T04:45:06.9288362Z OK
2022-05-18T04:45:06.9288498Z 
2022-05-18T04:45:06.9290825Z Generating XML reports...
2022-05-18T04:45:06.9330502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044503.xml
2022-05-18T04:45:08.3336781Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:08.3351245Z 
2022-05-18T04:45:08.3351408Z Running tests...
2022-05-18T04:45:08.3352095Z ----------------------------------------------------------------------
2022-05-18T04:45:09.9186867Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:09.9574155Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50409
2022-05-18T04:45:09.9678780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50410
2022-05-18T04:45:11.1717775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:11.1718707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:11.1719525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:11.1720230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:11.1726351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:11.1726889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:13.0759505Z ok (4.740s)
2022-05-18T04:45:13.0759725Z 
2022-05-18T04:45:13.0760126Z ----------------------------------------------------------------------
2022-05-18T04:45:13.0760770Z Ran 1 test in 4.741s
2022-05-18T04:45:13.0760963Z 
2022-05-18T04:45:13.0761063Z OK
2022-05-18T04:45:13.0761202Z 
2022-05-18T04:45:13.0761347Z Generating XML reports...
2022-05-18T04:45:13.0801669Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044508.xml
2022-05-18T04:45:14.4990922Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:14.5005512Z 
2022-05-18T04:45:14.5005836Z Running tests...
2022-05-18T04:45:14.5006541Z ----------------------------------------------------------------------
2022-05-18T04:45:16.0713127Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:16.1110971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50524
2022-05-18T04:45:16.1216461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50525
2022-05-18T04:45:17.3039652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:17.3040223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:17.3041047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:17.3041753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:17.3048048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:17.3048799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:17.3256109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:45:17.3256763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:45:17.3257489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:45:17.3258171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:45:17.5266794Z ok (3.026s)
2022-05-18T04:45:17.5267164Z 
2022-05-18T04:45:17.5267578Z ----------------------------------------------------------------------
2022-05-18T04:45:17.5267924Z Ran 1 test in 3.026s
2022-05-18T04:45:17.5268073Z 
2022-05-18T04:45:17.5268172Z OK
2022-05-18T04:45:17.5268308Z 
2022-05-18T04:45:17.5268445Z Generating XML reports...
2022-05-18T04:45:17.5309140Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044514.xml
2022-05-18T04:45:18.9362100Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:18.9376325Z 
2022-05-18T04:45:18.9376863Z Running tests...
2022-05-18T04:45:18.9377370Z ----------------------------------------------------------------------
2022-05-18T04:45:20.5250815Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:20.5647330Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50643
2022-05-18T04:45:20.5753652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50644
2022-05-18T04:45:21.7458128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:21.7458897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:21.7459795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:21.7460760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:21.7466524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:21.7467339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:21.9808766Z skip: Skipped due to small world size. (3.043s)
2022-05-18T04:45:21.9809097Z 
2022-05-18T04:45:21.9809648Z ----------------------------------------------------------------------
2022-05-18T04:45:21.9809981Z Ran 1 test in 3.043s
2022-05-18T04:45:21.9810153Z 
2022-05-18T04:45:21.9810266Z OK (skipped=1)
2022-05-18T04:45:21.9810520Z 
2022-05-18T04:45:21.9810756Z Generating XML reports...
2022-05-18T04:45:21.9850834Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044518.xml
2022-05-18T04:45:23.4108044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:23.4122893Z 
2022-05-18T04:45:23.4123493Z Running tests...
2022-05-18T04:45:23.4123980Z ----------------------------------------------------------------------
2022-05-18T04:45:24.9746868Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:25.0146300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50752
2022-05-18T04:45:25.0252181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50753
2022-05-18T04:45:26.1767569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:26.1768134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:26.1769245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:26.1769968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:26.1875272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:26.2781164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:27.8328450Z ok (4.420s)
2022-05-18T04:45:27.8328682Z 
2022-05-18T04:45:27.8329075Z ----------------------------------------------------------------------
2022-05-18T04:45:27.8329405Z Ran 1 test in 4.421s
2022-05-18T04:45:27.8329585Z 
2022-05-18T04:45:27.8329683Z OK
2022-05-18T04:45:27.8329820Z 
2022-05-18T04:45:27.8329955Z Generating XML reports...
2022-05-18T04:45:27.8370539Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044523.xml
2022-05-18T04:45:29.2681017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:29.2695668Z 
2022-05-18T04:45:29.2695881Z Running tests...
2022-05-18T04:45:29.2696359Z ----------------------------------------------------------------------
2022-05-18T04:45:30.8498782Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:30.8898245Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50863
2022-05-18T04:45:30.9005454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50864
2022-05-18T04:45:32.0797602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:32.0798570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:32.0799381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:32.0800090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:32.0905706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:32.1809478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:32.4058080Z ok (3.136s)
2022-05-18T04:45:32.4058320Z 
2022-05-18T04:45:32.4058755Z ----------------------------------------------------------------------
2022-05-18T04:45:32.4059100Z Ran 1 test in 3.136s
2022-05-18T04:45:32.4059269Z 
2022-05-18T04:45:32.4059364Z OK
2022-05-18T04:45:32.4059481Z 
2022-05-18T04:45:32.4059617Z Generating XML reports...
2022-05-18T04:45:32.4100038Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044529.xml
2022-05-18T04:45:33.8053871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:33.8068708Z 
2022-05-18T04:45:33.8069198Z Running tests...
2022-05-18T04:45:33.8069695Z ----------------------------------------------------------------------
2022-05-18T04:45:35.3742127Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:35.4140090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50972
2022-05-18T04:45:35.4248947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50973
2022-05-18T04:45:36.6242468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:36.6243017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:36.6243820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:36.6244818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:36.6350956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:36.7255064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:36.7364994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:45:36.7365545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:45:36.7366281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:45:36.7366963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:45:36.7573621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:45:36.7574140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:45:36.7574838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:45:36.7575516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:45:38.0541937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsjzunvq
2022-05-18T04:45:38.0542552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsjzunvq/_remote_module_non_scriptable.py
2022-05-18T04:45:38.0599152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpye9b9u2z
2022-05-18T04:45:38.0602826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpye9b9u2z/_remote_module_non_scriptable.py
2022-05-18T04:45:38.4329864Z ok (4.626s)
2022-05-18T04:45:38.4330257Z 
2022-05-18T04:45:38.4330943Z ----------------------------------------------------------------------
2022-05-18T04:45:38.4331576Z Ran 1 test in 4.626s
2022-05-18T04:45:38.4331885Z 
2022-05-18T04:45:38.4332027Z OK
2022-05-18T04:45:38.4332269Z 
2022-05-18T04:45:38.4332902Z Generating XML reports...
2022-05-18T04:45:38.4373708Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044533.xml
2022-05-18T04:45:39.8686955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:39.8701867Z 
2022-05-18T04:45:39.8702111Z Running tests...
2022-05-18T04:45:39.8702606Z ----------------------------------------------------------------------
2022-05-18T04:45:41.4619687Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:41.5017439Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51095
2022-05-18T04:45:41.5125613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51096
2022-05-18T04:45:42.7021459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:42.7022053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:42.7022870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:42.7023548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:42.7128838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:42.8032215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:42.8143142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:45:42.8143677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:45:42.8144412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:45:42.8145123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:45:42.8354102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:45:42.8354618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:45:42.8355303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:45:42.8355986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:45:44.1054396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5tsm7w2
2022-05-18T04:45:44.1055010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5tsm7w2/_remote_module_non_scriptable.py
2022-05-18T04:45:44.1415578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0clc9g5v
2022-05-18T04:45:44.1417044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0clc9g5v/_remote_module_non_scriptable.py
2022-05-18T04:45:44.5205979Z ok (4.650s)
2022-05-18T04:45:44.5206207Z 
2022-05-18T04:45:44.5206603Z ----------------------------------------------------------------------
2022-05-18T04:45:44.5206932Z Ran 1 test in 4.650s
2022-05-18T04:45:44.5207103Z 
2022-05-18T04:45:44.5207208Z OK
2022-05-18T04:45:44.5207350Z 
2022-05-18T04:45:44.5207491Z Generating XML reports...
2022-05-18T04:45:44.5247199Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044539.xml
2022-05-18T04:45:45.9597167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:45.9612167Z 
2022-05-18T04:45:45.9612334Z Running tests...
2022-05-18T04:45:45.9613619Z ----------------------------------------------------------------------
2022-05-18T04:45:47.5417617Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:47.5810073Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51218
2022-05-18T04:45:47.5917767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51219
2022-05-18T04:45:48.8017604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:48.8018179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:48.8018995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:48.8019703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:48.8025762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:48.8026264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:50.0951655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehb3j4d_
2022-05-18T04:45:50.0952553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehb3j4d_/_remote_module_non_scriptable.py
2022-05-18T04:45:50.1225895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2hfo_w_
2022-05-18T04:45:50.1228424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2hfo_w_/_remote_module_non_scriptable.py
2022-05-18T04:45:50.3135399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:45:50.3135964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:45:50.5997894Z ok (4.638s)
2022-05-18T04:45:50.5998080Z 
2022-05-18T04:45:50.5998762Z ----------------------------------------------------------------------
2022-05-18T04:45:50.5999127Z Ran 1 test in 4.639s
2022-05-18T04:45:50.5999296Z 
2022-05-18T04:45:50.5999375Z OK
2022-05-18T04:45:50.5999514Z 
2022-05-18T04:45:50.5999650Z Generating XML reports...
2022-05-18T04:45:50.6040873Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044545.xml
2022-05-18T04:45:52.0530475Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:52.0545129Z 
2022-05-18T04:45:52.0545408Z Running tests...
2022-05-18T04:45:52.0545832Z ----------------------------------------------------------------------
2022-05-18T04:45:53.6355226Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:53.6752305Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51333
2022-05-18T04:45:53.6858550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51334
2022-05-18T04:45:54.8869201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:45:54.8869776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:45:54.8870558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:54.8871265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:45:54.8877012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:54.8877937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:56.1834128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ih1m41o
2022-05-18T04:45:56.1834939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ih1m41o/_remote_module_non_scriptable.py
2022-05-18T04:45:56.1953841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nzqqpa9
2022-05-18T04:45:56.1956871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nzqqpa9/_remote_module_non_scriptable.py
2022-05-18T04:45:56.3905365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:45:56.3905916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:45:56.3929563Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:45:56.3930090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:45:56.6939283Z ok (4.639s)
2022-05-18T04:45:56.6939481Z 
2022-05-18T04:45:56.6939885Z ----------------------------------------------------------------------
2022-05-18T04:45:56.6940240Z Ran 1 test in 4.639s
2022-05-18T04:45:56.6940411Z 
2022-05-18T04:45:56.6940518Z OK
2022-05-18T04:45:56.6940658Z 
2022-05-18T04:45:56.6940778Z Generating XML reports...
2022-05-18T04:45:56.6981074Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044552.xml
2022-05-18T04:45:58.1192457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:45:58.1207380Z 
2022-05-18T04:45:58.1207764Z Running tests...
2022-05-18T04:45:58.1208261Z ----------------------------------------------------------------------
2022-05-18T04:45:59.7070874Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:59.7470916Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51448
2022-05-18T04:45:59.7577199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51449
2022-05-18T04:46:00.9588112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:00.9588686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:00.9589462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:00.9590164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:00.9594533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:00.9595033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:02.2746441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjgf09_c
2022-05-18T04:46:02.2747644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjgf09_c/_remote_module_non_scriptable.py
2022-05-18T04:46:02.2809154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxvze7oq
2022-05-18T04:46:02.2812253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxvze7oq/_remote_module_non_scriptable.py
2022-05-18T04:46:02.4804190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:02.4804763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:02.4819709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:02.4820220Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:02.4980836Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:02.4981331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:02.4995397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:02.4995925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:02.7655271Z ok (4.644s)
2022-05-18T04:46:02.7655512Z 
2022-05-18T04:46:02.7655923Z ----------------------------------------------------------------------
2022-05-18T04:46:02.7656244Z Ran 1 test in 4.645s
2022-05-18T04:46:02.7656416Z 
2022-05-18T04:46:02.7656510Z OK
2022-05-18T04:46:02.7659249Z 
2022-05-18T04:46:02.7659921Z Generating XML reports...
2022-05-18T04:46:02.7697502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044558.xml
2022-05-18T04:46:04.1905299Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:46:04.1919719Z 
2022-05-18T04:46:04.1920206Z Running tests...
2022-05-18T04:46:04.1920686Z ----------------------------------------------------------------------
2022-05-18T04:46:05.7671923Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:05.7826801Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.590s)
2022-05-18T04:46:05.7827441Z 
2022-05-18T04:46:05.7827726Z ----------------------------------------------------------------------
2022-05-18T04:46:05.7828063Z Ran 1 test in 1.591s
2022-05-18T04:46:05.7828232Z 
2022-05-18T04:46:05.7828326Z OK (skipped=1)
2022-05-18T04:46:05.7828779Z 
2022-05-18T04:46:05.7828917Z Generating XML reports...
2022-05-18T04:46:05.7861317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044604.xml
2022-05-18T04:46:07.1342790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:46:07.1356931Z 
2022-05-18T04:46:07.1357382Z Running tests...
2022-05-18T04:46:07.1357863Z ----------------------------------------------------------------------
2022-05-18T04:46:08.7120763Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:08.7510233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51597
2022-05-18T04:46:08.7615606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51598
2022-05-18T04:46:09.9600718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:09.9601276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:09.9602077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:09.9602778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:09.9608832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:09.9609313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:11.2522446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp347t85mo
2022-05-18T04:46:11.2523411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp347t85mo/_remote_module_non_scriptable.py
2022-05-18T04:46:11.2626359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnangvyvk
2022-05-18T04:46:11.2629075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnangvyvk/_remote_module_non_scriptable.py
2022-05-18T04:46:11.2828715Z <class 'torch.nn.parameter.Parameter'>
2022-05-18T04:46:11.5702159Z ok (4.434s)
2022-05-18T04:46:11.5702344Z 
2022-05-18T04:46:11.5703133Z ----------------------------------------------------------------------
2022-05-18T04:46:11.5703542Z Ran 1 test in 4.435s
2022-05-18T04:46:11.5703715Z 
2022-05-18T04:46:11.5703809Z OK
2022-05-18T04:46:11.5703951Z 
2022-05-18T04:46:11.5704085Z Generating XML reports...
2022-05-18T04:46:11.5745195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044607.xml
2022-05-18T04:46:12.9659340Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:46:12.9672529Z 
2022-05-18T04:46:12.9672866Z Running tests...
2022-05-18T04:46:12.9673321Z ----------------------------------------------------------------------
2022-05-18T04:46:14.5095984Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:14.5488642Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51708
2022-05-18T04:46:14.5593119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51709
2022-05-18T04:46:15.7234397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:15.7234952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:15.7235750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:15.7236452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:15.7242983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:15.7243915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:17.0017276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6cjijfh
2022-05-18T04:46:17.0018209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6cjijfh/_remote_module_non_scriptable.py
2022-05-18T04:46:17.0262465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_3w9y17
2022-05-18T04:46:17.0265285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_3w9y17/_remote_module_non_scriptable.py
2022-05-18T04:46:17.3667430Z ok (4.399s)
2022-05-18T04:46:17.3667836Z 
2022-05-18T04:46:17.3668280Z ----------------------------------------------------------------------
2022-05-18T04:46:17.3668628Z Ran 1 test in 4.399s
2022-05-18T04:46:17.3668795Z 
2022-05-18T04:46:17.3668894Z OK
2022-05-18T04:46:17.3669016Z 
2022-05-18T04:46:17.3669183Z Generating XML reports...
2022-05-18T04:46:17.3709990Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044612.xml
2022-05-18T04:46:18.7964650Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:46:18.7978994Z 
2022-05-18T04:46:18.7979165Z Running tests...
2022-05-18T04:46:18.7979992Z ----------------------------------------------------------------------
2022-05-18T04:46:20.3772294Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:20.4171513Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51819
2022-05-18T04:46:20.4277888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51820
2022-05-18T04:46:21.6066933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:21.6067466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:21.6068302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:21.6069216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:21.6174625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:21.7080353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:22.8758503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcg687weg
2022-05-18T04:46:22.8759145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcg687weg/_remote_module_non_scriptable.py
2022-05-18T04:46:22.9577561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4j1rmks
2022-05-18T04:46:22.9578435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4j1rmks/_remote_module_non_scriptable.py
2022-05-18T04:46:23.1610414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:23.1610959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:23.4368634Z ok (4.639s)
2022-05-18T04:46:23.4368976Z 
2022-05-18T04:46:23.4369527Z ----------------------------------------------------------------------
2022-05-18T04:46:23.4369874Z Ran 1 test in 4.639s
2022-05-18T04:46:23.4370041Z 
2022-05-18T04:46:23.4370137Z OK
2022-05-18T04:46:23.4372058Z 
2022-05-18T04:46:23.4372656Z Generating XML reports...
2022-05-18T04:46:23.4411211Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044618.xml
2022-05-18T04:46:24.8543467Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:46:24.8558377Z 
2022-05-18T04:46:24.8559033Z Running tests...
2022-05-18T04:46:24.8559523Z ----------------------------------------------------------------------
2022-05-18T04:46:26.4584590Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:26.4985915Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51934
2022-05-18T04:46:26.5092188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51935
2022-05-18T04:46:27.7253930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:27.7254518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:27.7255311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:27.7256022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:27.7262127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:27.7262600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:29.0215692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0hrhvrgw
2022-05-18T04:46:29.0216949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0hrhvrgw/_remote_module_non_scriptable.py
2022-05-18T04:46:29.0341140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_7d5369
2022-05-18T04:46:29.0344193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_7d5369/_remote_module_non_scriptable.py
2022-05-18T04:46:29.2254783Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:46:29.6173934Z ok (4.761s)
2022-05-18T04:46:29.6174156Z 
2022-05-18T04:46:29.6174544Z ----------------------------------------------------------------------
2022-05-18T04:46:29.6174890Z Ran 1 test in 4.762s
2022-05-18T04:46:29.6175056Z 
2022-05-18T04:46:29.6175153Z OK
2022-05-18T04:46:29.6175272Z 
2022-05-18T04:46:29.6175403Z Generating XML reports...
2022-05-18T04:46:29.6216495Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044624.xml
2022-05-18T04:46:31.0282678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:46:31.0298463Z 
2022-05-18T04:46:31.0298893Z Running tests...
2022-05-18T04:46:31.0299428Z ----------------------------------------------------------------------
2022-05-18T04:46:32.5717367Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:32.6108826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52049
2022-05-18T04:46:32.6218033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52050
2022-05-18T04:46:33.7990449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:33.7990995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:33.7991796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:33.7992507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:33.8099992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:33.9005008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:35.0466494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcdomwwg2
2022-05-18T04:46:35.0467124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcdomwwg2/_remote_module_non_scriptable.py
2022-05-18T04:46:35.1560505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1cd08pyk
2022-05-18T04:46:35.1562342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1cd08pyk/_remote_module_non_scriptable.py
2022-05-18T04:46:35.3458300Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:46:35.3459939Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:46:35.7298471Z ok (4.700s)
2022-05-18T04:46:35.7298691Z 
2022-05-18T04:46:35.7299066Z ----------------------------------------------------------------------
2022-05-18T04:46:35.7299431Z Ran 1 test in 4.700s
2022-05-18T04:46:35.7299603Z 
2022-05-18T04:46:35.7299697Z OK
2022-05-18T04:46:35.7299834Z 
2022-05-18T04:46:35.7299970Z Generating XML reports...
2022-05-18T04:46:35.7341787Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044631.xml
2022-05-18T04:46:37.1376926Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:46:37.1390139Z 
2022-05-18T04:46:37.1390582Z Running tests...
2022-05-18T04:46:37.1391093Z ----------------------------------------------------------------------
2022-05-18T04:46:38.6833671Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:38.7228537Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52164
2022-05-18T04:46:38.7334209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52165
2022-05-18T04:46:39.9485731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:39.9486287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:39.9487103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:39.9487801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:39.9494450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:39.9494922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:39.9592199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq0o40e1c
2022-05-18T04:46:39.9594762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq0o40e1c/_remote_module_non_scriptable.py
2022-05-18T04:46:39.9595578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp764e909x
2022-05-18T04:46:39.9598886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp764e909x/_remote_module_non_scriptable.py
2022-05-18T04:46:39.9742627Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:46:39.9744168Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:46:39.9748139Z /opt/conda/lib/python3.9/site-packages/torch/autograd/__init__.py:173: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T04:46:39.9749157Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-05-18T04:46:39.9750826Z /opt/conda/lib/python3.9/site-packages/torch/autograd/__init__.py:173: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T04:46:39.9751952Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-05-18T04:46:39.9752454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:39.9753168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:39.9756015Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:46:39.9757529Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:46:39.9762386Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:46:39.9763865Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:46:39.9767958Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:46:39.9769431Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:46:39.9773222Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:46:39.9774701Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:46:39.9778393Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:46:39.9779976Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:46:40.1384030Z ok (2.999s)
2022-05-18T04:46:40.1384272Z 
2022-05-18T04:46:40.1384676Z ----------------------------------------------------------------------
2022-05-18T04:46:40.1385003Z Ran 1 test in 2.999s
2022-05-18T04:46:40.1385172Z 
2022-05-18T04:46:40.1385270Z OK
2022-05-18T04:46:40.1385407Z 
2022-05-18T04:46:40.1385550Z Generating XML reports...
2022-05-18T04:46:40.1426703Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044637.xml
2022-05-18T04:46:41.5610577Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:46:41.5624999Z 
2022-05-18T04:46:41.5625443Z Running tests...
2022-05-18T04:46:41.5625951Z ----------------------------------------------------------------------
2022-05-18T04:46:43.1554833Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:43.1953991Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52277
2022-05-18T04:46:43.2061103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52278
2022-05-18T04:46:44.3978868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:44.3979444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:44.3980252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:44.3981202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:44.4086644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:44.4992365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:45.6679142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcszgv9zl
2022-05-18T04:46:45.6679783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcszgv9zl/_remote_module_non_scriptable.py
2022-05-18T04:46:45.7553193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8c2fl8s
2022-05-18T04:46:45.7554584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8c2fl8s/_remote_module_non_scriptable.py
2022-05-18T04:46:45.9455160Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:45.9455736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:46.3143221Z ok (4.751s)
2022-05-18T04:46:46.3143441Z 
2022-05-18T04:46:46.3143825Z ----------------------------------------------------------------------
2022-05-18T04:46:46.3144172Z Ran 1 test in 4.752s
2022-05-18T04:46:46.3144339Z 
2022-05-18T04:46:46.3144460Z OK
2022-05-18T04:46:46.3144579Z 
2022-05-18T04:46:46.3144720Z Generating XML reports...
2022-05-18T04:46:46.3185490Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044641.xml
2022-05-18T04:46:47.7275835Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:46:47.7290277Z 
2022-05-18T04:46:47.7290565Z Running tests...
2022-05-18T04:46:47.7291004Z ----------------------------------------------------------------------
2022-05-18T04:46:49.3004163Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:49.3403247Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52392
2022-05-18T04:46:49.3509120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52393
2022-05-18T04:46:50.5514295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:50.5514890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:50.5515702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:50.5516386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:50.5622016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:50.6529037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:51.8080815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjanzuu2
2022-05-18T04:46:51.8081430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjanzuu2/_remote_module_non_scriptable.py
2022-05-18T04:46:51.9141929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8o531nbp
2022-05-18T04:46:51.9142523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8o531nbp/_remote_module_non_scriptable.py
2022-05-18T04:46:51.9352865Z /opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T04:46:51.9354808Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T04:46:51.9357229Z /opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T04:46:51.9359789Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T04:46:52.3586971Z ok (4.629s)
2022-05-18T04:46:52.3587191Z 
2022-05-18T04:46:52.3587593Z ----------------------------------------------------------------------
2022-05-18T04:46:52.3587936Z Ran 1 test in 4.630s
2022-05-18T04:46:52.3588087Z 
2022-05-18T04:46:52.3588187Z OK
2022-05-18T04:46:52.3588322Z 
2022-05-18T04:46:52.3588456Z Generating XML reports...
2022-05-18T04:46:52.3628677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044647.xml
2022-05-18T04:46:53.7847139Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:46:53.7861642Z 
2022-05-18T04:46:53.7861801Z Running tests...
2022-05-18T04:46:53.7862504Z ----------------------------------------------------------------------
2022-05-18T04:46:55.3685545Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:55.4085373Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52537
2022-05-18T04:46:55.4192046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52538
2022-05-18T04:46:56.6224544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:46:56.6225102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:46:56.6225898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:56.6226617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:46:56.6232427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:56.6233368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:57.9361932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn50pxmwn
2022-05-18T04:46:57.9363080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn50pxmwn/_remote_module_non_scriptable.py
2022-05-18T04:46:57.9396285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4es1xhb_
2022-05-18T04:46:57.9399091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4es1xhb_/_remote_module_non_scriptable.py
2022-05-18T04:46:58.1276582Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:58.1277121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:46:58.1523256Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp4es1xhb_'>
2022-05-18T04:46:58.1523760Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:46:58.1524359Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpn50pxmwn'>
2022-05-18T04:46:58.1524801Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:46:58.4269945Z ok (4.640s)
2022-05-18T04:46:58.4271018Z 
2022-05-18T04:46:58.4271414Z ----------------------------------------------------------------------
2022-05-18T04:46:58.4271770Z Ran 1 test in 4.641s
2022-05-18T04:46:58.4271920Z 
2022-05-18T04:46:58.4272015Z OK
2022-05-18T04:46:58.4272156Z 
2022-05-18T04:46:58.4272299Z Generating XML reports...
2022-05-18T04:46:58.4312239Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044653.xml
2022-05-18T04:46:59.8453267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:46:59.8468027Z 
2022-05-18T04:46:59.8468314Z Running tests...
2022-05-18T04:46:59.8468792Z ----------------------------------------------------------------------
2022-05-18T04:47:01.4074883Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:01.4227091Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.576s)
2022-05-18T04:47:01.4227871Z 
2022-05-18T04:47:01.4228166Z ----------------------------------------------------------------------
2022-05-18T04:47:01.4228508Z Ran 1 test in 1.576s
2022-05-18T04:47:01.4228673Z 
2022-05-18T04:47:01.4228783Z OK (skipped=1)
2022-05-18T04:47:01.4228938Z 
2022-05-18T04:47:01.4229065Z Generating XML reports...
2022-05-18T04:47:01.4260993Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044659.xml
2022-05-18T04:47:02.7780645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:47:02.7795597Z 
2022-05-18T04:47:02.7795944Z Running tests...
2022-05-18T04:47:02.7796902Z ----------------------------------------------------------------------
2022-05-18T04:47:04.3554934Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:04.3949093Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52686
2022-05-18T04:47:04.4056109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52687
2022-05-18T04:47:05.5897943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:05.5898912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:05.5899932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:05.5900715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:05.5905811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:05.5906919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:05.6015092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:47:05.6016252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:47:05.6017005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:47:05.6017705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:47:06.8783028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuq2zf3fl
2022-05-18T04:47:06.8784024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuq2zf3fl/_remote_module_non_scriptable.py
2022-05-18T04:47:06.9326477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xsrv7hk
2022-05-18T04:47:06.9328069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xsrv7hk/_remote_module_non_scriptable.py
2022-05-18T04:47:07.1238934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:07.1239869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:07.1255193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:07.1256214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:07.7140588Z ok (4.934s)
2022-05-18T04:47:07.7140804Z 
2022-05-18T04:47:07.7141213Z ----------------------------------------------------------------------
2022-05-18T04:47:07.7141562Z Ran 1 test in 4.934s
2022-05-18T04:47:07.7141727Z 
2022-05-18T04:47:07.7141814Z OK
2022-05-18T04:47:07.7141948Z 
2022-05-18T04:47:07.7142082Z Generating XML reports...
2022-05-18T04:47:07.7182736Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044702.xml
2022-05-18T04:47:09.1364201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:47:09.1379451Z 
2022-05-18T04:47:09.1379713Z Running tests...
2022-05-18T04:47:09.1380151Z ----------------------------------------------------------------------
2022-05-18T04:47:10.7102320Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:10.7500820Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52807
2022-05-18T04:47:10.7610423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52808
2022-05-18T04:47:11.9348024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:11.9348611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:11.9349394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:11.9350095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:11.9457131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:11.9459320Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:47:12.0364518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:12.0365157Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:47:13.1821383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ut6m3pt
2022-05-18T04:47:13.1822005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ut6m3pt/_remote_module_non_scriptable.py
2022-05-18T04:47:13.2936378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvai_ooip
2022-05-18T04:47:13.2938006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvai_ooip/_remote_module_non_scriptable.py
2022-05-18T04:47:13.4889753Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:13.4890337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:13.4907337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:13.4907861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:13.5169801Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:47:13.5170415Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:47:13.7707818Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:47:13.7708411Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T04:47:13.7785278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:13.7785793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:13.7803450Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:13.7803968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:13.8050107Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:47:13.8051077Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T04:47:13.9579728Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T04:47:13.9580329Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T04:47:13.9656273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:13.9656787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:13.9673651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:13.9674132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:14.5701101Z ok (5.432s)
2022-05-18T04:47:14.5701304Z 
2022-05-18T04:47:14.5701690Z ----------------------------------------------------------------------
2022-05-18T04:47:14.5702029Z Ran 1 test in 5.432s
2022-05-18T04:47:14.5702200Z 
2022-05-18T04:47:14.5702303Z OK
2022-05-18T04:47:14.5702439Z 
2022-05-18T04:47:14.5702554Z Generating XML reports...
2022-05-18T04:47:14.5744023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044709.xml
2022-05-18T04:47:15.9989794Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:47:16.0005397Z 
2022-05-18T04:47:16.0005799Z Running tests...
2022-05-18T04:47:16.0006315Z ----------------------------------------------------------------------
2022-05-18T04:47:17.5770731Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:17.6166678Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52922
2022-05-18T04:47:17.6272719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52923
2022-05-18T04:47:18.8139422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:18.8139990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:18.8140778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:18.8141515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:18.8148436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:18.8149053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:18.8150649Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:47:18.8151750Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:47:20.0932421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_e5yjom5
2022-05-18T04:47:20.0933494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_e5yjom5/_remote_module_non_scriptable.py
2022-05-18T04:47:20.1105280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1jlpx3tq
2022-05-18T04:47:20.1108426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1jlpx3tq/_remote_module_non_scriptable.py
2022-05-18T04:47:20.3023900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:20.3024440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:20.3040367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:20.3040874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:20.3047614Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:47:20.3048620Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:47:20.3080267Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:47:20.3081119Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:47:20.3082606Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:47:20.3083257Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:47:20.3084205Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T04:47:20.3085612Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T04:47:20.6315334Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:47:20.6316445Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:47:20.6396722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:20.6397234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:20.6415090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:20.6415591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:20.6421115Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:47:20.6421977Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T04:47:20.6452410Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:47:20.6453032Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T04:47:20.6453837Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:47:20.6454510Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T04:47:21.2363786Z ok (5.235s)
2022-05-18T04:47:21.2364003Z 
2022-05-18T04:47:21.2364395Z ----------------------------------------------------------------------
2022-05-18T04:47:21.2364720Z Ran 1 test in 5.236s
2022-05-18T04:47:21.2364898Z 
2022-05-18T04:47:21.2364997Z OK
2022-05-18T04:47:21.2365138Z 
2022-05-18T04:47:21.2365273Z Generating XML reports...
2022-05-18T04:47:21.2405999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044715.xml
2022-05-18T04:47:22.6411955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:47:22.6425885Z 
2022-05-18T04:47:22.6426406Z Running tests...
2022-05-18T04:47:22.6426911Z ----------------------------------------------------------------------
2022-05-18T04:47:24.1817649Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:24.2205547Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53037
2022-05-18T04:47:24.2312589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53038
2022-05-18T04:47:25.4380762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:25.4381347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:25.4382133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:25.4382863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:25.4388443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:25.4389224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:26.7514977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr60fqd8s
2022-05-18T04:47:26.7515637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr60fqd8s/_remote_module_non_scriptable.py
2022-05-18T04:47:26.7807624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp16ri7u81
2022-05-18T04:47:26.7809105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp16ri7u81/_remote_module_non_scriptable.py
2022-05-18T04:47:27.0178265Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:27.0178850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:27.0874194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:27.0874746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:27.5398747Z ok (4.897s)
2022-05-18T04:47:27.5399634Z 
2022-05-18T04:47:27.5400250Z ----------------------------------------------------------------------
2022-05-18T04:47:27.5400612Z Ran 1 test in 4.897s
2022-05-18T04:47:27.5400763Z 
2022-05-18T04:47:27.5400862Z OK
2022-05-18T04:47:27.5401006Z 
2022-05-18T04:47:27.5401142Z Generating XML reports...
2022-05-18T04:47:27.5441196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044722.xml
2022-05-18T04:47:28.9327700Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:47:28.9341278Z 
2022-05-18T04:47:28.9341602Z Running tests...
2022-05-18T04:47:28.9342057Z ----------------------------------------------------------------------
2022-05-18T04:47:30.4718895Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:30.5108724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53182
2022-05-18T04:47:30.5216287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53183
2022-05-18T04:47:31.6720218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:31.6720779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:31.6721584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:31.6722290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:31.6829932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:31.7734509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:32.9565854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0bzvl13y
2022-05-18T04:47:32.9567181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0bzvl13y/_remote_module_non_scriptable.py
2022-05-18T04:47:33.0659979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9vprkxw
2022-05-18T04:47:33.0662168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9vprkxw/_remote_module_non_scriptable.py
2022-05-18T04:47:33.3035583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:33.3036134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:33.3698449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:33.3698983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:33.7310998Z ok (4.797s)
2022-05-18T04:47:33.7311467Z 
2022-05-18T04:47:33.7311888Z ----------------------------------------------------------------------
2022-05-18T04:47:33.7312231Z Ran 1 test in 4.797s
2022-05-18T04:47:33.7312400Z 
2022-05-18T04:47:33.7312478Z OK
2022-05-18T04:47:33.7312614Z 
2022-05-18T04:47:33.7312754Z Generating XML reports...
2022-05-18T04:47:33.7354361Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044728.xml
2022-05-18T04:47:35.1474676Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:47:35.1490220Z 
2022-05-18T04:47:35.1490720Z Running tests...
2022-05-18T04:47:35.1491283Z ----------------------------------------------------------------------
2022-05-18T04:47:36.7212893Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:36.7600246Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53327
2022-05-18T04:47:36.7705838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53328
2022-05-18T04:47:37.9445335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:37.9445918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:37.9446722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:37.9447428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:37.9552930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:38.0459489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:39.2185713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnm4e3_5_
2022-05-18T04:47:39.2186324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnm4e3_5_/_remote_module_non_scriptable.py
2022-05-18T04:47:39.3418699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplphloxzi
2022-05-18T04:47:39.3420042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplphloxzi/_remote_module_non_scriptable.py
2022-05-18T04:47:39.5778573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:39.5779124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:39.6459061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:39.6459603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:39.9789712Z ok (4.829s)
2022-05-18T04:47:39.9789942Z 
2022-05-18T04:47:39.9790370Z ----------------------------------------------------------------------
2022-05-18T04:47:39.9790722Z Ran 1 test in 4.830s
2022-05-18T04:47:39.9790892Z 
2022-05-18T04:47:39.9790970Z OK
2022-05-18T04:47:39.9792310Z 
2022-05-18T04:47:39.9792740Z Generating XML reports...
2022-05-18T04:47:39.9833516Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044735.xml
2022-05-18T04:47:41.4088711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:47:41.4104295Z 
2022-05-18T04:47:41.4104753Z Running tests...
2022-05-18T04:47:41.4105261Z ----------------------------------------------------------------------
2022-05-18T04:47:42.9832729Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:43.0222056Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53472
2022-05-18T04:47:43.0328989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53473
2022-05-18T04:47:44.1774407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:44.1775506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:44.1776617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:44.1777311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:44.1881227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:44.2789259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:45.4600949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhzt880u
2022-05-18T04:47:45.4602085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhzt880u/_remote_module_non_scriptable.py
2022-05-18T04:47:45.5622073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7yy19w7
2022-05-18T04:47:45.5622897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7yy19w7/_remote_module_non_scriptable.py
2022-05-18T04:47:45.7969946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:45.7970506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:45.8628402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:45.8628967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:46.2412176Z ok (4.830s)
2022-05-18T04:47:46.2412398Z 
2022-05-18T04:47:46.2412832Z ----------------------------------------------------------------------
2022-05-18T04:47:46.2413176Z Ran 1 test in 4.831s
2022-05-18T04:47:46.2413327Z 
2022-05-18T04:47:46.2413427Z OK
2022-05-18T04:47:46.2413569Z 
2022-05-18T04:47:46.2413708Z Generating XML reports...
2022-05-18T04:47:46.2454003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044741.xml
2022-05-18T04:47:47.6415995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:47:47.6430599Z 
2022-05-18T04:47:47.6430749Z Running tests...
2022-05-18T04:47:47.6431498Z ----------------------------------------------------------------------
2022-05-18T04:47:49.1860021Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:49.2248471Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53617
2022-05-18T04:47:49.2356122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53618
2022-05-18T04:47:50.4527146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:50.4527745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:50.4528550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:50.4529252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:50.4637439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:50.5541578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:51.7460137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptg4zuul1
2022-05-18T04:47:51.7461177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptg4zuul1/_remote_module_non_scriptable.py
2022-05-18T04:47:51.8370384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97x25c_z
2022-05-18T04:47:51.8371255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97x25c_z/_remote_module_non_scriptable.py
2022-05-18T04:47:52.0866306Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:52.0866880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:52.1554739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:52.1555332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:52.5439008Z ok (4.900s)
2022-05-18T04:47:52.5439316Z 
2022-05-18T04:47:52.5439988Z ----------------------------------------------------------------------
2022-05-18T04:47:52.5440329Z Ran 1 test in 4.901s
2022-05-18T04:47:52.5440497Z 
2022-05-18T04:47:52.5440592Z OK
2022-05-18T04:47:52.5440732Z 
2022-05-18T04:47:52.5440869Z Generating XML reports...
2022-05-18T04:47:52.5480267Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044747.xml
2022-05-18T04:47:53.9575686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:47:53.9589943Z 
2022-05-18T04:47:53.9590265Z Running tests...
2022-05-18T04:47:53.9590716Z ----------------------------------------------------------------------
2022-05-18T04:47:55.5296133Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:55.5686221Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53762
2022-05-18T04:47:55.5793838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53763
2022-05-18T04:47:56.7687160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:47:56.7687729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:47:56.7688539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:56.7689244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:47:56.7797481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:56.8701208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:58.0461742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4nr0z867
2022-05-18T04:47:58.0462373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4nr0z867/_remote_module_non_scriptable.py
2022-05-18T04:47:58.1451680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnl_w2gwi
2022-05-18T04:47:58.1452747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnl_w2gwi/_remote_module_non_scriptable.py
2022-05-18T04:47:58.3927082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:58.3928079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:58.4615773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:58.4616842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:47:58.7887515Z ok (4.829s)
2022-05-18T04:47:58.7887746Z 
2022-05-18T04:47:58.7888139Z ----------------------------------------------------------------------
2022-05-18T04:47:58.7888706Z Ran 1 test in 4.830s
2022-05-18T04:47:58.7888894Z 
2022-05-18T04:47:58.7888996Z OK
2022-05-18T04:47:58.7889114Z 
2022-05-18T04:47:58.7889251Z Generating XML reports...
2022-05-18T04:47:58.7929422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044753.xml
2022-05-18T04:48:00.2080394Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:48:00.2094968Z 
2022-05-18T04:48:00.2095255Z Running tests...
2022-05-18T04:48:00.2095707Z ----------------------------------------------------------------------
2022-05-18T04:48:01.7956054Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:01.8352480Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53907
2022-05-18T04:48:01.8466347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53908
2022-05-18T04:48:03.0247727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:03.0248289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:03.0249101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:03.0249780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:03.0255505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:03.0256260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:04.3576599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplp2ir0hx
2022-05-18T04:48:04.3577349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplp2ir0hx/_remote_module_non_scriptable.py
2022-05-18T04:48:04.3873862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3zonccy1
2022-05-18T04:48:04.3875265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3zonccy1/_remote_module_non_scriptable.py
2022-05-18T04:48:04.6251245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:04.6251840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:04.6947889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:04.6948559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:05.0551285Z ok (4.845s)
2022-05-18T04:48:05.0551731Z 
2022-05-18T04:48:05.0552334Z ----------------------------------------------------------------------
2022-05-18T04:48:05.0553164Z Ran 1 test in 4.846s
2022-05-18T04:48:05.0553530Z 
2022-05-18T04:48:05.0553692Z OK
2022-05-18T04:48:05.0554028Z 
2022-05-18T04:48:05.0555510Z Generating XML reports...
2022-05-18T04:48:05.0604734Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044800.xml
2022-05-18T04:48:06.4535938Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:48:06.4549488Z 
2022-05-18T04:48:06.4549883Z Running tests...
2022-05-18T04:48:06.4550399Z ----------------------------------------------------------------------
2022-05-18T04:48:07.9888740Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:08.0274501Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54052
2022-05-18T04:48:08.0383094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54053
2022-05-18T04:48:09.2378815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:09.2379390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:09.2380194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:09.2380875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:09.2386529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:09.2387066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:10.5437075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppizezx4u
2022-05-18T04:48:10.5438199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppizezx4u/_remote_module_non_scriptable.py
2022-05-18T04:48:10.5710308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpidgpfnbd
2022-05-18T04:48:10.5712593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpidgpfnbd/_remote_module_non_scriptable.py
2022-05-18T04:48:10.8087906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:10.8088487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:10.8764972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:10.8765536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:11.2464900Z ok (4.791s)
2022-05-18T04:48:11.2465126Z 
2022-05-18T04:48:11.2465623Z ----------------------------------------------------------------------
2022-05-18T04:48:11.2466159Z Ran 1 test in 4.792s
2022-05-18T04:48:11.2466338Z 
2022-05-18T04:48:11.2466417Z OK
2022-05-18T04:48:11.2466584Z 
2022-05-18T04:48:11.2466722Z Generating XML reports...
2022-05-18T04:48:11.2508705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044806.xml
2022-05-18T04:48:12.6757983Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:48:12.6773599Z 
2022-05-18T04:48:12.6773805Z Running tests...
2022-05-18T04:48:12.6774506Z ----------------------------------------------------------------------
2022-05-18T04:48:14.2580592Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:14.2978630Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54197
2022-05-18T04:48:14.3085044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54198
2022-05-18T04:48:15.4751550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:15.4752155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:15.4752955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:15.4753670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:15.4861254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:15.5765260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:16.7631094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphhmy6790
2022-05-18T04:48:16.7631986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphhmy6790/_remote_module_non_scriptable.py
2022-05-18T04:48:16.8587188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpaw6a_k
2022-05-18T04:48:16.8587837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpaw6a_k/_remote_module_non_scriptable.py
2022-05-18T04:48:17.1103747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:17.1104326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:17.1809921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:17.1810461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:17.5169507Z ok (4.839s)
2022-05-18T04:48:17.5169870Z 
2022-05-18T04:48:17.5170533Z ----------------------------------------------------------------------
2022-05-18T04:48:17.5171176Z Ran 1 test in 4.840s
2022-05-18T04:48:17.5171479Z 
2022-05-18T04:48:17.5171646Z OK
2022-05-18T04:48:17.5171896Z 
2022-05-18T04:48:17.5172149Z Generating XML reports...
2022-05-18T04:48:17.5213155Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044812.xml
2022-05-18T04:48:18.9360702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:48:18.9375966Z 
2022-05-18T04:48:18.9376231Z Running tests...
2022-05-18T04:48:18.9376664Z ----------------------------------------------------------------------
2022-05-18T04:48:20.5025706Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:20.5429103Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54342
2022-05-18T04:48:20.5536682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54343
2022-05-18T04:48:21.7514647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:21.7515212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:21.7516013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:21.7516700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:21.7523224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:21.7523721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:23.0836768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfai45vfz
2022-05-18T04:48:23.0838017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfai45vfz/_remote_module_non_scriptable.py
2022-05-18T04:48:23.0892719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfb5ovwy8
2022-05-18T04:48:23.0895584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfb5ovwy8/_remote_module_non_scriptable.py
2022-05-18T04:48:23.3371600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:23.3372151Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:23.4055624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:23.4056196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:23.7620952Z ok (4.824s)
2022-05-18T04:48:23.7621200Z 
2022-05-18T04:48:23.7621588Z ----------------------------------------------------------------------
2022-05-18T04:48:23.7621912Z Ran 1 test in 4.824s
2022-05-18T04:48:23.7622080Z 
2022-05-18T04:48:23.7622178Z OK
2022-05-18T04:48:23.7622317Z 
2022-05-18T04:48:23.7622452Z Generating XML reports...
2022-05-18T04:48:23.7664194Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044818.xml
2022-05-18T04:48:25.1767427Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:48:25.1782121Z 
2022-05-18T04:48:25.1782398Z Running tests...
2022-05-18T04:48:25.1782832Z ----------------------------------------------------------------------
2022-05-18T04:48:26.7572584Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:26.7970682Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54487
2022-05-18T04:48:26.8077967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54488
2022-05-18T04:48:27.9671393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:27.9671959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:27.9672745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:27.9673448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:27.9780749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:28.0685333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:29.2365433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzqlbc57d
2022-05-18T04:48:29.2366292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzqlbc57d/_remote_module_non_scriptable.py
2022-05-18T04:48:29.3500846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnifb6xg8
2022-05-18T04:48:29.3501868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnifb6xg8/_remote_module_non_scriptable.py
2022-05-18T04:48:29.5858085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:29.5858617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:29.6574350Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:29.6574928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:30.0159614Z ok (4.837s)
2022-05-18T04:48:30.0159846Z 
2022-05-18T04:48:30.0160215Z ----------------------------------------------------------------------
2022-05-18T04:48:30.0160581Z Ran 1 test in 4.838s
2022-05-18T04:48:30.0160750Z 
2022-05-18T04:48:30.0160855Z OK
2022-05-18T04:48:30.0160995Z 
2022-05-18T04:48:30.0161131Z Generating XML reports...
2022-05-18T04:48:30.0201505Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044825.xml
2022-05-18T04:48:31.4427899Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:48:31.4443260Z 
2022-05-18T04:48:31.4443769Z Running tests...
2022-05-18T04:48:31.4444277Z ----------------------------------------------------------------------
2022-05-18T04:48:33.0089410Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:33.0489235Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54632
2022-05-18T04:48:33.0596254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54633
2022-05-18T04:48:34.2575187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:34.2575754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:34.2576771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:34.2577498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:34.2586537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:34.2587034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:35.5543884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpek7o9k9d
2022-05-18T04:48:35.5544858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpek7o9k9d/_remote_module_non_scriptable.py
2022-05-18T04:48:35.5577885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_6pglqc
2022-05-18T04:48:35.5580917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_6pglqc/_remote_module_non_scriptable.py
2022-05-18T04:48:35.7909982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:35.7910551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:35.8574607Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:35.8575168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:36.2679356Z ok (4.823s)
2022-05-18T04:48:36.2679571Z 
2022-05-18T04:48:36.2679960Z ----------------------------------------------------------------------
2022-05-18T04:48:36.2680303Z Ran 1 test in 4.824s
2022-05-18T04:48:36.2680760Z 
2022-05-18T04:48:36.2680859Z OK
2022-05-18T04:48:36.2681001Z 
2022-05-18T04:48:36.2681139Z Generating XML reports...
2022-05-18T04:48:36.2722013Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044831.xml
2022-05-18T04:48:37.6840263Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:48:37.6855189Z 
2022-05-18T04:48:37.6855494Z Running tests...
2022-05-18T04:48:37.6855944Z ----------------------------------------------------------------------
2022-05-18T04:48:39.2627463Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:39.2782372Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.592s)
2022-05-18T04:48:39.2782970Z 
2022-05-18T04:48:39.2783254Z ----------------------------------------------------------------------
2022-05-18T04:48:39.2783592Z Ran 1 test in 1.593s
2022-05-18T04:48:39.2783762Z 
2022-05-18T04:48:39.2783872Z OK (skipped=1)
2022-05-18T04:48:39.2784012Z 
2022-05-18T04:48:39.2784152Z Generating XML reports...
2022-05-18T04:48:39.2818369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044837.xml
2022-05-18T04:48:40.6265694Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:48:40.6281978Z 
2022-05-18T04:48:40.6282424Z Running tests...
2022-05-18T04:48:40.6282910Z ----------------------------------------------------------------------
2022-05-18T04:48:42.1833252Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:42.2224103Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54811
2022-05-18T04:48:42.2328074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54812
2022-05-18T04:48:43.4182618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:43.4183368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:43.4184190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:43.4184896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:43.4290394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:43.5196617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:44.6896700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmii4q8y0
2022-05-18T04:48:44.6897632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmii4q8y0/_remote_module_non_scriptable.py
2022-05-18T04:48:44.7451223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusfbhbih
2022-05-18T04:48:44.7452417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusfbhbih/_remote_module_non_scriptable.py
2022-05-18T04:48:45.8417177Z ok (5.213s)
2022-05-18T04:48:45.8417413Z 
2022-05-18T04:48:45.8417809Z ----------------------------------------------------------------------
2022-05-18T04:48:45.8418138Z Ran 1 test in 5.214s
2022-05-18T04:48:45.8418305Z 
2022-05-18T04:48:45.8418402Z OK
2022-05-18T04:48:45.8418539Z 
2022-05-18T04:48:45.8418675Z Generating XML reports...
2022-05-18T04:48:45.8458663Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044840.xml
2022-05-18T04:48:47.2614402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:48:47.2628230Z 
2022-05-18T04:48:47.2628658Z Running tests...
2022-05-18T04:48:47.2629150Z ----------------------------------------------------------------------
2022-05-18T04:48:48.8367330Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:48.8762636Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54922
2022-05-18T04:48:48.8868513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54923
2022-05-18T04:48:50.0689195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:50.0689779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:50.0690548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:50.0691281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:50.0799172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:50.1703084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:51.4771694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnahhgsdb
2022-05-18T04:48:51.4772580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnahhgsdb/_remote_module_non_scriptable.py
2022-05-18T04:48:51.6000991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_5m5n_p
2022-05-18T04:48:51.6002270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_5m5n_p/_remote_module_non_scriptable.py
2022-05-18T04:48:51.6185865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:51.6186595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:51.6287981Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpx_5m5n_p'>
2022-05-18T04:48:51.6288462Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:48:51.6289738Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpnahhgsdb'>
2022-05-18T04:48:51.6290236Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:48:51.8950685Z ok (4.632s)
2022-05-18T04:48:51.8951065Z 
2022-05-18T04:48:51.8951655Z ----------------------------------------------------------------------
2022-05-18T04:48:51.8952036Z Ran 1 test in 4.632s
2022-05-18T04:48:51.8952205Z 
2022-05-18T04:48:51.8952308Z OK
2022-05-18T04:48:51.8952426Z 
2022-05-18T04:48:51.8952569Z Generating XML reports...
2022-05-18T04:48:51.8993513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044847.xml
2022-05-18T04:48:53.3244809Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:48:53.3259569Z 
2022-05-18T04:48:53.3260000Z Running tests...
2022-05-18T04:48:53.3260470Z ----------------------------------------------------------------------
2022-05-18T04:48:54.9151829Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:54.9551733Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55037
2022-05-18T04:48:54.9659312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55038
2022-05-18T04:48:56.1445122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:48:56.1445697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:48:56.1446493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:56.1447480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:48:56.1552919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:56.1653520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplj0368fa
2022-05-18T04:48:56.1656085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplj0368fa/_remote_module_non_scriptable.py
2022-05-18T04:48:56.2455975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:56.2557160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_n6o1v9
2022-05-18T04:48:56.2559588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_n6o1v9/_remote_module_non_scriptable.py
2022-05-18T04:48:56.2725848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:56.2726553Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:48:56.5713490Z ok (3.245s)
2022-05-18T04:48:56.5713681Z 
2022-05-18T04:48:56.5714087Z ----------------------------------------------------------------------
2022-05-18T04:48:56.5714422Z Ran 1 test in 3.245s
2022-05-18T04:48:56.5714590Z 
2022-05-18T04:48:56.5714687Z OK
2022-05-18T04:48:56.5714825Z 
2022-05-18T04:48:56.5714964Z Generating XML reports...
2022-05-18T04:48:56.5756830Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044853.xml
2022-05-18T04:48:57.9977481Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:48:57.9991689Z 
2022-05-18T04:48:57.9992160Z Running tests...
2022-05-18T04:48:57.9992682Z ----------------------------------------------------------------------
2022-05-18T04:48:59.5796216Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:59.6195190Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55180
2022-05-18T04:48:59.6302165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55181
2022-05-18T04:49:00.8044103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:00.8044694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:00.8045475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:00.8046183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:00.8151768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:00.9057845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:02.0855674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb65atzfp
2022-05-18T04:49:02.0856488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb65atzfp/_remote_module_non_scriptable.py
2022-05-18T04:49:02.1458623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wm3ffye
2022-05-18T04:49:02.1459810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wm3ffye/_remote_module_non_scriptable.py
2022-05-18T04:49:02.3329654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:02.3330208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:02.6382029Z ok (4.639s)
2022-05-18T04:49:02.6382218Z 
2022-05-18T04:49:02.6382610Z ----------------------------------------------------------------------
2022-05-18T04:49:02.6383202Z Ran 1 test in 4.639s
2022-05-18T04:49:02.6383378Z 
2022-05-18T04:49:02.6383475Z OK
2022-05-18T04:49:02.6383599Z 
2022-05-18T04:49:02.6383732Z Generating XML reports...
2022-05-18T04:49:02.6424596Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044857.xml
2022-05-18T04:49:04.0590154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:49:04.0604864Z 
2022-05-18T04:49:04.0605173Z Running tests...
2022-05-18T04:49:04.0605622Z ----------------------------------------------------------------------
2022-05-18T04:49:05.6475145Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:05.6862350Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55295
2022-05-18T04:49:05.6968990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55296
2022-05-18T04:49:06.9260010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:06.9260563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:06.9261389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:06.9262096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:06.9267637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:06.9268119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:06.9475514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:49:06.9476034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:49:06.9476748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:49:06.9477422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:49:06.9583935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:49:06.9584486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:49:06.9585200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:49:06.9585877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:49:08.2523797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_zu68dh
2022-05-18T04:49:08.2524798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_zu68dh/_remote_module_non_scriptable.py
2022-05-18T04:49:08.2629838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7l6nlo4
2022-05-18T04:49:08.2632593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7l6nlo4/_remote_module_non_scriptable.py
2022-05-18T04:49:08.6058123Z ok (4.545s)
2022-05-18T04:49:08.6058425Z 
2022-05-18T04:49:08.6059008Z ----------------------------------------------------------------------
2022-05-18T04:49:08.6059356Z Ran 1 test in 4.545s
2022-05-18T04:49:08.6059523Z 
2022-05-18T04:49:08.6059607Z OK
2022-05-18T04:49:08.6059744Z 
2022-05-18T04:49:08.6059875Z Generating XML reports...
2022-05-18T04:49:08.6099890Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044904.xml
2022-05-18T04:49:10.0099837Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:49:10.0114182Z 
2022-05-18T04:49:10.0114394Z Running tests...
2022-05-18T04:49:10.0115006Z ----------------------------------------------------------------------
2022-05-18T04:49:11.5725964Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:11.6122575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55418
2022-05-18T04:49:11.6229027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55419
2022-05-18T04:49:12.8062122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:12.8062697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:12.8063511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:12.8064199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:12.8069538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:12.8070035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:12.8178708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:49:12.8179240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:49:12.8179976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:49:12.8180680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:49:12.8288445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:49:12.8289571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:49:12.8290501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:49:12.8291434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:49:14.1069125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphl3q9oak
2022-05-18T04:49:14.1070003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphl3q9oak/_remote_module_non_scriptable.py
2022-05-18T04:49:14.1195573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppzg37v1s
2022-05-18T04:49:14.1198447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppzg37v1s/_remote_module_non_scriptable.py
2022-05-18T04:49:24.4470818Z ok (14.435s)
2022-05-18T04:49:24.4471109Z 
2022-05-18T04:49:24.4471878Z ----------------------------------------------------------------------
2022-05-18T04:49:24.4472563Z Ran 1 test in 14.436s
2022-05-18T04:49:24.4472717Z 
2022-05-18T04:49:24.4472819Z OK
2022-05-18T04:49:24.4475835Z 
2022-05-18T04:49:24.4476161Z Generating XML reports...
2022-05-18T04:49:24.4513197Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044910.xml
2022-05-18T04:49:25.8743429Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:49:25.8758164Z 
2022-05-18T04:49:25.8758485Z Running tests...
2022-05-18T04:49:25.8759263Z ----------------------------------------------------------------------
2022-05-18T04:49:27.4564130Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:27.4962686Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55541
2022-05-18T04:49:27.5069447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55542
2022-05-18T04:49:28.7067318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:28.7067892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:28.7068720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:28.7069408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:28.7175333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:28.8081184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:29.9726578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptebyy71g
2022-05-18T04:49:29.9727983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptebyy71g/_remote_module_non_scriptable.py
2022-05-18T04:49:30.0657617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpblwd36a5
2022-05-18T04:49:30.0659217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpblwd36a5/_remote_module_non_scriptable.py
2022-05-18T04:49:31.2161186Z ok (5.340s)
2022-05-18T04:49:31.2161554Z 
2022-05-18T04:49:31.2162036Z ----------------------------------------------------------------------
2022-05-18T04:49:31.2162392Z Ran 1 test in 5.340s
2022-05-18T04:49:31.2162541Z 
2022-05-18T04:49:31.2162637Z OK
2022-05-18T04:49:31.2165383Z 
2022-05-18T04:49:31.2165754Z Generating XML reports...
2022-05-18T04:49:31.2203976Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044925.xml
2022-05-18T04:49:32.6333637Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:49:32.6347638Z 
2022-05-18T04:49:32.6347942Z Running tests...
2022-05-18T04:49:32.6348381Z ----------------------------------------------------------------------
2022-05-18T04:49:34.1952034Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:34.2343471Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55656
2022-05-18T04:49:34.2448664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55657
2022-05-18T04:49:35.4435182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:35.4435745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:35.4436537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:35.4437227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:35.4542484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:35.5450091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:36.6922084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkt9jxilz
2022-05-18T04:49:36.6923099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkt9jxilz/_remote_module_non_scriptable.py
2022-05-18T04:49:36.8035496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbua786n0
2022-05-18T04:49:36.8036591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbua786n0/_remote_module_non_scriptable.py
2022-05-18T04:49:37.8539858Z ok (5.219s)
2022-05-18T04:49:37.8540222Z 
2022-05-18T04:49:37.8540991Z ----------------------------------------------------------------------
2022-05-18T04:49:37.8541853Z Ran 1 test in 5.219s
2022-05-18T04:49:37.8542005Z 
2022-05-18T04:49:37.8542101Z OK
2022-05-18T04:49:37.8542235Z 
2022-05-18T04:49:37.8542368Z Generating XML reports...
2022-05-18T04:49:37.8582612Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044932.xml
2022-05-18T04:49:39.2772217Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:49:39.2786525Z 
2022-05-18T04:49:39.2786835Z Running tests...
2022-05-18T04:49:39.2787515Z ----------------------------------------------------------------------
2022-05-18T04:49:40.8680798Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:40.9089170Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55771
2022-05-18T04:49:40.9195286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55772
2022-05-18T04:49:42.1098019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:42.1098582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:42.1099375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:42.1100084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:42.1105922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:42.1106420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:43.3832242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43uo08cb
2022-05-18T04:49:43.3833384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43uo08cb/_remote_module_non_scriptable.py
2022-05-18T04:49:43.4030726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpliru_b_w
2022-05-18T04:49:43.4033359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpliru_b_w/_remote_module_non_scriptable.py
2022-05-18T04:49:43.9276286Z ok (4.649s)
2022-05-18T04:49:43.9276517Z 
2022-05-18T04:49:43.9277150Z ----------------------------------------------------------------------
2022-05-18T04:49:43.9277518Z Ran 1 test in 4.649s
2022-05-18T04:49:43.9277683Z 
2022-05-18T04:49:43.9277780Z OK
2022-05-18T04:49:43.9277920Z 
2022-05-18T04:49:43.9278307Z Generating XML reports...
2022-05-18T04:49:43.9319904Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044939.xml
2022-05-18T04:49:45.3361334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:49:45.3375749Z 
2022-05-18T04:49:45.3376182Z Running tests...
2022-05-18T04:49:45.3376652Z ----------------------------------------------------------------------
2022-05-18T04:49:46.8752337Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:46.9142450Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55882
2022-05-18T04:49:46.9250623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55883
2022-05-18T04:49:48.0872472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:48.0873033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:48.0873806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:48.0874511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:48.0982106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:48.1886944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:49.3357558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpymqq1a0q
2022-05-18T04:49:49.3359129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpymqq1a0q/_remote_module_non_scriptable.py
2022-05-18T04:49:49.4459138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvb5qos53
2022-05-18T04:49:49.4460259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvb5qos53/_remote_module_non_scriptable.py
2022-05-18T04:49:49.6230570Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:49:49.6289406Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:49:49.6387133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:49.6387648Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:49:49.9328651Z ok (4.595s)
2022-05-18T04:49:49.9329053Z 
2022-05-18T04:49:49.9329806Z ----------------------------------------------------------------------
2022-05-18T04:49:49.9330411Z Ran 1 test in 4.595s
2022-05-18T04:49:49.9330585Z 
2022-05-18T04:49:49.9330683Z OK
2022-05-18T04:49:49.9330827Z 
2022-05-18T04:49:49.9330965Z Generating XML reports...
2022-05-18T04:49:49.9372616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044945.xml
2022-05-18T04:49:51.3503668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:49:51.3518217Z 
2022-05-18T04:49:51.3518572Z Running tests...
2022-05-18T04:49:51.3519245Z ----------------------------------------------------------------------
2022-05-18T04:49:52.9333859Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:52.9740382Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55997
2022-05-18T04:49:52.9845890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55998
2022-05-18T04:49:54.1680554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:49:54.1681108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:49:54.1681910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:54.1682613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:49:54.1688494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:54.1689377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:55.4444416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6o8xhuj
2022-05-18T04:49:55.4445650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6o8xhuj/_remote_module_non_scriptable.py
2022-05-18T04:49:55.4685349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0uiwc8r8
2022-05-18T04:49:55.4687695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0uiwc8r8/_remote_module_non_scriptable.py
2022-05-18T04:49:55.4854527Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:49:55.4855308Z   warnings.warn(
2022-05-18T04:49:55.4856353Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:49:55.4857094Z   warnings.warn(
2022-05-18T04:49:55.9926610Z ok (4.640s)
2022-05-18T04:49:55.9926856Z 
2022-05-18T04:49:55.9927642Z ----------------------------------------------------------------------
2022-05-18T04:49:55.9928016Z Ran 1 test in 4.641s
2022-05-18T04:49:55.9928190Z 
2022-05-18T04:49:55.9928289Z OK
2022-05-18T04:49:55.9928434Z 
2022-05-18T04:49:55.9928555Z Generating XML reports...
2022-05-18T04:49:55.9968495Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044951.xml
2022-05-18T04:49:57.4155626Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:49:57.4171617Z 
2022-05-18T04:49:57.4171899Z Running tests...
2022-05-18T04:49:57.4172363Z ----------------------------------------------------------------------
2022-05-18T04:49:58.9898555Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:59.0048432Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.587s)
2022-05-18T04:49:59.0049192Z 
2022-05-18T04:49:59.0049577Z ----------------------------------------------------------------------
2022-05-18T04:49:59.0050079Z Ran 1 test in 1.588s
2022-05-18T04:49:59.0050250Z 
2022-05-18T04:49:59.0050364Z OK (skipped=1)
2022-05-18T04:49:59.0050523Z 
2022-05-18T04:49:59.0050652Z Generating XML reports...
2022-05-18T04:49:59.0080594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044957.xml
2022-05-18T04:50:00.3724608Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:50:00.3739063Z 
2022-05-18T04:50:00.3739311Z Running tests...
2022-05-18T04:50:00.3739786Z ----------------------------------------------------------------------
2022-05-18T04:50:01.9604859Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:02.0013431Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56146
2022-05-18T04:50:02.0121529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56147
2022-05-18T04:50:03.2070973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:50:03.2071544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:50:03.2072361Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:03.2073357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:03.2078170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:03.2078959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:04.5008210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3qqvdji
2022-05-18T04:50:04.5009563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3qqvdji/_remote_module_non_scriptable.py
2022-05-18T04:50:04.5160259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitgy4gzm
2022-05-18T04:50:04.5163141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitgy4gzm/_remote_module_non_scriptable.py
2022-05-18T04:50:06.1632873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:06.1633435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:06.7067380Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:50:06.7124148Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:50:07.9255463Z ok (7.551s)
2022-05-18T04:50:07.9255910Z 
2022-05-18T04:50:07.9256329Z ----------------------------------------------------------------------
2022-05-18T04:50:07.9256672Z Ran 1 test in 7.552s
2022-05-18T04:50:07.9256833Z 
2022-05-18T04:50:07.9256930Z OK
2022-05-18T04:50:07.9257067Z 
2022-05-18T04:50:07.9257206Z Generating XML reports...
2022-05-18T04:50:07.9297502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045000.xml
2022-05-18T04:50:09.3341415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:50:09.3355256Z 
2022-05-18T04:50:09.3355527Z Running tests...
2022-05-18T04:50:09.3355998Z ----------------------------------------------------------------------
2022-05-18T04:50:10.8772732Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:10.9165280Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56265
2022-05-18T04:50:10.9271771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56266
2022-05-18T04:50:12.1194602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:50:12.1195456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:50:12.1196394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:12.1197226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:12.1202483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:12.1203551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:13.4247686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn73lisjm
2022-05-18T04:50:13.4248864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn73lisjm/_remote_module_non_scriptable.py
2022-05-18T04:50:13.4254801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppy0awttb
2022-05-18T04:50:13.4258031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppy0awttb/_remote_module_non_scriptable.py
2022-05-18T04:50:13.7347429Z ok (4.399s)
2022-05-18T04:50:13.7347679Z 
2022-05-18T04:50:13.7348301Z ----------------------------------------------------------------------
2022-05-18T04:50:13.7348651Z Ran 1 test in 4.399s
2022-05-18T04:50:13.7348818Z 
2022-05-18T04:50:13.7348916Z OK
2022-05-18T04:50:13.7349058Z 
2022-05-18T04:50:13.7349194Z Generating XML reports...
2022-05-18T04:50:13.7389529Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045009.xml
2022-05-18T04:50:15.1722306Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:50:15.1737720Z 
2022-05-18T04:50:15.1738209Z Running tests...
2022-05-18T04:50:15.1738721Z ----------------------------------------------------------------------
2022-05-18T04:50:16.7709137Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:16.8101619Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56376
2022-05-18T04:50:16.8209160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56377
2022-05-18T04:50:17.9773104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:50:17.9773687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:50:17.9774692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:17.9775579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:17.9781458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:17.9782404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:19.2538607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_9m89d3
2022-05-18T04:50:19.2539426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_9m89d3/_remote_module_non_scriptable.py
2022-05-18T04:50:19.2901546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7ddu9ea
2022-05-18T04:50:19.2903166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7ddu9ea/_remote_module_non_scriptable.py
2022-05-18T04:50:19.3069678Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:50:19.3070453Z   warnings.warn(
2022-05-18T04:50:19.3071510Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:50:19.3072446Z   warnings.warn(
2022-05-18T04:50:19.4814551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:19.4815111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:19.4884260Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:50:19.4885839Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:50:19.5029133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:19.5029643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:19.5115405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:19.5115889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:19.8289649Z ok (4.655s)
2022-05-18T04:50:19.8289882Z 
2022-05-18T04:50:19.8290535Z ----------------------------------------------------------------------
2022-05-18T04:50:19.8290931Z Ran 1 test in 4.655s
2022-05-18T04:50:19.8291080Z 
2022-05-18T04:50:19.8291188Z OK
2022-05-18T04:50:19.8291324Z 
2022-05-18T04:50:19.8291460Z Generating XML reports...
2022-05-18T04:50:19.8332731Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045015.xml
2022-05-18T04:50:21.2647684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:50:21.2662286Z 
2022-05-18T04:50:21.2662766Z Running tests...
2022-05-18T04:50:21.2663257Z ----------------------------------------------------------------------
2022-05-18T04:50:22.8693807Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:22.9084148Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56491
2022-05-18T04:50:22.9189684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56492
2022-05-18T04:50:24.0673629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:50:24.0674262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:50:24.0675077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:24.0675793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:24.0782399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:24.1687288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:25.3654915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkq7p8448
2022-05-18T04:50:25.3656113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkq7p8448/_remote_module_non_scriptable.py
2022-05-18T04:50:25.4114622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfi4akum
2022-05-18T04:50:25.4115920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfi4akum/_remote_module_non_scriptable.py
2022-05-18T04:50:25.4279628Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:50:25.4280495Z   warnings.warn(
2022-05-18T04:50:25.4281537Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:50:25.4282284Z   warnings.warn(
2022-05-18T04:50:25.6027534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:25.6028091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:25.9268877Z ok (4.660s)
2022-05-18T04:50:25.9269202Z 
2022-05-18T04:50:25.9269600Z ----------------------------------------------------------------------
2022-05-18T04:50:25.9269949Z Ran 1 test in 4.661s
2022-05-18T04:50:25.9270116Z 
2022-05-18T04:50:25.9270213Z OK
2022-05-18T04:50:25.9270331Z 
2022-05-18T04:50:25.9270465Z Generating XML reports...
2022-05-18T04:50:25.9310944Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045021.xml
2022-05-18T04:50:27.3511834Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:50:27.3533997Z 
2022-05-18T04:50:27.3534311Z Running tests...
2022-05-18T04:50:27.3534994Z ----------------------------------------------------------------------
2022-05-18T04:50:28.9504936Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:28.9660344Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.612s)
2022-05-18T04:50:28.9661044Z 
2022-05-18T04:50:28.9661321Z ----------------------------------------------------------------------
2022-05-18T04:50:28.9661656Z Ran 1 test in 1.613s
2022-05-18T04:50:28.9661823Z 
2022-05-18T04:50:28.9661937Z OK (skipped=1)
2022-05-18T04:50:28.9662095Z 
2022-05-18T04:50:28.9662223Z Generating XML reports...
2022-05-18T04:50:28.9694459Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045027.xml
2022-05-18T04:50:30.3380697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:50:30.3395062Z 
2022-05-18T04:50:30.3395209Z Running tests...
2022-05-18T04:50:30.3395948Z ----------------------------------------------------------------------
2022-05-18T04:50:31.8733881Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:31.9124258Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56640
2022-05-18T04:50:31.9233008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56641
2022-05-18T04:50:33.1069934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:50:33.1070505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:50:33.1071602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:33.1072330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:33.1177324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:33.2083962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:34.3906175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mc64dil
2022-05-18T04:50:34.3907006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mc64dil/_remote_module_non_scriptable.py
2022-05-18T04:50:34.4671312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6gwjvua
2022-05-18T04:50:34.4672242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6gwjvua/_remote_module_non_scriptable.py
2022-05-18T04:50:34.7704242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:34.7704792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:35.8329785Z ok (5.493s)
2022-05-18T04:50:35.8330037Z 
2022-05-18T04:50:35.8330444Z ----------------------------------------------------------------------
2022-05-18T04:50:35.8330801Z Ran 1 test in 5.493s
2022-05-18T04:50:35.8330971Z 
2022-05-18T04:50:35.8331068Z OK
2022-05-18T04:50:35.8331207Z 
2022-05-18T04:50:35.8331327Z Generating XML reports...
2022-05-18T04:50:35.8372724Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045030.xml
2022-05-18T04:50:37.2358580Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:50:37.2372235Z 
2022-05-18T04:50:37.2372565Z Running tests...
2022-05-18T04:50:37.2373057Z ----------------------------------------------------------------------
2022-05-18T04:50:38.7855997Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:38.8246898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56759
2022-05-18T04:50:38.8351215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56760
2022-05-18T04:50:40.0322017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:50:40.0322606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:50:40.0323424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:40.0324131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:40.0432446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:40.1336636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:41.2870678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82sq0rev
2022-05-18T04:50:41.2871629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82sq0rev/_remote_module_non_scriptable.py
2022-05-18T04:50:41.3868231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahhbk1g4
2022-05-18T04:50:41.3869367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahhbk1g4/_remote_module_non_scriptable.py
2022-05-18T04:50:41.6425794Z ok (4.405s)
2022-05-18T04:50:41.6426054Z 
2022-05-18T04:50:41.6426630Z ----------------------------------------------------------------------
2022-05-18T04:50:41.6426988Z Ran 1 test in 4.405s
2022-05-18T04:50:41.6427162Z 
2022-05-18T04:50:41.6427240Z OK
2022-05-18T04:50:41.6427784Z 
2022-05-18T04:50:41.6427944Z Generating XML reports...
2022-05-18T04:50:41.6469019Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045037.xml
2022-05-18T04:50:43.0757034Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:50:43.0772644Z 
2022-05-18T04:50:43.0772881Z Running tests...
2022-05-18T04:50:43.0773311Z ----------------------------------------------------------------------
2022-05-18T04:50:44.6661992Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:44.7060508Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56870
2022-05-18T04:50:44.7169194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56871
2022-05-18T04:50:45.9214880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:50:45.9215478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:50:45.9216277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:45.9216999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:45.9222882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:45.9223401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:47.1922490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjs9rpc74
2022-05-18T04:50:47.1923427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjs9rpc74/_remote_module_non_scriptable.py
2022-05-18T04:50:47.2261300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqboqh_je
2022-05-18T04:50:47.2263445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqboqh_je/_remote_module_non_scriptable.py
2022-05-18T04:50:47.5246033Z ok (4.447s)
2022-05-18T04:50:47.5246268Z 
2022-05-18T04:50:47.5246661Z ----------------------------------------------------------------------
2022-05-18T04:50:47.5247272Z Ran 1 test in 4.447s
2022-05-18T04:50:47.5247439Z 
2022-05-18T04:50:47.5247546Z OK
2022-05-18T04:50:47.5247688Z 
2022-05-18T04:50:47.5247831Z Generating XML reports...
2022-05-18T04:50:47.5288299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045043.xml
2022-05-18T04:50:48.9450762Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:50:48.9464804Z 
2022-05-18T04:50:48.9465255Z Running tests...
2022-05-18T04:50:48.9465763Z ----------------------------------------------------------------------
2022-05-18T04:50:50.5093841Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:50.5494843Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56981
2022-05-18T04:50:50.5601941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56982
2022-05-18T04:50:51.7624899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:50:51.7625652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:50:51.7626456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:51.7627161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:50:51.7633051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:51.7633921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:53.0319923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdnq12vlw
2022-05-18T04:50:53.0321193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdnq12vlw/_remote_module_non_scriptable.py
2022-05-18T04:50:53.0595906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwoox_62i
2022-05-18T04:50:53.0598446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwoox_62i/_remote_module_non_scriptable.py
2022-05-18T04:50:53.2465617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:53.2466726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:50:53.5680141Z ok (4.621s)
2022-05-18T04:50:53.5680510Z 
2022-05-18T04:50:53.5681283Z ----------------------------------------------------------------------
2022-05-18T04:50:53.5681943Z Ran 1 test in 4.621s
2022-05-18T04:50:53.5682114Z 
2022-05-18T04:50:53.5682211Z OK
2022-05-18T04:50:53.5682348Z 
2022-05-18T04:50:53.5684279Z Generating XML reports...
2022-05-18T04:50:53.5721424Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045048.xml
2022-05-18T04:50:54.9593510Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:50:54.9607502Z 
2022-05-18T04:50:54.9607953Z Running tests...
2022-05-18T04:50:54.9608460Z ----------------------------------------------------------------------
2022-05-18T04:50:56.5061644Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:56.5211330Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.560s)
2022-05-18T04:50:56.5211923Z 
2022-05-18T04:50:56.5212204Z ----------------------------------------------------------------------
2022-05-18T04:50:56.5212530Z Ran 1 test in 1.560s
2022-05-18T04:50:56.5212697Z 
2022-05-18T04:50:56.5212808Z OK (skipped=1)
2022-05-18T04:50:56.5212968Z 
2022-05-18T04:50:56.5213318Z Generating XML reports...
2022-05-18T04:50:56.5244767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045054.xml
2022-05-18T04:50:57.8954133Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:50:57.8969729Z 
2022-05-18T04:50:57.8970174Z Running tests...
2022-05-18T04:50:57.8970604Z ----------------------------------------------------------------------
2022-05-18T04:50:59.4633466Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:59.5036173Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57130
2022-05-18T04:50:59.5144706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57131
2022-05-18T04:51:00.6920946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:00.6921526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:00.6922321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:00.6923028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:00.7028777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:00.7935289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:01.9678531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptzvqa3ml
2022-05-18T04:51:01.9679239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptzvqa3ml/_remote_module_non_scriptable.py
2022-05-18T04:51:02.0502924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnizyeypx
2022-05-18T04:51:02.0504022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnizyeypx/_remote_module_non_scriptable.py
2022-05-18T04:51:02.2465773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:51:02.2808539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:51:02.2809097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:51:02.2809601Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:51:02.2966532Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmptzvqa3ml'>
2022-05-18T04:51:02.2967035Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T04:51:02.6224730Z ok (4.725s)
2022-05-18T04:51:02.6224916Z 
2022-05-18T04:51:02.6225304Z ----------------------------------------------------------------------
2022-05-18T04:51:02.6225653Z Ran 1 test in 4.726s
2022-05-18T04:51:02.6225823Z 
2022-05-18T04:51:02.6225920Z OK
2022-05-18T04:51:02.6226058Z 
2022-05-18T04:51:02.6226191Z Generating XML reports...
2022-05-18T04:51:02.6267577Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045057.xml
2022-05-18T04:51:04.0280474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:51:04.0294653Z 
2022-05-18T04:51:04.0294930Z Running tests...
2022-05-18T04:51:04.0295373Z ----------------------------------------------------------------------
2022-05-18T04:51:05.5939316Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:05.6335782Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57245
2022-05-18T04:51:05.6442367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57246
2022-05-18T04:51:06.8236446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:06.8237051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:06.8237827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:06.8238794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:06.8343545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:06.9250280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:08.0774365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphrxl5t4z
2022-05-18T04:51:08.0775015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphrxl5t4z/_remote_module_non_scriptable.py
2022-05-18T04:51:08.1651447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexzcp18n
2022-05-18T04:51:08.1652472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexzcp18n/_remote_module_non_scriptable.py
2022-05-18T04:51:08.6522835Z ok (4.622s)
2022-05-18T04:51:08.6523023Z 
2022-05-18T04:51:08.6523416Z ----------------------------------------------------------------------
2022-05-18T04:51:08.6523750Z Ran 1 test in 4.623s
2022-05-18T04:51:08.6523919Z 
2022-05-18T04:51:08.6524018Z OK
2022-05-18T04:51:08.6524156Z 
2022-05-18T04:51:08.6524289Z Generating XML reports...
2022-05-18T04:51:08.6566132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045104.xml
2022-05-18T04:51:10.0817478Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:51:10.0831602Z 
2022-05-18T04:51:10.0832255Z Running tests...
2022-05-18T04:51:10.0832768Z ----------------------------------------------------------------------
2022-05-18T04:51:11.6563888Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:11.6962884Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57360
2022-05-18T04:51:11.7068991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57361
2022-05-18T04:51:12.9034791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:12.9035630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:12.9036441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:12.9037142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:12.9043280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:12.9044126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:12.9151212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:51:12.9151983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:51:12.9152659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:51:12.9153351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:51:13.1117524Z ok (3.028s)
2022-05-18T04:51:13.1118455Z 
2022-05-18T04:51:13.1119225Z ----------------------------------------------------------------------
2022-05-18T04:51:13.1119604Z Ran 1 test in 3.029s
2022-05-18T04:51:13.1119776Z 
2022-05-18T04:51:13.1120088Z OK
2022-05-18T04:51:13.1120249Z 
2022-05-18T04:51:13.1120378Z Generating XML reports...
2022-05-18T04:51:13.1161105Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045110.xml
2022-05-18T04:51:14.5131629Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:51:14.5145520Z 
2022-05-18T04:51:14.5145960Z Running tests...
2022-05-18T04:51:14.5146460Z ----------------------------------------------------------------------
2022-05-18T04:51:16.0900896Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:16.1292417Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57475
2022-05-18T04:51:16.1398077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57476
2022-05-18T04:51:17.3541522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:17.3542119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:17.3542921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:17.3543605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:17.3650852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:17.4553372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:17.4665927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:51:17.4666453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:51:17.4667146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:51:17.4667848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:51:17.7453222Z ok (3.230s)
2022-05-18T04:51:17.7453455Z 
2022-05-18T04:51:17.7453852Z ----------------------------------------------------------------------
2022-05-18T04:51:17.7454198Z Ran 1 test in 3.231s
2022-05-18T04:51:17.7454349Z 
2022-05-18T04:51:17.7454445Z OK
2022-05-18T04:51:17.7454587Z 
2022-05-18T04:51:17.7454722Z Generating XML reports...
2022-05-18T04:51:17.7496134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045114.xml
2022-05-18T04:51:19.1694324Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:51:19.1708902Z 
2022-05-18T04:51:19.1709244Z Running tests...
2022-05-18T04:51:19.1709701Z ----------------------------------------------------------------------
2022-05-18T04:51:20.7476481Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:20.7866842Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57590
2022-05-18T04:51:20.7972566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57591
2022-05-18T04:51:21.9728553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:21.9729136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:21.9729915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:21.9730638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:21.9735950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:21.9736467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:23.2632074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6a_28ug
2022-05-18T04:51:23.2632988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6a_28ug/_remote_module_non_scriptable.py
2022-05-18T04:51:23.2638903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5ysu1z7
2022-05-18T04:51:23.2642477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5ysu1z7/_remote_module_non_scriptable.py
2022-05-18T04:51:23.4600388Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:51:23.4600953Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:51:23.4694449Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:51:23.4696037Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:51:23.8054161Z ok (4.634s)
2022-05-18T04:51:23.8054414Z 
2022-05-18T04:51:23.8054819Z ----------------------------------------------------------------------
2022-05-18T04:51:23.8055148Z Ran 1 test in 4.634s
2022-05-18T04:51:23.8055316Z 
2022-05-18T04:51:23.8055412Z OK
2022-05-18T04:51:23.8055550Z 
2022-05-18T04:51:23.8055687Z Generating XML reports...
2022-05-18T04:51:23.8096956Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045119.xml
2022-05-18T04:51:25.2079329Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:51:25.2093464Z 
2022-05-18T04:51:25.2093765Z Running tests...
2022-05-18T04:51:25.2094206Z ----------------------------------------------------------------------
2022-05-18T04:51:26.7598699Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:26.7990235Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57705
2022-05-18T04:51:26.8095552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57706
2022-05-18T04:51:28.0131669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:28.0132257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:28.0133048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:28.0133734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:28.0242170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:28.1146867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:29.2954492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqdsslqs
2022-05-18T04:51:29.2955880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqdsslqs/_remote_module_non_scriptable.py
2022-05-18T04:51:29.3881901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwk68r0vk
2022-05-18T04:51:29.3883265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwk68r0vk/_remote_module_non_scriptable.py
2022-05-18T04:51:29.5720554Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:51:29.5942960Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:51:29.5943484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:51:29.9179781Z ok (4.708s)
2022-05-18T04:51:29.9180004Z 
2022-05-18T04:51:29.9180394Z ----------------------------------------------------------------------
2022-05-18T04:51:29.9180724Z Ran 1 test in 4.709s
2022-05-18T04:51:29.9180895Z 
2022-05-18T04:51:29.9180999Z OK
2022-05-18T04:51:29.9181139Z 
2022-05-18T04:51:29.9181280Z Generating XML reports...
2022-05-18T04:51:29.9222012Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045125.xml
2022-05-18T04:51:31.3135045Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:51:31.3148914Z 
2022-05-18T04:51:31.3149301Z Running tests...
2022-05-18T04:51:31.3149789Z ----------------------------------------------------------------------
2022-05-18T04:51:32.8442328Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:32.8831736Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57820
2022-05-18T04:51:32.8937795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57821
2022-05-18T04:51:34.0673427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:34.0673998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:34.0674808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:34.0675533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:34.0681888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:34.0682398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:34.2989814Z ok (2.984s)
2022-05-18T04:51:34.2990112Z 
2022-05-18T04:51:34.2990666Z ----------------------------------------------------------------------
2022-05-18T04:51:34.2991016Z Ran 1 test in 2.984s
2022-05-18T04:51:34.2991184Z 
2022-05-18T04:51:34.2991262Z OK
2022-05-18T04:51:34.2991398Z 
2022-05-18T04:51:34.2991536Z Generating XML reports...
2022-05-18T04:51:34.3032471Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045131.xml
2022-05-18T04:51:35.7010099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:51:35.7024007Z 
2022-05-18T04:51:35.7024415Z Running tests...
2022-05-18T04:51:35.7024895Z ----------------------------------------------------------------------
2022-05-18T04:51:37.2836949Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:37.3235608Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57929
2022-05-18T04:51:37.3343103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57930
2022-05-18T04:51:38.5115458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:38.5116024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:38.5116826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:38.5117533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:38.5225121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:38.6126974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:38.8395087Z ok (3.137s)
2022-05-18T04:51:38.8395309Z 
2022-05-18T04:51:38.8395683Z ----------------------------------------------------------------------
2022-05-18T04:51:38.8396026Z Ran 1 test in 3.137s
2022-05-18T04:51:38.8397473Z 
2022-05-18T04:51:38.8397991Z OK
2022-05-18T04:51:38.8398494Z 
2022-05-18T04:51:38.8398646Z Generating XML reports...
2022-05-18T04:51:38.8437353Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045135.xml
2022-05-18T04:51:40.2432187Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:51:40.2446472Z 
2022-05-18T04:51:40.2446788Z Running tests...
2022-05-18T04:51:40.2447453Z ----------------------------------------------------------------------
2022-05-18T04:51:41.8195392Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:41.8593118Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58042
2022-05-18T04:51:41.8701209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58043
2022-05-18T04:51:43.0667234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:43.0667807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:43.0668610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:43.0669312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:43.0674938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:43.0675712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:43.2751638Z ok (3.030s)
2022-05-18T04:51:43.2751879Z 
2022-05-18T04:51:43.2752297Z ----------------------------------------------------------------------
2022-05-18T04:51:43.2752658Z Ran 1 test in 3.030s
2022-05-18T04:51:43.2752830Z 
2022-05-18T04:51:43.2752927Z OK
2022-05-18T04:51:43.2755186Z 
2022-05-18T04:51:43.2755752Z Generating XML reports...
2022-05-18T04:51:43.2794438Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045140.xml
2022-05-18T04:51:44.6754222Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:51:44.6768819Z 
2022-05-18T04:51:44.6769352Z Running tests...
2022-05-18T04:51:44.6769879Z ----------------------------------------------------------------------
2022-05-18T04:51:44.6789422Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T04:51:44.6789729Z 
2022-05-18T04:51:44.6790062Z ----------------------------------------------------------------------
2022-05-18T04:51:44.6790734Z Ran 1 test in 0.002s
2022-05-18T04:51:44.6790930Z 
2022-05-18T04:51:44.6791042Z OK (skipped=1)
2022-05-18T04:51:44.6791200Z 
2022-05-18T04:51:44.6791329Z Generating XML reports...
2022-05-18T04:51:44.6826276Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045144.xml
2022-05-18T04:51:45.9340428Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:51:45.9354827Z 
2022-05-18T04:51:45.9354990Z Running tests...
2022-05-18T04:51:45.9355445Z ----------------------------------------------------------------------
2022-05-18T04:51:47.5172729Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:47.5569672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58184
2022-05-18T04:51:47.5676720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58185
2022-05-18T04:51:48.7174057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:48.7174621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:48.7175415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:48.7176110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:48.7181800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:48.7182540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:48.7391635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:51:48.7392170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:51:48.7392863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:51:48.7393555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:51:49.0728484Z ok (3.137s)
2022-05-18T04:51:49.0728672Z 
2022-05-18T04:51:49.0729062Z ----------------------------------------------------------------------
2022-05-18T04:51:49.0729398Z Ran 1 test in 3.137s
2022-05-18T04:51:49.0729570Z 
2022-05-18T04:51:49.0729666Z OK
2022-05-18T04:51:49.0729801Z 
2022-05-18T04:51:49.0729929Z Generating XML reports...
2022-05-18T04:51:49.0770293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045145.xml
2022-05-18T04:51:50.4970630Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:51:50.4985178Z 
2022-05-18T04:51:50.4985565Z Running tests...
2022-05-18T04:51:50.4986005Z ----------------------------------------------------------------------
2022-05-18T04:51:52.0656924Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:52.1049221Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58303
2022-05-18T04:51:52.1155617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58304
2022-05-18T04:51:53.2527404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:53.2527979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:53.2528788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:53.2529483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:53.2535385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:53.2535896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:53.4205325Z skip: Skipped due to small world size. (2.922s)
2022-05-18T04:51:53.4205903Z 
2022-05-18T04:51:53.4206579Z ----------------------------------------------------------------------
2022-05-18T04:51:53.4206938Z Ran 1 test in 2.922s
2022-05-18T04:51:53.4207100Z 
2022-05-18T04:51:53.4207211Z OK (skipped=1)
2022-05-18T04:51:53.4207349Z 
2022-05-18T04:51:53.4207479Z Generating XML reports...
2022-05-18T04:51:53.4247656Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045150.xml
2022-05-18T04:51:54.8460097Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:51:54.8474070Z 
2022-05-18T04:51:54.8474226Z Running tests...
2022-05-18T04:51:54.8474719Z ----------------------------------------------------------------------
2022-05-18T04:51:56.4229740Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:56.4621026Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58412
2022-05-18T04:51:56.4728083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58413
2022-05-18T04:51:57.6766928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:51:57.6767553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:51:57.6768618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:57.6769327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:51:57.6875535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:57.7778461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:57.9781457Z ok (3.130s)
2022-05-18T04:51:57.9781685Z 
2022-05-18T04:51:57.9782088Z ----------------------------------------------------------------------
2022-05-18T04:51:57.9782413Z Ran 1 test in 3.131s
2022-05-18T04:51:57.9782594Z 
2022-05-18T04:51:57.9782704Z OK
2022-05-18T04:51:57.9782844Z 
2022-05-18T04:51:57.9782981Z Generating XML reports...
2022-05-18T04:51:57.9824926Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045154.xml
2022-05-18T04:51:59.3844196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:51:59.3858385Z 
2022-05-18T04:51:59.3858845Z Running tests...
2022-05-18T04:51:59.3859341Z ----------------------------------------------------------------------
2022-05-18T04:52:00.9537523Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:00.9937971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58521
2022-05-18T04:52:01.0045169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58522
2022-05-18T04:52:02.2945891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:02.2946452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:02.2947258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:02.2947964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:02.3054133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:02.3956327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:02.4374138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:52:02.4475909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:52:02.4476628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:52:02.4477325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:52:02.4621154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:52:02.4621683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:52:02.4622376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:52:02.4623073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:52:02.4743953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:52:02.4744865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:52:02.4745612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:52:02.4746540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:52:02.7104587Z ok (3.324s)
2022-05-18T04:52:02.7104935Z 
2022-05-18T04:52:02.7105410Z ----------------------------------------------------------------------
2022-05-18T04:52:02.7105749Z Ran 1 test in 3.325s
2022-05-18T04:52:02.7105932Z 
2022-05-18T04:52:02.7106030Z OK
2022-05-18T04:52:02.7106169Z 
2022-05-18T04:52:02.7106312Z Generating XML reports...
2022-05-18T04:52:02.7146761Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045159.xml
2022-05-18T04:52:04.1170335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:04.1184086Z 
2022-05-18T04:52:04.1184493Z Running tests...
2022-05-18T04:52:04.1185091Z ----------------------------------------------------------------------
2022-05-18T04:52:05.6564115Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:05.6955662Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58654
2022-05-18T04:52:05.7066068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58655
2022-05-18T04:52:06.8595620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:06.8596167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:06.8596957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:06.8597656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:06.8604861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:06.8605345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:06.8711928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:52:06.8712492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:52:06.8713477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:52:06.8714202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:52:07.1120420Z ok (2.993s)
2022-05-18T04:52:07.1120640Z 
2022-05-18T04:52:07.1121015Z ----------------------------------------------------------------------
2022-05-18T04:52:07.1121375Z Ran 1 test in 2.994s
2022-05-18T04:52:07.1121544Z 
2022-05-18T04:52:07.1121642Z OK
2022-05-18T04:52:07.1121784Z 
2022-05-18T04:52:07.1121918Z Generating XML reports...
2022-05-18T04:52:07.1163066Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045204.xml
2022-05-18T04:52:08.5194367Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:08.5209107Z 
2022-05-18T04:52:08.5209557Z Running tests...
2022-05-18T04:52:08.5210134Z ----------------------------------------------------------------------
2022-05-18T04:52:10.0941121Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:10.1341096Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58769
2022-05-18T04:52:10.1448815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58770
2022-05-18T04:52:11.3608410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:11.3609447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:11.3610468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:11.3611170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:11.3616786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:11.3617871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:11.5501897Z ok (3.029s)
2022-05-18T04:52:11.5502267Z 
2022-05-18T04:52:11.5503016Z ----------------------------------------------------------------------
2022-05-18T04:52:11.5503577Z Ran 1 test in 3.029s
2022-05-18T04:52:11.5503746Z 
2022-05-18T04:52:11.5503824Z OK
2022-05-18T04:52:11.5503959Z 
2022-05-18T04:52:11.5504724Z Generating XML reports...
2022-05-18T04:52:11.5551915Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045208.xml
2022-05-18T04:52:12.9525261Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:12.9538969Z 
2022-05-18T04:52:12.9539468Z Running tests...
2022-05-18T04:52:12.9539977Z ----------------------------------------------------------------------
2022-05-18T04:52:14.5278719Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:14.5676781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58878
2022-05-18T04:52:14.5784789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58879
2022-05-18T04:52:15.7810424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:15.7811024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:15.7811824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:15.7812554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:15.7818302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:15.7819328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:15.9838936Z ok (3.030s)
2022-05-18T04:52:15.9839464Z 
2022-05-18T04:52:15.9840190Z ----------------------------------------------------------------------
2022-05-18T04:52:15.9840893Z Ran 1 test in 3.030s
2022-05-18T04:52:15.9841110Z 
2022-05-18T04:52:15.9841206Z OK
2022-05-18T04:52:15.9841344Z 
2022-05-18T04:52:15.9842202Z Generating XML reports...
2022-05-18T04:52:15.9881148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045212.xml
2022-05-18T04:52:17.3837614Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:17.3852313Z 
2022-05-18T04:52:17.3852592Z Running tests...
2022-05-18T04:52:17.3853024Z ----------------------------------------------------------------------
2022-05-18T04:52:18.9498085Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:18.9900304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58987
2022-05-18T04:52:19.0008363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58988
2022-05-18T04:52:20.2086087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:20.2086638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:20.2087450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:20.2088414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:20.2093910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:20.2094801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:20.2303271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:52:20.2304064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:52:20.2304787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:52:20.2306088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:52:20.5061633Z ok (3.120s)
2022-05-18T04:52:20.5061852Z 
2022-05-18T04:52:20.5062265Z ----------------------------------------------------------------------
2022-05-18T04:52:20.5062594Z Ran 1 test in 3.121s
2022-05-18T04:52:20.5062771Z 
2022-05-18T04:52:20.5062872Z OK
2022-05-18T04:52:20.5063008Z 
2022-05-18T04:52:20.5063144Z Generating XML reports...
2022-05-18T04:52:20.5103687Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045217.xml
2022-05-18T04:52:21.9051325Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:21.9065529Z 
2022-05-18T04:52:21.9065856Z Running tests...
2022-05-18T04:52:21.9066522Z ----------------------------------------------------------------------
2022-05-18T04:52:23.4873542Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:23.5265159Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59102
2022-05-18T04:52:23.5371833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59103
2022-05-18T04:52:24.7187714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:24.7188259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:24.7189280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:24.7190011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:24.7295396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:24.8199428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:24.8312344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:52:24.8312875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:52:24.8313569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:52:24.8314251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:52:25.1425355Z ok (3.236s)
2022-05-18T04:52:25.1427257Z 
2022-05-18T04:52:25.1428169Z ----------------------------------------------------------------------
2022-05-18T04:52:25.1428604Z Ran 1 test in 3.236s
2022-05-18T04:52:25.1428775Z 
2022-05-18T04:52:25.1428881Z OK
2022-05-18T04:52:25.1429018Z 
2022-05-18T04:52:25.1429154Z Generating XML reports...
2022-05-18T04:52:25.1467845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045221.xml
2022-05-18T04:52:26.5393216Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:26.5407229Z 
2022-05-18T04:52:26.5407679Z Running tests...
2022-05-18T04:52:26.5408189Z ----------------------------------------------------------------------
2022-05-18T04:52:28.0905766Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:28.1305207Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59217
2022-05-18T04:52:28.1410634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59218
2022-05-18T04:52:29.3510113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:29.3510683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:29.3511487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:29.3512190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:29.3518112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:29.3518868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:30.6364642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxvb58ea
2022-05-18T04:52:30.6365266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxvb58ea/_remote_module_non_scriptable.py
2022-05-18T04:52:30.6855279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78_28kqu
2022-05-18T04:52:30.6856573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78_28kqu/_remote_module_non_scriptable.py
2022-05-18T04:52:31.1488590Z ok (4.608s)
2022-05-18T04:52:31.1488771Z 
2022-05-18T04:52:31.1489161Z ----------------------------------------------------------------------
2022-05-18T04:52:31.1489520Z Ran 1 test in 4.608s
2022-05-18T04:52:31.1489669Z 
2022-05-18T04:52:31.1489766Z OK
2022-05-18T04:52:31.1489903Z 
2022-05-18T04:52:31.1490031Z Generating XML reports...
2022-05-18T04:52:31.1530516Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045226.xml
2022-05-18T04:52:32.5667775Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:32.5682378Z 
2022-05-18T04:52:32.5682656Z Running tests...
2022-05-18T04:52:32.5683101Z ----------------------------------------------------------------------
2022-05-18T04:52:34.1354520Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:34.1755666Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59332
2022-05-18T04:52:34.1864211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59333
2022-05-18T04:52:35.3407931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:35.3408558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:35.3409351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:35.3410067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:35.3416562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:35.3417063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:35.4913861Z ok (2.923s)
2022-05-18T04:52:35.4914079Z 
2022-05-18T04:52:35.4914452Z ----------------------------------------------------------------------
2022-05-18T04:52:35.4914776Z Ran 1 test in 2.923s
2022-05-18T04:52:35.4914949Z 
2022-05-18T04:52:35.4915308Z OK
2022-05-18T04:52:35.4915457Z 
2022-05-18T04:52:35.4915597Z Generating XML reports...
2022-05-18T04:52:35.4956164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045232.xml
2022-05-18T04:52:36.9037998Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:36.9053192Z 
2022-05-18T04:52:36.9053634Z Running tests...
2022-05-18T04:52:36.9054131Z ----------------------------------------------------------------------
2022-05-18T04:52:38.4955108Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:38.5360864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59441
2022-05-18T04:52:38.5470094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59442
2022-05-18T04:52:39.7249110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:39.7249692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:39.7250501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:39.7251217Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:39.7356493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:39.8259966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:40.0522293Z ok (3.146s)
2022-05-18T04:52:40.0522561Z 
2022-05-18T04:52:40.0523016Z ----------------------------------------------------------------------
2022-05-18T04:52:40.0523573Z Ran 1 test in 3.147s
2022-05-18T04:52:40.0523779Z 
2022-05-18T04:52:40.0523859Z OK
2022-05-18T04:52:40.0524044Z 
2022-05-18T04:52:40.0524213Z Generating XML reports...
2022-05-18T04:52:40.0564784Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045236.xml
2022-05-18T04:52:41.4528234Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:41.4542113Z 
2022-05-18T04:52:41.4542663Z Running tests...
2022-05-18T04:52:41.4543515Z ----------------------------------------------------------------------
2022-05-18T04:52:42.9881781Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:43.0270084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59550
2022-05-18T04:52:43.0380538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59551
2022-05-18T04:52:44.1793840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:44.1794408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:44.1795233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:44.1795917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:44.1904379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:44.2804448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:44.5432343Z ok (3.089s)
2022-05-18T04:52:44.5432605Z 
2022-05-18T04:52:44.5433031Z ----------------------------------------------------------------------
2022-05-18T04:52:44.5433378Z Ran 1 test in 3.089s
2022-05-18T04:52:44.5433549Z 
2022-05-18T04:52:44.5433648Z OK
2022-05-18T04:52:44.5433767Z 
2022-05-18T04:52:44.5433910Z Generating XML reports...
2022-05-18T04:52:44.5474232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045241.xml
2022-05-18T04:52:45.9925219Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:45.9940835Z 
2022-05-18T04:52:45.9941079Z Running tests...
2022-05-18T04:52:45.9941544Z ----------------------------------------------------------------------
2022-05-18T04:52:47.5892180Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:47.6299488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59663
2022-05-18T04:52:47.6409661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59664
2022-05-18T04:52:48.8390882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:48.8391458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:48.8392294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:48.8392995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:48.8398711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:48.8399478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:49.1461966Z ok (3.152s)
2022-05-18T04:52:49.1462155Z 
2022-05-18T04:52:49.1462541Z ----------------------------------------------------------------------
2022-05-18T04:52:49.1462882Z Ran 1 test in 3.152s
2022-05-18T04:52:49.1463059Z 
2022-05-18T04:52:49.1463138Z OK
2022-05-18T04:52:49.1463274Z 
2022-05-18T04:52:49.1463411Z Generating XML reports...
2022-05-18T04:52:49.1504585Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045245.xml
2022-05-18T04:52:50.5393547Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:50.5408113Z 
2022-05-18T04:52:50.5408423Z Running tests...
2022-05-18T04:52:50.5408875Z ----------------------------------------------------------------------
2022-05-18T04:52:50.5428470Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T04:52:50.5428862Z 
2022-05-18T04:52:50.5429159Z ----------------------------------------------------------------------
2022-05-18T04:52:50.5429478Z Ran 1 test in 0.002s
2022-05-18T04:52:50.5429645Z 
2022-05-18T04:52:50.5429760Z OK (skipped=1)
2022-05-18T04:52:50.5429922Z 
2022-05-18T04:52:50.5430051Z Generating XML reports...
2022-05-18T04:52:50.5464645Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045250.xml
2022-05-18T04:52:51.7988739Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:51.8004124Z 
2022-05-18T04:52:51.8004404Z Running tests...
2022-05-18T04:52:51.8004867Z ----------------------------------------------------------------------
2022-05-18T04:52:51.8025372Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T04:52:51.8025784Z 
2022-05-18T04:52:51.8026115Z ----------------------------------------------------------------------
2022-05-18T04:52:51.8026436Z Ran 1 test in 0.002s
2022-05-18T04:52:51.8026605Z 
2022-05-18T04:52:51.8026717Z OK (skipped=1)
2022-05-18T04:52:51.8026873Z 
2022-05-18T04:52:51.8026998Z Generating XML reports...
2022-05-18T04:52:51.8063085Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045251.xml
2022-05-18T04:52:53.0590759Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:53.0605999Z 
2022-05-18T04:52:53.0606314Z Running tests...
2022-05-18T04:52:53.0606757Z ----------------------------------------------------------------------
2022-05-18T04:52:54.6490823Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:54.6883177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59842
2022-05-18T04:52:54.6989229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59843
2022-05-18T04:52:55.8867386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:52:55.8868264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:52:55.8869082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:55.8869784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:52:55.8875840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:55.8876919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:56.1039707Z skip: Skipped due to small world size. (3.043s)
2022-05-18T04:52:56.1040181Z 
2022-05-18T04:52:56.1040852Z ----------------------------------------------------------------------
2022-05-18T04:52:56.1041470Z Ran 1 test in 3.043s
2022-05-18T04:52:56.1041766Z 
2022-05-18T04:52:56.1041972Z OK (skipped=1)
2022-05-18T04:52:56.1042269Z 
2022-05-18T04:52:56.1042504Z Generating XML reports...
2022-05-18T04:52:56.1083411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045253.xml
2022-05-18T04:52:57.5088337Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:52:57.5102594Z 
2022-05-18T04:52:57.5102750Z Running tests...
2022-05-18T04:52:57.5103642Z ----------------------------------------------------------------------
2022-05-18T04:52:59.0867252Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:59.1270738Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59951
2022-05-18T04:52:59.1380074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59952
2022-05-18T04:53:00.2952518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:00.2953061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:00.2953868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:00.2954567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:00.2959678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:00.2960693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:02.3083875Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T04:53:02.6464489Z ok (5.136s)
2022-05-18T04:53:02.6464675Z 
2022-05-18T04:53:02.6465071Z ----------------------------------------------------------------------
2022-05-18T04:53:02.6465427Z Ran 1 test in 5.136s
2022-05-18T04:53:02.6465600Z 
2022-05-18T04:53:02.6465698Z OK
2022-05-18T04:53:02.6465817Z 
2022-05-18T04:53:02.6465947Z Generating XML reports...
2022-05-18T04:53:02.6506709Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045257.xml
2022-05-18T04:53:04.0322255Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:04.0336706Z 
2022-05-18T04:53:04.0337005Z Running tests...
2022-05-18T04:53:04.0337446Z ----------------------------------------------------------------------
2022-05-18T04:53:05.6268625Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:05.6658885Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60060
2022-05-18T04:53:05.6765261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60061
2022-05-18T04:53:06.8572178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:06.8572782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:06.8573582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:06.8574291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:06.8681945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:06.9582563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:06.9698652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:53:06.9699337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:53:06.9700040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:53:06.9700739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:53:06.9702826Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-05-18T04:53:06.9703252Z No ranks successfully processed in monitoredBarrier.
2022-05-18T04:53:06.9731870Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-05-18T04:53:07.2819704Z ok (3.248s)
2022-05-18T04:53:07.2820061Z 
2022-05-18T04:53:07.2820704Z ----------------------------------------------------------------------
2022-05-18T04:53:07.2821071Z Ran 1 test in 3.248s
2022-05-18T04:53:07.2821242Z 
2022-05-18T04:53:07.2821322Z OK
2022-05-18T04:53:07.2821458Z 
2022-05-18T04:53:07.2821592Z Generating XML reports...
2022-05-18T04:53:07.2862317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045304.xml
2022-05-18T04:53:08.7065431Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:08.7079419Z 
2022-05-18T04:53:08.7079626Z Running tests...
2022-05-18T04:53:08.7080258Z ----------------------------------------------------------------------
2022-05-18T04:53:10.2835306Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:10.3236434Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60175
2022-05-18T04:53:10.3344567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60176
2022-05-18T04:53:11.5356219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:11.5356784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:11.5357574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:11.5358522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:11.5464280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:11.6366407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:11.6477857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:53:11.6478625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:53:11.6479340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:53:11.6480023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:53:11.7487119Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-05-18T04:53:12.0399257Z ok (3.331s)
2022-05-18T04:53:12.0399509Z 
2022-05-18T04:53:12.0400037Z ----------------------------------------------------------------------
2022-05-18T04:53:12.0400569Z Ran 1 test in 3.332s
2022-05-18T04:53:12.0400743Z 
2022-05-18T04:53:12.0400838Z OK
2022-05-18T04:53:12.0400974Z 
2022-05-18T04:53:12.0401108Z Generating XML reports...
2022-05-18T04:53:12.0442063Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045308.xml
2022-05-18T04:53:13.4398806Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:13.4412515Z 
2022-05-18T04:53:13.4412993Z Running tests...
2022-05-18T04:53:13.4413490Z ----------------------------------------------------------------------
2022-05-18T04:53:14.9827379Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:15.0229705Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60290
2022-05-18T04:53:15.0338033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60291
2022-05-18T04:53:16.2134137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:16.2134675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:16.2135699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:16.2136429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:16.2142240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:16.2142740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:16.4388171Z skip: Skipped due to small world size. (2.997s)
2022-05-18T04:53:16.4388419Z 
2022-05-18T04:53:16.4388814Z ----------------------------------------------------------------------
2022-05-18T04:53:16.4389185Z Ran 1 test in 2.998s
2022-05-18T04:53:16.4389334Z 
2022-05-18T04:53:16.4389443Z OK (skipped=1)
2022-05-18T04:53:16.4389597Z 
2022-05-18T04:53:16.4389723Z Generating XML reports...
2022-05-18T04:53:16.4430305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045313.xml
2022-05-18T04:53:17.8423834Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:17.8437897Z 
2022-05-18T04:53:17.8438023Z Running tests...
2022-05-18T04:53:17.8438867Z ----------------------------------------------------------------------
2022-05-18T04:53:17.8465705Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T04:53:17.8466416Z 
2022-05-18T04:53:17.8467016Z ----------------------------------------------------------------------
2022-05-18T04:53:17.8467429Z Ran 1 test in 0.003s
2022-05-18T04:53:17.8467904Z 
2022-05-18T04:53:17.8468018Z OK (skipped=1)
2022-05-18T04:53:17.8468158Z 
2022-05-18T04:53:17.8468284Z Generating XML reports...
2022-05-18T04:53:17.8503456Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045317.xml
2022-05-18T04:53:19.0654471Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:19.0668576Z 
2022-05-18T04:53:19.0669018Z Running tests...
2022-05-18T04:53:19.0670036Z ----------------------------------------------------------------------
2022-05-18T04:53:19.0698266Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T04:53:19.0699015Z 
2022-05-18T04:53:19.0699444Z ----------------------------------------------------------------------
2022-05-18T04:53:19.0699789Z Ran 1 test in 0.003s
2022-05-18T04:53:19.0699956Z 
2022-05-18T04:53:19.0700069Z OK (skipped=1)
2022-05-18T04:53:19.0700239Z 
2022-05-18T04:53:19.0700348Z Generating XML reports...
2022-05-18T04:53:19.0735832Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045319.xml
2022-05-18T04:53:20.3361627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:20.3376265Z 
2022-05-18T04:53:20.3376737Z Running tests...
2022-05-18T04:53:20.3377250Z ----------------------------------------------------------------------
2022-05-18T04:53:20.3402194Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-05-18T04:53:20.3402537Z 
2022-05-18T04:53:20.3402812Z ----------------------------------------------------------------------
2022-05-18T04:53:20.3403132Z Ran 1 test in 0.003s
2022-05-18T04:53:20.3403297Z 
2022-05-18T04:53:20.3403411Z OK (skipped=1)
2022-05-18T04:53:20.3403569Z 
2022-05-18T04:53:20.3403698Z Generating XML reports...
2022-05-18T04:53:20.3439504Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045320.xml
2022-05-18T04:53:21.6046427Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:21.6060717Z 
2022-05-18T04:53:21.6061028Z Running tests...
2022-05-18T04:53:21.6061801Z ----------------------------------------------------------------------
2022-05-18T04:53:21.6092289Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T04:53:21.6092627Z 
2022-05-18T04:53:21.6092902Z ----------------------------------------------------------------------
2022-05-18T04:53:21.6093237Z Ran 1 test in 0.003s
2022-05-18T04:53:21.6093402Z 
2022-05-18T04:53:21.6093516Z OK (skipped=1)
2022-05-18T04:53:21.6093655Z 
2022-05-18T04:53:21.6093784Z Generating XML reports...
2022-05-18T04:53:21.6129223Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045321.xml
2022-05-18T04:53:22.8458958Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:22.8471743Z 
2022-05-18T04:53:22.8472176Z Running tests...
2022-05-18T04:53:22.8472664Z ----------------------------------------------------------------------
2022-05-18T04:53:22.8498274Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-05-18T04:53:22.8498764Z 
2022-05-18T04:53:22.8499059Z ----------------------------------------------------------------------
2022-05-18T04:53:22.8499398Z Ran 1 test in 0.003s
2022-05-18T04:53:22.8499565Z 
2022-05-18T04:53:22.8499678Z OK (skipped=1)
2022-05-18T04:53:22.8499816Z 
2022-05-18T04:53:22.8499945Z Generating XML reports...
2022-05-18T04:53:22.8534065Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045322.xml
2022-05-18T04:53:24.1049237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:24.1063672Z 
2022-05-18T04:53:24.1063888Z Running tests...
2022-05-18T04:53:24.1064337Z ----------------------------------------------------------------------
2022-05-18T04:53:24.1087502Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:53:24.1087805Z 
2022-05-18T04:53:24.1088099Z ----------------------------------------------------------------------
2022-05-18T04:53:24.1088443Z Ran 1 test in 0.002s
2022-05-18T04:53:24.1088608Z 
2022-05-18T04:53:24.1088722Z OK (skipped=1)
2022-05-18T04:53:24.1088881Z 
2022-05-18T04:53:24.1088991Z Generating XML reports...
2022-05-18T04:53:24.1125577Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045324.xml
2022-05-18T04:53:25.3436511Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:25.3451101Z 
2022-05-18T04:53:25.3451547Z Running tests...
2022-05-18T04:53:25.3452048Z ----------------------------------------------------------------------
2022-05-18T04:53:25.3477291Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:53:25.3477617Z 
2022-05-18T04:53:25.3477911Z ----------------------------------------------------------------------
2022-05-18T04:53:25.3478628Z Ran 1 test in 0.003s
2022-05-18T04:53:25.3478801Z 
2022-05-18T04:53:25.3478896Z OK (skipped=1)
2022-05-18T04:53:25.3479058Z 
2022-05-18T04:53:25.3479187Z Generating XML reports...
2022-05-18T04:53:25.3514307Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045325.xml
2022-05-18T04:53:26.5591277Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:26.5607092Z 
2022-05-18T04:53:26.5607582Z Running tests...
2022-05-18T04:53:26.5608065Z ----------------------------------------------------------------------
2022-05-18T04:53:26.5631041Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:53:26.5631404Z 
2022-05-18T04:53:26.5632171Z ----------------------------------------------------------------------
2022-05-18T04:53:26.5632539Z Ran 1 test in 0.002s
2022-05-18T04:53:26.5632708Z 
2022-05-18T04:53:26.5632829Z OK (skipped=1)
2022-05-18T04:53:26.5633001Z 
2022-05-18T04:53:26.5633133Z Generating XML reports...
2022-05-18T04:53:26.5669384Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045326.xml
2022-05-18T04:53:27.8264756Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:27.8278901Z 
2022-05-18T04:53:27.8279296Z Running tests...
2022-05-18T04:53:27.8279825Z ----------------------------------------------------------------------
2022-05-18T04:53:29.4289008Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:29.4689225Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60663
2022-05-18T04:53:29.4796737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60664
2022-05-18T04:53:30.6600982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:30.6601543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:30.6602324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:30.6603028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:30.6708776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:30.7611662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:30.9850356Z ok (3.157s)
2022-05-18T04:53:30.9850769Z 
2022-05-18T04:53:30.9851475Z ----------------------------------------------------------------------
2022-05-18T04:53:30.9852108Z Ran 1 test in 3.157s
2022-05-18T04:53:30.9852417Z 
2022-05-18T04:53:30.9852586Z OK
2022-05-18T04:53:30.9852848Z 
2022-05-18T04:53:30.9853087Z Generating XML reports...
2022-05-18T04:53:30.9894777Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045327.xml
2022-05-18T04:53:32.3852566Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:32.3867630Z 
2022-05-18T04:53:32.3868110Z Running tests...
2022-05-18T04:53:32.3868616Z ----------------------------------------------------------------------
2022-05-18T04:53:33.9629903Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:34.0031211Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60772
2022-05-18T04:53:34.0138617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60773
2022-05-18T04:53:35.1761751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:35.1762299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:35.1763099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:35.1763802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:35.1869362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:35.2774083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:35.5190241Z ok (3.132s)
2022-05-18T04:53:35.5190502Z 
2022-05-18T04:53:35.5191150Z ----------------------------------------------------------------------
2022-05-18T04:53:35.5191832Z Ran 1 test in 3.132s
2022-05-18T04:53:35.5192032Z 
2022-05-18T04:53:35.5192134Z OK
2022-05-18T04:53:35.5192295Z 
2022-05-18T04:53:35.5192438Z Generating XML reports...
2022-05-18T04:53:35.5233333Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045332.xml
2022-05-18T04:53:36.9427205Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:36.9441967Z 
2022-05-18T04:53:36.9442248Z Running tests...
2022-05-18T04:53:36.9442709Z ----------------------------------------------------------------------
2022-05-18T04:53:36.9463742Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:53:36.9464074Z 
2022-05-18T04:53:36.9464372Z ----------------------------------------------------------------------
2022-05-18T04:53:36.9464698Z Ran 1 test in 0.002s
2022-05-18T04:53:36.9464865Z 
2022-05-18T04:53:36.9464989Z OK (skipped=1)
2022-05-18T04:53:36.9465149Z 
2022-05-18T04:53:36.9465277Z Generating XML reports...
2022-05-18T04:53:36.9501533Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045336.xml
2022-05-18T04:53:38.2054641Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:38.2069243Z 
2022-05-18T04:53:38.2069556Z Running tests...
2022-05-18T04:53:38.2070013Z ----------------------------------------------------------------------
2022-05-18T04:53:38.2090735Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:53:38.2091395Z 
2022-05-18T04:53:38.2091667Z ----------------------------------------------------------------------
2022-05-18T04:53:38.2092004Z Ran 1 test in 0.002s
2022-05-18T04:53:38.2092177Z 
2022-05-18T04:53:38.2092291Z OK (skipped=1)
2022-05-18T04:53:38.2092460Z 
2022-05-18T04:53:38.2092590Z Generating XML reports...
2022-05-18T04:53:38.2128569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045338.xml
2022-05-18T04:53:39.4579722Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:39.4594247Z 
2022-05-18T04:53:39.4594373Z Running tests...
2022-05-18T04:53:39.4595117Z ----------------------------------------------------------------------
2022-05-18T04:53:41.0222327Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:41.0614199Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60947
2022-05-18T04:53:41.0720339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60948
2022-05-18T04:53:42.2196304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:42.2196930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:42.2197731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:42.2198687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:42.2304109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:42.3209776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:43.4936064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaqmjrgf1
2022-05-18T04:53:43.4936738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaqmjrgf1/_remote_module_non_scriptable.py
2022-05-18T04:53:43.5784061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v8hoq5t
2022-05-18T04:53:43.5785226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v8hoq5t/_remote_module_non_scriptable.py
2022-05-18T04:53:44.0799727Z ok (4.620s)
2022-05-18T04:53:44.0800106Z 
2022-05-18T04:53:44.0800561Z ----------------------------------------------------------------------
2022-05-18T04:53:44.0800909Z Ran 1 test in 4.620s
2022-05-18T04:53:44.0801076Z 
2022-05-18T04:53:44.0801202Z OK
2022-05-18T04:53:44.0801469Z 
2022-05-18T04:53:44.0801666Z Generating XML reports...
2022-05-18T04:53:44.0841014Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045339.xml
2022-05-18T04:53:45.4879853Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:45.4894280Z 
2022-05-18T04:53:45.4894781Z Running tests...
2022-05-18T04:53:45.4895385Z ----------------------------------------------------------------------
2022-05-18T04:53:47.0311568Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:47.0703527Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61062
2022-05-18T04:53:47.0808149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61063
2022-05-18T04:53:48.2369108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:48.2369669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:48.2370467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:48.2371417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:48.2476411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:48.3382933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:49.4985961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_0zwlqw
2022-05-18T04:53:49.4986566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_0zwlqw/_remote_module_non_scriptable.py
2022-05-18T04:53:49.6056674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppolraebt
2022-05-18T04:53:49.6057774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppolraebt/_remote_module_non_scriptable.py
2022-05-18T04:53:50.0888761Z ok (4.599s)
2022-05-18T04:53:50.0889789Z 
2022-05-18T04:53:50.0890231Z ----------------------------------------------------------------------
2022-05-18T04:53:50.0890582Z Ran 1 test in 4.599s
2022-05-18T04:53:50.0890750Z 
2022-05-18T04:53:50.0890847Z OK
2022-05-18T04:53:50.0890989Z 
2022-05-18T04:53:50.0891127Z Generating XML reports...
2022-05-18T04:53:50.0931382Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045345.xml
2022-05-18T04:53:51.5167854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:51.5183903Z 
2022-05-18T04:53:51.5184371Z Running tests...
2022-05-18T04:53:51.5184870Z ----------------------------------------------------------------------
2022-05-18T04:53:53.0856356Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:53.1259206Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61177
2022-05-18T04:53:53.1368418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61178
2022-05-18T04:53:54.2887134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:53:54.2887736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:53:54.2888758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:54.2889502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:53:54.2996953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:54.3901451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:56.8463406Z ok (5.328s)
2022-05-18T04:53:56.8463634Z 
2022-05-18T04:53:56.8464034Z ----------------------------------------------------------------------
2022-05-18T04:53:56.8464404Z Ran 1 test in 5.328s
2022-05-18T04:53:56.8464571Z 
2022-05-18T04:53:56.8464648Z OK
2022-05-18T04:53:56.8467181Z 
2022-05-18T04:53:56.8467717Z Generating XML reports...
2022-05-18T04:53:56.8505064Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045351.xml
2022-05-18T04:53:58.2728208Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:53:58.2744227Z 
2022-05-18T04:53:58.2744648Z Running tests...
2022-05-18T04:53:58.2745163Z ----------------------------------------------------------------------
2022-05-18T04:53:59.8344867Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:59.8749096Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61289
2022-05-18T04:53:59.8856595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61290
2022-05-18T04:54:01.0644162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:01.0644726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:01.0645561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:01.0646270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:01.0752070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:01.1657558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:03.6954058Z ok (5.421s)
2022-05-18T04:54:03.6954270Z 
2022-05-18T04:54:03.6954688Z ----------------------------------------------------------------------
2022-05-18T04:54:03.6955038Z Ran 1 test in 5.421s
2022-05-18T04:54:03.6955237Z 
2022-05-18T04:54:03.6955314Z OK
2022-05-18T04:54:03.6955451Z 
2022-05-18T04:54:03.6955586Z Generating XML reports...
2022-05-18T04:54:03.6996420Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045358.xml
2022-05-18T04:54:05.0944866Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:54:05.0959523Z 
2022-05-18T04:54:05.0959689Z Running tests...
2022-05-18T04:54:05.0960576Z ----------------------------------------------------------------------
2022-05-18T04:54:06.6843932Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:06.6999449Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.604s)
2022-05-18T04:54:06.7000370Z 
2022-05-18T04:54:06.7000675Z ----------------------------------------------------------------------
2022-05-18T04:54:06.7000998Z Ran 1 test in 1.604s
2022-05-18T04:54:06.7001245Z 
2022-05-18T04:54:06.7001448Z OK (skipped=1)
2022-05-18T04:54:06.7001697Z 
2022-05-18T04:54:06.7002129Z Generating XML reports...
2022-05-18T04:54:06.7034568Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045405.xml
2022-05-18T04:54:08.0795040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:54:08.0809762Z 
2022-05-18T04:54:08.0809917Z Running tests...
2022-05-18T04:54:08.0810839Z ----------------------------------------------------------------------
2022-05-18T04:54:09.6466410Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:09.6621706Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.581s)
2022-05-18T04:54:09.6622280Z 
2022-05-18T04:54:09.6622569Z ----------------------------------------------------------------------
2022-05-18T04:54:09.6622891Z Ran 1 test in 1.581s
2022-05-18T04:54:09.6623060Z 
2022-05-18T04:54:09.6623171Z OK (skipped=1)
2022-05-18T04:54:09.6623328Z 
2022-05-18T04:54:09.6623456Z Generating XML reports...
2022-05-18T04:54:09.6656968Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045408.xml
2022-05-18T04:54:11.0379996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:54:11.0394328Z 
2022-05-18T04:54:11.0394484Z Running tests...
2022-05-18T04:54:11.0395275Z ----------------------------------------------------------------------
2022-05-18T04:54:12.5956431Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:12.6359433Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61469
2022-05-18T04:54:12.6465644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61470
2022-05-18T04:54:13.8361190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:13.8361786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:13.8362569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:13.8363266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:13.8369026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:13.8369521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:14.0516683Z skip: Need at least 4 CUDA devices (3.012s)
2022-05-18T04:54:14.0517171Z 
2022-05-18T04:54:14.0517592Z ----------------------------------------------------------------------
2022-05-18T04:54:14.0517940Z Ran 1 test in 3.012s
2022-05-18T04:54:14.0518333Z 
2022-05-18T04:54:14.0518454Z OK (skipped=1)
2022-05-18T04:54:14.0518596Z 
2022-05-18T04:54:14.0518730Z Generating XML reports...
2022-05-18T04:54:14.0559623Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045411.xml
2022-05-18T04:54:15.4497738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:54:15.4512345Z 
2022-05-18T04:54:15.4512661Z Running tests...
2022-05-18T04:54:15.4513129Z ----------------------------------------------------------------------
2022-05-18T04:54:17.0203614Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:17.0605266Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61578
2022-05-18T04:54:17.0712772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61579
2022-05-18T04:54:18.2333598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:18.2334140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:18.2334948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:18.2335648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:18.2341590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:18.2342261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:18.4761846Z skip: Need at least 4 CUDA devices (3.024s)
2022-05-18T04:54:18.4762110Z 
2022-05-18T04:54:18.4762476Z ----------------------------------------------------------------------
2022-05-18T04:54:18.4762834Z Ran 1 test in 3.025s
2022-05-18T04:54:18.4763001Z 
2022-05-18T04:54:18.4763115Z OK (skipped=1)
2022-05-18T04:54:18.4763274Z 
2022-05-18T04:54:18.4763401Z Generating XML reports...
2022-05-18T04:54:18.4804630Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045415.xml
2022-05-18T04:54:19.8743331Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:54:19.8758063Z 
2022-05-18T04:54:19.8758425Z Running tests...
2022-05-18T04:54:19.8759548Z ----------------------------------------------------------------------
2022-05-18T04:54:21.4505351Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:21.4906501Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61687
2022-05-18T04:54:21.5013539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61688
2022-05-18T04:54:22.6699095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:22.6699672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:22.6700471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:22.6701176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:22.6809398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:22.7709136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:22.7825404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:54:22.7825925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:54:22.7826600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:22.7827299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:23.0068168Z ok (3.131s)
2022-05-18T04:54:23.0068406Z 
2022-05-18T04:54:23.0068807Z ----------------------------------------------------------------------
2022-05-18T04:54:23.0069129Z Ran 1 test in 3.131s
2022-05-18T04:54:23.0069316Z 
2022-05-18T04:54:23.0069412Z OK
2022-05-18T04:54:23.0069550Z 
2022-05-18T04:54:23.0069683Z Generating XML reports...
2022-05-18T04:54:23.0111474Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045419.xml
2022-05-18T04:54:24.4564928Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:54:24.4579052Z 
2022-05-18T04:54:24.4579398Z Running tests...
2022-05-18T04:54:24.4579831Z ----------------------------------------------------------------------
2022-05-18T04:54:26.0309247Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:26.0700108Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61806
2022-05-18T04:54:26.0805474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61807
2022-05-18T04:54:27.2500948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:27.2502058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:27.2503415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:27.2504820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:27.2509142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:27.2511114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:27.2619684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:54:27.2620730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:54:27.2622085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:27.2623960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:27.5857040Z ok (3.127s)
2022-05-18T04:54:27.5857272Z 
2022-05-18T04:54:27.5857926Z ----------------------------------------------------------------------
2022-05-18T04:54:27.5858305Z Ran 1 test in 3.128s
2022-05-18T04:54:27.5858482Z 
2022-05-18T04:54:27.5858580Z OK
2022-05-18T04:54:27.5858716Z 
2022-05-18T04:54:27.5858852Z Generating XML reports...
2022-05-18T04:54:27.5899716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045424.xml
2022-05-18T04:54:28.9897724Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:54:28.9918749Z 
2022-05-18T04:54:28.9919067Z Running tests...
2022-05-18T04:54:28.9919605Z ----------------------------------------------------------------------
2022-05-18T04:54:30.5604266Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:30.6011184Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61925
2022-05-18T04:54:30.6119795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61926
2022-05-18T04:54:31.7983696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:31.7984316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:31.7985111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:31.7985814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:31.7991184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:31.7991816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:31.8200086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:54:31.8201193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:54:31.8201940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:31.8202928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:32.0174620Z ok (3.026s)
2022-05-18T04:54:32.0174860Z 
2022-05-18T04:54:32.0175263Z ----------------------------------------------------------------------
2022-05-18T04:54:32.0175614Z Ran 1 test in 3.026s
2022-05-18T04:54:32.0175789Z 
2022-05-18T04:54:32.0175887Z OK
2022-05-18T04:54:32.0176026Z 
2022-05-18T04:54:32.0176169Z Generating XML reports...
2022-05-18T04:54:32.0217637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045428.xml
2022-05-18T04:54:33.4289274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:54:33.4303596Z 
2022-05-18T04:54:33.4303952Z Running tests...
2022-05-18T04:54:33.4304400Z ----------------------------------------------------------------------
2022-05-18T04:54:35.0190745Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:35.0591461Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62044
2022-05-18T04:54:35.0699721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62045
2022-05-18T04:54:36.2349643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:36.2350492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:36.2351313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:36.2352071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:36.2457265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:36.3360399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:36.3470765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:54:36.3471262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:54:36.3471979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:36.3472689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:54:36.5752003Z ok (3.144s)
2022-05-18T04:54:36.5752202Z 
2022-05-18T04:54:36.5752684Z ----------------------------------------------------------------------
2022-05-18T04:54:36.5753035Z Ran 1 test in 3.145s
2022-05-18T04:54:36.5753204Z 
2022-05-18T04:54:36.5753309Z OK
2022-05-18T04:54:36.5753448Z 
2022-05-18T04:54:36.5753565Z Generating XML reports...
2022-05-18T04:54:36.5794826Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045433.xml
2022-05-18T04:54:37.9963995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:54:37.9978689Z 
2022-05-18T04:54:37.9978963Z Running tests...
2022-05-18T04:54:37.9979409Z ----------------------------------------------------------------------
2022-05-18T04:54:39.5758324Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:39.6159563Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62163
2022-05-18T04:54:39.6266554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62164
2022-05-18T04:54:40.8413988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:40.8414539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:40.8415337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:40.8416055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:40.8524812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:40.9426704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:41.1319574Z skip: Skipped due to small world size. (3.134s)
2022-05-18T04:54:41.1319884Z 
2022-05-18T04:54:41.1320302Z ----------------------------------------------------------------------
2022-05-18T04:54:41.1320646Z Ran 1 test in 3.134s
2022-05-18T04:54:41.1320815Z 
2022-05-18T04:54:41.1320910Z OK (skipped=1)
2022-05-18T04:54:41.1321077Z 
2022-05-18T04:54:41.1321210Z Generating XML reports...
2022-05-18T04:54:41.1363040Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045437.xml
2022-05-18T04:54:42.5375976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:54:42.5391311Z 
2022-05-18T04:54:42.5391667Z Running tests...
2022-05-18T04:54:42.5392216Z ----------------------------------------------------------------------
2022-05-18T04:54:44.1137262Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:44.1537440Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62272
2022-05-18T04:54:44.1647315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62273
2022-05-18T04:54:45.3246251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:45.3246844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:45.3247646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:45.3248357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:45.3254923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:45.3255621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:45.5698487Z skip: Skipped due to small world size. (3.030s)
2022-05-18T04:54:45.5698754Z 
2022-05-18T04:54:45.5699127Z ----------------------------------------------------------------------
2022-05-18T04:54:45.5699499Z Ran 1 test in 3.031s
2022-05-18T04:54:45.5699677Z 
2022-05-18T04:54:45.5699795Z OK (skipped=1)
2022-05-18T04:54:45.5699958Z 
2022-05-18T04:54:45.5700088Z Generating XML reports...
2022-05-18T04:54:45.5740512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045442.xml
2022-05-18T04:54:46.9724691Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:54:46.9739338Z 
2022-05-18T04:54:46.9739767Z Running tests...
2022-05-18T04:54:46.9740293Z ----------------------------------------------------------------------
2022-05-18T04:54:48.5491117Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:48.5890275Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62381
2022-05-18T04:54:48.5999483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62382
2022-05-18T04:54:49.7957393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:49.7957973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:49.7959100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:49.7959811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:49.7965779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:49.7966296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:50.0049836Z skip: Skipped due to small world size. (3.031s)
2022-05-18T04:54:50.0050143Z 
2022-05-18T04:54:50.0050543Z ----------------------------------------------------------------------
2022-05-18T04:54:50.0050909Z Ran 1 test in 3.031s
2022-05-18T04:54:50.0051078Z 
2022-05-18T04:54:50.0051191Z OK (skipped=1)
2022-05-18T04:54:50.0051358Z 
2022-05-18T04:54:50.0051472Z Generating XML reports...
2022-05-18T04:54:50.0091800Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045446.xml
2022-05-18T04:54:51.4062952Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:54:51.4077687Z 
2022-05-18T04:54:51.4078093Z Running tests...
2022-05-18T04:54:51.4078853Z ----------------------------------------------------------------------
2022-05-18T04:54:52.9775098Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:53.0165745Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62490
2022-05-18T04:54:53.0271829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62491
2022-05-18T04:54:54.2045739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:54.2046288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:54.2047075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:54.2047779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:54.2054007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:54.2054490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:54.4321828Z skip: Skipped due to small world size. (3.024s)
2022-05-18T04:54:54.4322092Z 
2022-05-18T04:54:54.4322489Z ----------------------------------------------------------------------
2022-05-18T04:54:54.4322839Z Ran 1 test in 3.024s
2022-05-18T04:54:54.4323008Z 
2022-05-18T04:54:54.4323122Z OK (skipped=1)
2022-05-18T04:54:54.4323283Z 
2022-05-18T04:54:54.4323412Z Generating XML reports...
2022-05-18T04:54:54.4364031Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045451.xml
2022-05-18T04:54:55.8339258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:54:55.8353684Z 
2022-05-18T04:54:55.8353948Z Running tests...
2022-05-18T04:54:55.8354417Z ----------------------------------------------------------------------
2022-05-18T04:54:57.4225786Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:57.4626237Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62599
2022-05-18T04:54:57.4733993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62600
2022-05-18T04:54:58.6420586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:54:58.6421185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:54:58.6421994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:58.6422682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:54:58.6530246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:58.7431509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:58.9787629Z ok (3.143s)
2022-05-18T04:54:58.9787857Z 
2022-05-18T04:54:58.9788265Z ----------------------------------------------------------------------
2022-05-18T04:54:58.9788593Z Ran 1 test in 3.143s
2022-05-18T04:54:58.9788760Z 
2022-05-18T04:54:58.9788864Z OK
2022-05-18T04:54:58.9789020Z 
2022-05-18T04:54:58.9789159Z Generating XML reports...
2022-05-18T04:54:58.9830618Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045455.xml
2022-05-18T04:55:00.3778097Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:00.3792265Z 
2022-05-18T04:55:00.3792715Z Running tests...
2022-05-18T04:55:00.3793208Z ----------------------------------------------------------------------
2022-05-18T04:55:01.9709446Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:02.0114925Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62712
2022-05-18T04:55:02.0223573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62713
2022-05-18T04:55:03.2033176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:03.2033745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:03.2034543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:03.2035246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:03.2041618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:03.2042090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:03.4273606Z ok (3.048s)
2022-05-18T04:55:03.4273853Z 
2022-05-18T04:55:03.4274248Z ----------------------------------------------------------------------
2022-05-18T04:55:03.4274594Z Ran 1 test in 3.048s
2022-05-18T04:55:03.4274742Z 
2022-05-18T04:55:03.4274846Z OK
2022-05-18T04:55:03.4274981Z 
2022-05-18T04:55:03.4275117Z Generating XML reports...
2022-05-18T04:55:03.4315502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045500.xml
2022-05-18T04:55:04.8314295Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:04.8329972Z 
2022-05-18T04:55:04.8330533Z Running tests...
2022-05-18T04:55:04.8331018Z ----------------------------------------------------------------------
2022-05-18T04:55:04.8353504Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-05-18T04:55:04.8354069Z 
2022-05-18T04:55:04.8354373Z ----------------------------------------------------------------------
2022-05-18T04:55:04.8354786Z Ran 1 test in 0.002s
2022-05-18T04:55:04.8354938Z 
2022-05-18T04:55:04.8355054Z OK (skipped=1)
2022-05-18T04:55:04.8355213Z 
2022-05-18T04:55:04.8355342Z Generating XML reports...
2022-05-18T04:55:04.8392208Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045504.xml
2022-05-18T04:55:06.0825629Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:06.0840736Z 
2022-05-18T04:55:06.0841165Z Running tests...
2022-05-18T04:55:06.0841644Z ----------------------------------------------------------------------
2022-05-18T04:55:07.6702202Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:07.7101764Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62858
2022-05-18T04:55:07.7210017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62859
2022-05-18T04:55:08.8874766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:08.8875328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:08.8876133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:08.8876820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:08.8984377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:08.9886067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:09.2263562Z ok (3.142s)
2022-05-18T04:55:09.2263748Z 
2022-05-18T04:55:09.2264315Z ----------------------------------------------------------------------
2022-05-18T04:55:09.2264941Z Ran 1 test in 3.142s
2022-05-18T04:55:09.2265111Z 
2022-05-18T04:55:09.2265207Z OK
2022-05-18T04:55:09.2265347Z 
2022-05-18T04:55:09.2265481Z Generating XML reports...
2022-05-18T04:55:09.2305556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045506.xml
2022-05-18T04:55:10.6465465Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:10.6480487Z 
2022-05-18T04:55:10.6480733Z Running tests...
2022-05-18T04:55:10.6481174Z ----------------------------------------------------------------------
2022-05-18T04:55:12.2301301Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:12.2690757Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62971
2022-05-18T04:55:12.2797816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62972
2022-05-18T04:55:13.4576146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:13.4576727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:13.4577513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:13.4578209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:13.4686219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:13.5587552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:13.7851679Z ok (3.137s)
2022-05-18T04:55:13.7851874Z 
2022-05-18T04:55:13.7852264Z ----------------------------------------------------------------------
2022-05-18T04:55:13.7852611Z Ran 1 test in 3.137s
2022-05-18T04:55:13.7852790Z 
2022-05-18T04:55:13.7852913Z OK
2022-05-18T04:55:13.7853053Z 
2022-05-18T04:55:13.7853170Z Generating XML reports...
2022-05-18T04:55:13.7894023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045510.xml
2022-05-18T04:55:15.1852593Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:15.1867532Z 
2022-05-18T04:55:15.1867995Z Running tests...
2022-05-18T04:55:15.1868470Z ----------------------------------------------------------------------
2022-05-18T04:55:15.1892206Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-05-18T04:55:15.1892518Z 
2022-05-18T04:55:15.1892803Z ----------------------------------------------------------------------
2022-05-18T04:55:15.1893140Z Ran 1 test in 0.003s
2022-05-18T04:55:15.1893289Z 
2022-05-18T04:55:15.1893406Z OK (skipped=1)
2022-05-18T04:55:15.1893567Z 
2022-05-18T04:55:15.1893695Z Generating XML reports...
2022-05-18T04:55:15.1930054Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045515.xml
2022-05-18T04:55:16.4442266Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:16.4456624Z 
2022-05-18T04:55:16.4457051Z Running tests...
2022-05-18T04:55:16.4457566Z ----------------------------------------------------------------------
2022-05-18T04:55:16.4481607Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-05-18T04:55:16.4481928Z 
2022-05-18T04:55:16.4482219Z ----------------------------------------------------------------------
2022-05-18T04:55:16.4482563Z Ran 1 test in 0.003s
2022-05-18T04:55:16.4482714Z 
2022-05-18T04:55:16.4482828Z OK (skipped=1)
2022-05-18T04:55:16.4482991Z 
2022-05-18T04:55:16.4483128Z Generating XML reports...
2022-05-18T04:55:16.4519290Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045516.xml
2022-05-18T04:55:17.7066984Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:17.7082089Z 
2022-05-18T04:55:17.7082532Z Running tests...
2022-05-18T04:55:17.7083016Z ----------------------------------------------------------------------
2022-05-18T04:55:19.2757900Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:19.3150829Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63150
2022-05-18T04:55:19.3258626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63151
2022-05-18T04:55:20.5075478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:20.5076053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:20.5076856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:20.5077593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:20.5185669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:20.6086529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:20.8310010Z ok (3.122s)
2022-05-18T04:55:20.8310236Z 
2022-05-18T04:55:20.8310618Z ----------------------------------------------------------------------
2022-05-18T04:55:20.8310965Z Ran 1 test in 3.123s
2022-05-18T04:55:20.8311142Z 
2022-05-18T04:55:20.8311240Z OK
2022-05-18T04:55:20.8311363Z 
2022-05-18T04:55:20.8311499Z Generating XML reports...
2022-05-18T04:55:20.8351874Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045517.xml
2022-05-18T04:55:22.2317059Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:22.2331438Z 
2022-05-18T04:55:22.2331877Z Running tests...
2022-05-18T04:55:22.2332385Z ----------------------------------------------------------------------
2022-05-18T04:55:23.7852217Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:23.8245835Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63263
2022-05-18T04:55:23.8350776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63264
2022-05-18T04:55:24.9935399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:24.9935935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:24.9936734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:24.9937461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:24.9942924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:24.9943406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:25.2400585Z ok (3.007s)
2022-05-18T04:55:25.2401221Z 
2022-05-18T04:55:25.2402063Z ----------------------------------------------------------------------
2022-05-18T04:55:25.2402788Z Ran 1 test in 3.007s
2022-05-18T04:55:25.2403022Z 
2022-05-18T04:55:25.2403121Z OK
2022-05-18T04:55:25.2403261Z 
2022-05-18T04:55:25.2403400Z Generating XML reports...
2022-05-18T04:55:25.2444767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045522.xml
2022-05-18T04:55:26.6459038Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:26.6473507Z 
2022-05-18T04:55:26.6473973Z Running tests...
2022-05-18T04:55:26.6474936Z ----------------------------------------------------------------------
2022-05-18T04:55:28.2252744Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:28.2652658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63376
2022-05-18T04:55:28.2760794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63377
2022-05-18T04:55:29.4138488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:29.4139653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:29.4140492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:29.4141200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:29.4247577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:29.5148940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:29.6811046Z ok (3.033s)
2022-05-18T04:55:29.6811236Z 
2022-05-18T04:55:29.6811774Z ----------------------------------------------------------------------
2022-05-18T04:55:29.6812119Z Ran 1 test in 3.034s
2022-05-18T04:55:29.6812285Z 
2022-05-18T04:55:29.6812388Z OK
2022-05-18T04:55:29.6812526Z 
2022-05-18T04:55:29.6812660Z Generating XML reports...
2022-05-18T04:55:29.6853411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045526.xml
2022-05-18T04:55:31.0804467Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:31.0819441Z 
2022-05-18T04:55:31.0819762Z Running tests...
2022-05-18T04:55:31.0820239Z ----------------------------------------------------------------------
2022-05-18T04:55:32.6734639Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:32.7135725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63485
2022-05-18T04:55:32.7245236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63486
2022-05-18T04:55:33.8831206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:33.8831763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:33.8832553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:33.8833257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:33.8942111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:33.9844340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:34.2297178Z ok (3.147s)
2022-05-18T04:55:34.2297646Z 
2022-05-18T04:55:34.2298153Z ----------------------------------------------------------------------
2022-05-18T04:55:34.2298528Z Ran 1 test in 3.148s
2022-05-18T04:55:34.2298700Z 
2022-05-18T04:55:34.2298799Z OK
2022-05-18T04:55:34.2298916Z 
2022-05-18T04:55:34.2299058Z Generating XML reports...
2022-05-18T04:55:34.2339442Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045531.xml
2022-05-18T04:55:35.6212975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:35.6227143Z 
2022-05-18T04:55:35.6227385Z Running tests...
2022-05-18T04:55:35.6227982Z ----------------------------------------------------------------------
2022-05-18T04:55:35.6247694Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T04:55:35.6248575Z 
2022-05-18T04:55:35.6249088Z ----------------------------------------------------------------------
2022-05-18T04:55:35.6249531Z Ran 1 test in 0.002s
2022-05-18T04:55:35.6249709Z 
2022-05-18T04:55:35.6249826Z OK (skipped=1)
2022-05-18T04:55:35.6249984Z 
2022-05-18T04:55:35.6250114Z Generating XML reports...
2022-05-18T04:55:35.6284246Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045535.xml
2022-05-18T04:55:36.8729330Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:36.8744801Z 
2022-05-18T04:55:36.8745222Z Running tests...
2022-05-18T04:55:36.8746463Z ----------------------------------------------------------------------
2022-05-18T04:55:36.8767942Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T04:55:36.8768303Z 
2022-05-18T04:55:36.8768629Z ----------------------------------------------------------------------
2022-05-18T04:55:36.8768953Z Ran 1 test in 0.002s
2022-05-18T04:55:36.8769122Z 
2022-05-18T04:55:36.8769235Z OK (skipped=1)
2022-05-18T04:55:36.8769405Z 
2022-05-18T04:55:36.8769535Z Generating XML reports...
2022-05-18T04:55:36.8806958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045536.xml
2022-05-18T04:55:38.1330885Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:38.1345295Z 
2022-05-18T04:55:38.1345457Z Running tests...
2022-05-18T04:55:38.1346197Z ----------------------------------------------------------------------
2022-05-18T04:55:39.7135761Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:39.7536725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63664
2022-05-18T04:55:39.7645145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63665
2022-05-18T04:55:40.9329473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:40.9330306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:40.9331182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:40.9331903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:40.9437265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:41.0340355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:41.0549610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:55:41.0550110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:55:41.0550832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:55:41.0551529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:55:41.2698351Z ok (3.135s)
2022-05-18T04:55:41.2698565Z 
2022-05-18T04:55:41.2699031Z ----------------------------------------------------------------------
2022-05-18T04:55:41.2699563Z Ran 1 test in 3.135s
2022-05-18T04:55:41.2699735Z 
2022-05-18T04:55:41.2699830Z OK
2022-05-18T04:55:41.2699966Z 
2022-05-18T04:55:41.2700100Z Generating XML reports...
2022-05-18T04:55:41.2740023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045538.xml
2022-05-18T04:55:42.6694333Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:42.6709208Z 
2022-05-18T04:55:42.6709538Z Running tests...
2022-05-18T04:55:42.6709983Z ----------------------------------------------------------------------
2022-05-18T04:55:44.2466072Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:44.2867042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63783
2022-05-18T04:55:44.2974754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63784
2022-05-18T04:55:45.4550306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:45.4550852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:45.4551631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:45.4552346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:45.4660235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:45.5564358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:45.7024081Z skip: Skipped due to small world size. (3.031s)
2022-05-18T04:55:45.7024354Z 
2022-05-18T04:55:45.7025017Z ----------------------------------------------------------------------
2022-05-18T04:55:45.7025376Z Ran 1 test in 3.031s
2022-05-18T04:55:45.7025546Z 
2022-05-18T04:55:45.7025663Z OK (skipped=1)
2022-05-18T04:55:45.7025829Z 
2022-05-18T04:55:45.7025959Z Generating XML reports...
2022-05-18T04:55:45.7067544Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045542.xml
2022-05-18T04:55:47.0995248Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:47.1010623Z 
2022-05-18T04:55:47.1010971Z Running tests...
2022-05-18T04:55:47.1011406Z ----------------------------------------------------------------------
2022-05-18T04:55:48.6770164Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:48.7171177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63892
2022-05-18T04:55:48.7279813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63893
2022-05-18T04:55:49.9232953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:49.9233535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:49.9234338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:49.9235067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:49.9241246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:49.9241934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:50.1331433Z ok (3.032s)
2022-05-18T04:55:50.1331652Z 
2022-05-18T04:55:50.1332078Z ----------------------------------------------------------------------
2022-05-18T04:55:50.1332419Z Ran 1 test in 3.032s
2022-05-18T04:55:50.1332570Z 
2022-05-18T04:55:50.1332669Z OK
2022-05-18T04:55:50.1332807Z 
2022-05-18T04:55:50.1332944Z Generating XML reports...
2022-05-18T04:55:50.1373829Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045547.xml
2022-05-18T04:55:51.5377334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:51.5390680Z 
2022-05-18T04:55:51.5390972Z Running tests...
2022-05-18T04:55:51.5391432Z ----------------------------------------------------------------------
2022-05-18T04:55:53.0774237Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:53.1163695Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64001
2022-05-18T04:55:53.1271708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64002
2022-05-18T04:55:54.3367621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:54.3368185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:54.3369000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:54.3369691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:54.3376040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:54.3376536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:54.5321497Z ok (2.993s)
2022-05-18T04:55:54.5321828Z 
2022-05-18T04:55:54.5322349Z ----------------------------------------------------------------------
2022-05-18T04:55:54.5322699Z Ran 1 test in 2.993s
2022-05-18T04:55:54.5322867Z 
2022-05-18T04:55:54.5322965Z OK
2022-05-18T04:55:54.5323104Z 
2022-05-18T04:55:54.5323220Z Generating XML reports...
2022-05-18T04:55:54.5363586Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045551.xml
2022-05-18T04:55:55.9379316Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:55:55.9394295Z 
2022-05-18T04:55:55.9394781Z Running tests...
2022-05-18T04:55:55.9395233Z ----------------------------------------------------------------------
2022-05-18T04:55:57.5081203Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:55:57.5484648Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64110
2022-05-18T04:55:57.5592128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64111
2022-05-18T04:55:58.7323775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:55:58.7324316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:55:58.7325130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:58.7325832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:55:58.7332476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:55:58.7332978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:55:58.9643117Z ok (3.024s)
2022-05-18T04:55:58.9643344Z 
2022-05-18T04:55:58.9643739Z ----------------------------------------------------------------------
2022-05-18T04:55:58.9644090Z Ran 1 test in 3.025s
2022-05-18T04:55:58.9644259Z 
2022-05-18T04:55:58.9644336Z OK
2022-05-18T04:55:58.9644475Z 
2022-05-18T04:55:58.9644610Z Generating XML reports...
2022-05-18T04:55:58.9685223Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045555.xml
2022-05-18T04:56:00.3682259Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:00.3696878Z 
2022-05-18T04:56:00.3697308Z Running tests...
2022-05-18T04:56:00.3698091Z ----------------------------------------------------------------------
2022-05-18T04:56:01.9452050Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:01.9858720Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64219
2022-05-18T04:56:01.9968835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64220
2022-05-18T04:56:03.1423728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:03.1424284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:03.1425098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:03.1425802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:03.1431852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:03.1432804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:03.3018891Z ok (2.932s)
2022-05-18T04:56:03.3019103Z 
2022-05-18T04:56:03.3019946Z ----------------------------------------------------------------------
2022-05-18T04:56:03.3020329Z Ran 1 test in 2.932s
2022-05-18T04:56:03.3020503Z 
2022-05-18T04:56:03.3020602Z OK
2022-05-18T04:56:03.3020721Z 
2022-05-18T04:56:03.3020862Z Generating XML reports...
2022-05-18T04:56:03.3062543Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045600.xml
2022-05-18T04:56:04.7259496Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:04.7274183Z 
2022-05-18T04:56:04.7274723Z Running tests...
2022-05-18T04:56:04.7275224Z ----------------------------------------------------------------------
2022-05-18T04:56:06.3196275Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:06.3599848Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64332
2022-05-18T04:56:06.3709942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64333
2022-05-18T04:56:07.5419546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:07.5420117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:07.5420932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:07.5421635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:07.5426886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:07.5427409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:07.7758281Z ok (3.048s)
2022-05-18T04:56:07.7758997Z 
2022-05-18T04:56:07.7759750Z ----------------------------------------------------------------------
2022-05-18T04:56:07.7760181Z Ran 1 test in 3.048s
2022-05-18T04:56:07.7760353Z 
2022-05-18T04:56:07.7760452Z OK
2022-05-18T04:56:07.7760569Z 
2022-05-18T04:56:07.7760750Z Generating XML reports...
2022-05-18T04:56:07.7800442Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045604.xml
2022-05-18T04:56:09.1866041Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:09.1880748Z 
2022-05-18T04:56:09.1881039Z Running tests...
2022-05-18T04:56:09.1881478Z ----------------------------------------------------------------------
2022-05-18T04:56:10.7671436Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:10.8063085Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64445
2022-05-18T04:56:10.8170813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64446
2022-05-18T04:56:12.0014474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:12.0015037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:12.0015848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:12.0016530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:12.0122271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:12.1024127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:12.3224030Z ok (3.134s)
2022-05-18T04:56:12.3224245Z 
2022-05-18T04:56:12.3224860Z ----------------------------------------------------------------------
2022-05-18T04:56:12.3225219Z Ran 1 test in 3.134s
2022-05-18T04:56:12.3225386Z 
2022-05-18T04:56:12.3225501Z OK
2022-05-18T04:56:12.3225641Z 
2022-05-18T04:56:12.3225757Z Generating XML reports...
2022-05-18T04:56:12.3266662Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045609.xml
2022-05-18T04:56:13.6896485Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:13.6911303Z 
2022-05-18T04:56:13.6911694Z Running tests...
2022-05-18T04:56:13.6912196Z ----------------------------------------------------------------------
2022-05-18T04:56:13.6931258Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T04:56:13.6931582Z 
2022-05-18T04:56:13.6931875Z ----------------------------------------------------------------------
2022-05-18T04:56:13.6932449Z Ran 1 test in 0.002s
2022-05-18T04:56:13.6932674Z 
2022-05-18T04:56:13.6932788Z OK (skipped=1)
2022-05-18T04:56:13.6932952Z 
2022-05-18T04:56:13.6933061Z Generating XML reports...
2022-05-18T04:56:13.6968953Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045613.xml
2022-05-18T04:56:14.9500443Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:14.9514860Z 
2022-05-18T04:56:14.9515450Z Running tests...
2022-05-18T04:56:14.9515961Z ----------------------------------------------------------------------
2022-05-18T04:56:14.9535552Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T04:56:14.9536156Z 
2022-05-18T04:56:14.9536753Z ----------------------------------------------------------------------
2022-05-18T04:56:14.9537480Z Ran 1 test in 0.002s
2022-05-18T04:56:14.9537742Z 
2022-05-18T04:56:14.9537868Z OK (skipped=1)
2022-05-18T04:56:14.9538029Z 
2022-05-18T04:56:14.9538158Z Generating XML reports...
2022-05-18T04:56:14.9573725Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045614.xml
2022-05-18T04:56:16.2166610Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:16.2181026Z 
2022-05-18T04:56:16.2181406Z Running tests...
2022-05-18T04:56:16.2182348Z ----------------------------------------------------------------------
2022-05-18T04:56:16.2204396Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T04:56:16.2205056Z 
2022-05-18T04:56:16.2205616Z ----------------------------------------------------------------------
2022-05-18T04:56:16.2206061Z Ran 1 test in 0.002s
2022-05-18T04:56:16.2206512Z 
2022-05-18T04:56:16.2206625Z OK (skipped=1)
2022-05-18T04:56:16.2206784Z 
2022-05-18T04:56:16.2206914Z Generating XML reports...
2022-05-18T04:56:16.2241559Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045616.xml
2022-05-18T04:56:17.4766229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:17.4779841Z 
2022-05-18T04:56:17.4780140Z Running tests...
2022-05-18T04:56:17.4780588Z ----------------------------------------------------------------------
2022-05-18T04:56:19.0633761Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:19.1024334Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64657
2022-05-18T04:56:19.1131260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64658
2022-05-18T04:56:20.2933361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:20.2933990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:20.2934791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:20.2935500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:20.3040996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:20.3943775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:20.6184202Z ok (3.140s)
2022-05-18T04:56:20.6184429Z 
2022-05-18T04:56:20.6184824Z ----------------------------------------------------------------------
2022-05-18T04:56:20.6185157Z Ran 1 test in 3.140s
2022-05-18T04:56:20.6185325Z 
2022-05-18T04:56:20.6185453Z OK
2022-05-18T04:56:20.6185592Z 
2022-05-18T04:56:20.6185728Z Generating XML reports...
2022-05-18T04:56:20.6225892Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045617.xml
2022-05-18T04:56:22.0310747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:22.0325293Z 
2022-05-18T04:56:22.0325560Z Running tests...
2022-05-18T04:56:22.0325996Z ----------------------------------------------------------------------
2022-05-18T04:56:23.6063781Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:23.6466107Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64770
2022-05-18T04:56:23.6575552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64771
2022-05-18T04:56:24.8772064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:24.8772647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:24.8773442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:24.8774163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:24.8882432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:24.9782535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:25.1627207Z ok (3.130s)
2022-05-18T04:56:25.1627395Z 
2022-05-18T04:56:25.1627775Z ----------------------------------------------------------------------
2022-05-18T04:56:25.1628130Z Ran 1 test in 3.130s
2022-05-18T04:56:25.1628300Z 
2022-05-18T04:56:25.1628378Z OK
2022-05-18T04:56:25.1628517Z 
2022-05-18T04:56:25.1628853Z Generating XML reports...
2022-05-18T04:56:25.1670156Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045622.xml
2022-05-18T04:56:26.5644982Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:26.5659004Z 
2022-05-18T04:56:26.5659303Z Running tests...
2022-05-18T04:56:26.5659744Z ----------------------------------------------------------------------
2022-05-18T04:56:28.1296545Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:28.1694635Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64879
2022-05-18T04:56:28.1803550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64880
2022-05-18T04:56:29.3412947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:29.3413497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:29.3414317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:29.3417011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:29.3422453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:29.3426022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:29.5854312Z ok (3.019s)
2022-05-18T04:56:29.5854539Z 
2022-05-18T04:56:29.5854923Z ----------------------------------------------------------------------
2022-05-18T04:56:29.5855277Z Ran 1 test in 3.019s
2022-05-18T04:56:29.5855447Z 
2022-05-18T04:56:29.5855553Z OK
2022-05-18T04:56:29.5855694Z 
2022-05-18T04:56:29.5855839Z Generating XML reports...
2022-05-18T04:56:29.5895460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045626.xml
2022-05-18T04:56:30.9964034Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:30.9978823Z 
2022-05-18T04:56:30.9979392Z Running tests...
2022-05-18T04:56:30.9980525Z ----------------------------------------------------------------------
2022-05-18T04:56:32.5860701Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:32.6263289Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64992
2022-05-18T04:56:32.6373917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64993
2022-05-18T04:56:33.7868760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:33.7869304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:33.7870139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:33.7870839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:33.7976189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:33.8879174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:34.0428163Z ok (3.045s)
2022-05-18T04:56:34.0428518Z 
2022-05-18T04:56:34.0428936Z ----------------------------------------------------------------------
2022-05-18T04:56:34.0429288Z Ran 1 test in 3.045s
2022-05-18T04:56:34.0429457Z 
2022-05-18T04:56:34.0429556Z OK
2022-05-18T04:56:34.0429706Z 
2022-05-18T04:56:34.0429842Z Generating XML reports...
2022-05-18T04:56:34.0472065Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045630.xml
2022-05-18T04:56:35.4426827Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:35.4442773Z 
2022-05-18T04:56:35.4443159Z Running tests...
2022-05-18T04:56:35.4444079Z ----------------------------------------------------------------------
2022-05-18T04:56:37.0229935Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:37.0621178Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65105
2022-05-18T04:56:37.0726761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65106
2022-05-18T04:56:38.2316390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:38.2316965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:38.2317781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:38.2318811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:38.2424256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:38.3327971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:38.5780607Z ok (3.133s)
2022-05-18T04:56:38.5780834Z 
2022-05-18T04:56:38.5781245Z ----------------------------------------------------------------------
2022-05-18T04:56:38.5781577Z Ran 1 test in 3.134s
2022-05-18T04:56:38.5781747Z 
2022-05-18T04:56:38.5781851Z OK
2022-05-18T04:56:38.5781990Z 
2022-05-18T04:56:38.5782137Z Generating XML reports...
2022-05-18T04:56:38.5822242Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045635.xml
2022-05-18T04:56:40.0074016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:40.0088625Z 
2022-05-18T04:56:40.0089094Z Running tests...
2022-05-18T04:56:40.0089590Z ----------------------------------------------------------------------
2022-05-18T04:56:41.6282125Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:41.6682437Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65304
2022-05-18T04:56:41.6790473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65305
2022-05-18T04:56:42.8801813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:42.8802378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:42.8803190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:42.8803887Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:42.8909820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:42.9816925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:44.5867796Z ok (4.578s)
2022-05-18T04:56:44.5868016Z 
2022-05-18T04:56:44.5868381Z ----------------------------------------------------------------------
2022-05-18T04:56:44.5868732Z Ran 1 test in 4.578s
2022-05-18T04:56:44.5868902Z 
2022-05-18T04:56:44.5869005Z OK
2022-05-18T04:56:44.5869144Z 
2022-05-18T04:56:44.5869280Z Generating XML reports...
2022-05-18T04:56:44.5909461Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045640.xml
2022-05-18T04:56:45.9877226Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:45.9891388Z 
2022-05-18T04:56:45.9891895Z Running tests...
2022-05-18T04:56:45.9892385Z ----------------------------------------------------------------------
2022-05-18T04:56:47.5539355Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:47.5934035Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65505
2022-05-18T04:56:47.6043160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65506
2022-05-18T04:56:48.8097100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:48.8097652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:48.8098455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:48.8099177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:48.8105176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:48.8105666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:50.6121722Z ok (4.623s)
2022-05-18T04:56:50.6122037Z 
2022-05-18T04:56:50.6122453Z ----------------------------------------------------------------------
2022-05-18T04:56:50.6122803Z Ran 1 test in 4.623s
2022-05-18T04:56:50.6122973Z 
2022-05-18T04:56:50.6123051Z OK
2022-05-18T04:56:50.6123192Z 
2022-05-18T04:56:50.6125211Z Generating XML reports...
2022-05-18T04:56:50.6164147Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045645.xml
2022-05-18T04:56:52.0472526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:52.0488293Z 
2022-05-18T04:56:52.0488629Z Running tests...
2022-05-18T04:56:52.0489080Z ----------------------------------------------------------------------
2022-05-18T04:56:53.6409948Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:53.6812084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65620
2022-05-18T04:56:53.6923727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65621
2022-05-18T04:56:54.8802404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:54.8802972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:54.8803759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:54.8804468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:54.8810949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:54.8811496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:56:54.8912811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7jfwcqe
2022-05-18T04:56:54.8915487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7jfwcqe/_remote_module_non_scriptable.py
2022-05-18T04:56:54.8924496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7bh95yg
2022-05-18T04:56:54.8927454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7bh95yg/_remote_module_non_scriptable.py
2022-05-18T04:56:55.0974733Z ok (3.048s)
2022-05-18T04:56:55.0974967Z 
2022-05-18T04:56:55.0975383Z ----------------------------------------------------------------------
2022-05-18T04:56:55.0975738Z Ran 1 test in 3.049s
2022-05-18T04:56:55.0976188Z 
2022-05-18T04:56:55.0976297Z OK
2022-05-18T04:56:55.0976436Z 
2022-05-18T04:56:55.0976579Z Generating XML reports...
2022-05-18T04:56:55.1016482Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045652.xml
2022-05-18T04:56:56.5090636Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:56:56.5105798Z 
2022-05-18T04:56:56.5106161Z Running tests...
2022-05-18T04:56:56.5106825Z ----------------------------------------------------------------------
2022-05-18T04:56:58.1014074Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:56:58.1408160Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65733
2022-05-18T04:56:58.1515157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65734
2022-05-18T04:56:59.3790495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:56:59.3791067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:56:59.3791871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:59.3792582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:56:59.3798470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:56:59.3799345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:57:00.6779275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptxke0q1h
2022-05-18T04:57:00.6780120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptxke0q1h/_remote_module_non_scriptable.py
2022-05-18T04:57:00.7020718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zr7zoph
2022-05-18T04:57:00.7022924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zr7zoph/_remote_module_non_scriptable.py
2022-05-18T04:57:00.9590820Z ok (4.448s)
2022-05-18T04:57:00.9591079Z 
2022-05-18T04:57:00.9591759Z ----------------------------------------------------------------------
2022-05-18T04:57:00.9592115Z Ran 1 test in 4.448s
2022-05-18T04:57:00.9592286Z 
2022-05-18T04:57:00.9592384Z OK
2022-05-18T04:57:00.9592521Z 
2022-05-18T04:57:00.9593936Z Generating XML reports...
2022-05-18T04:57:00.9632964Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045656.xml
2022-05-18T04:57:02.3767599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:57:02.3783265Z 
2022-05-18T04:57:02.3783661Z Running tests...
2022-05-18T04:57:02.3784097Z ----------------------------------------------------------------------
2022-05-18T04:57:03.9599695Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:04.0008211Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65844
2022-05-18T04:57:04.0117624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65845
2022-05-18T04:57:05.1900674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:57:05.1901276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:57:05.1902079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:05.1902783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:05.1908725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:57:05.1909453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:57:06.4893220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbr57mbfa
2022-05-18T04:57:06.4894308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbr57mbfa/_remote_module_non_scriptable.py
2022-05-18T04:57:06.4949854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7uzxulok
2022-05-18T04:57:06.4952382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7uzxulok/_remote_module_non_scriptable.py
2022-05-18T04:57:06.6740278Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:57:06.6782356Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:57:07.0208414Z ok (4.642s)
2022-05-18T04:57:07.0208636Z 
2022-05-18T04:57:07.0209030Z ----------------------------------------------------------------------
2022-05-18T04:57:07.0209375Z Ran 1 test in 4.643s
2022-05-18T04:57:07.0209522Z 
2022-05-18T04:57:07.0209618Z OK
2022-05-18T04:57:07.0209781Z 
2022-05-18T04:57:07.0209918Z Generating XML reports...
2022-05-18T04:57:07.0250455Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045702.xml
2022-05-18T04:57:08.4418478Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:57:08.4433535Z 
2022-05-18T04:57:08.4433682Z Running tests...
2022-05-18T04:57:08.4434158Z ----------------------------------------------------------------------
2022-05-18T04:57:10.0088846Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:10.0493297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65959
2022-05-18T04:57:10.0602273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65960
2022-05-18T04:57:11.2133998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:57:11.2134580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:57:11.2135358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:11.2136075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:11.2142038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:57:11.2142663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:57:11.2250338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:57:11.2250863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:57:11.2251564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:57:11.2252543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:57:11.2358680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:57:11.2359187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:57:11.2359876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:57:11.2360573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:57:12.5040559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmd5e3p7
2022-05-18T04:57:12.5041454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmd5e3p7/_remote_module_non_scriptable.py
2022-05-18T04:57:12.5467293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0v8dw4d3
2022-05-18T04:57:12.5469109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0v8dw4d3/_remote_module_non_scriptable.py
2022-05-18T04:57:17.8763252Z ok (9.433s)
2022-05-18T04:57:17.8763483Z 
2022-05-18T04:57:17.8763918Z ----------------------------------------------------------------------
2022-05-18T04:57:17.8764266Z Ran 1 test in 9.433s
2022-05-18T04:57:17.8764431Z 
2022-05-18T04:57:17.8764508Z OK
2022-05-18T04:57:17.8764642Z 
2022-05-18T04:57:17.8764775Z Generating XML reports...
2022-05-18T04:57:17.8805302Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045708.xml
2022-05-18T04:57:19.2980867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:57:19.2995077Z 
2022-05-18T04:57:19.2995529Z Running tests...
2022-05-18T04:57:19.2996048Z ----------------------------------------------------------------------
2022-05-18T04:57:20.8663211Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:20.9067889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66082
2022-05-18T04:57:20.9177953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66083
2022-05-18T04:57:22.1152861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:57:22.1153409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:57:22.1154211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:22.1154893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:22.1260748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:57:22.2163055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:57:22.2276271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:57:22.2276811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:57:22.2277515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:57:22.2278441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:57:22.2485307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:57:22.2485806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:57:22.2486737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:57:22.2487481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:57:23.5097307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbq09s9x
2022-05-18T04:57:23.5097986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbq09s9x/_remote_module_non_scriptable.py
2022-05-18T04:57:23.5413299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz11_iarz
2022-05-18T04:57:23.5415346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz11_iarz/_remote_module_non_scriptable.py
2022-05-18T04:57:28.8349613Z ok (9.535s)
2022-05-18T04:57:28.8349846Z 
2022-05-18T04:57:28.8350255Z ----------------------------------------------------------------------
2022-05-18T04:57:28.8350583Z Ran 1 test in 9.535s
2022-05-18T04:57:28.8350774Z 
2022-05-18T04:57:28.8350871Z OK
2022-05-18T04:57:28.8351010Z 
2022-05-18T04:57:28.8351150Z Generating XML reports...
2022-05-18T04:57:28.8392003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045719.xml
2022-05-18T04:57:29.2415605Z Running distributed tests for the gloo backend with file init_method
2022-05-18T04:57:29.2417637Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:57:29.241427]
2022-05-18T04:57:30.4007555Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:57:30.4049447Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-05-18T04:57:30.4083873Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4084378Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4084808Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4085236Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4085672Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4086149Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4086645Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4087124Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4087641Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4088185Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4088750Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4089261Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4089796Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4090319Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4090807Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4091281Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4091746Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4092181Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4092600Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4093048Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4093531Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4094137Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4094550Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4094947Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4095378Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4095787Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4096205Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4096628Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4097030Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4097417Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4097814Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4098220Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4098599Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4098993Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4099402Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4099808Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4100224Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4100654Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4101076Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4101611Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4102059Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4102490Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4102911Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4103347Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4103785Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4104187Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4104630Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4105069Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4105483Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4105887Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4106309Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4106737Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4107131Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4107547Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4107960Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4108361Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4108739Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4109138Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4109537Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4109903Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4110283Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4110673Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4111063Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4124223Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4124726Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4125107Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4125500Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4125902Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4126284Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4126684Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4127102Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4127484Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4127867Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4128252Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4128647Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4129031Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4129438Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4129839Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4130212Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4130620Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4131054Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4131477Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4132032Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4132491Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4132956Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4133394Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4133839Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4134275Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4134695Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4135128Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4135583Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4136052Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4136497Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4136956Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4137414Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4137848Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4138228Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4138612Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4138986Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4139335Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4139717Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4140116Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4140495Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4140884Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4141288Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4141758Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4142155Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4142558Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4142969Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4143378Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4143800Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4144214Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4144609Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4145002Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4145420Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4145848Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4146249Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4146641Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4147007Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4147374Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4147772Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4148166Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4148563Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4149006Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4149592Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4150048Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4150448Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4150870Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4151312Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4151769Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4152224Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4152669Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4153094Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4153525Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4153939Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4154314Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4154694Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4155115Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4155529Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4155970Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4156395Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4156814Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4157274Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4157768Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4159055Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4159780Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4160410Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4161008Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4161625Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4162232Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4162845Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4163437Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4163986Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4164489Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4164941Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4165313Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4165716Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4166216Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4166600Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4167030Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4167481Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4167945Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4168440Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4168866Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4169264Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4169663Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4170103Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4170539Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4170963Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4171362Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4171791Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4172227Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4172637Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4173052Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4173463Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4173886Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4174276Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4174699Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4175163Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4175579Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4175966Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4176428Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4176848Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4177270Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4177657Z test_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4178024Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4178378Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4178762Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4179145Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4179508Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4179900Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4180288Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4180641Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4181006Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4181389Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4181774Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4182170Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4182544Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4182897Z test_isend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4183263Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4183669Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4184155Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4184596Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4185048Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4185473Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4185904Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4186329Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4186759Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4187186Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4187590Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4188008Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4188427Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4188825Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4189224Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4189635Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4190108Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4190582Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4191056Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4191506Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4191952Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4192411Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4192850Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4193275Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4193689Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4194186Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4194651Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4195115Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4195628Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4196098Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4196497Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4196893Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4197298Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4197690Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4198608Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4199124Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4199523Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4199877Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4200243Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4200621Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4201003Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4201356Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4201723Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4202212Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4202585Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4202953Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4203326Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4203692Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4204071Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4204458Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4204858Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4205225Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4205611Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4205986Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4206350Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4206784Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4207237Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4207654Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4208060Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4208469Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4208899Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4209300Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4209699Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4210115Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4210537Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4210957Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4211365Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4211757Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4212165Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4212630Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4213074Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4213514Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:30.4213962Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:57:31.5212628Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:57:31.5226428Z 
2022-05-18T04:57:31.5226638Z Running tests...
2022-05-18T04:57:31.5227087Z ----------------------------------------------------------------------
2022-05-18T04:57:33.0658266Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:33.1049824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66238
2022-05-18T04:57:33.1156633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66239
2022-05-18T04:57:34.3173886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:57:34.3174503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:57:34.3175298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:34.3176246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:34.3282023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:57:34.4187514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:57:35.5677958Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:57:35.5679261Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:57:35.6827313Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:57:35.6828340Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:57:36.5407294Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:57:36.5408338Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:57:36.5409029Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:57:36.5409865Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:57:36.5549012Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:57:36.5550203Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:57:36.5550862Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:57:36.5551704Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:57:36.5693093Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:57:36.5693809Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-05-18T04:57:36.5694662Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:57:36.5695582Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-05-18T04:57:36.9248349Z ok (5.402s)
2022-05-18T04:57:36.9248584Z 
2022-05-18T04:57:36.9248974Z ----------------------------------------------------------------------
2022-05-18T04:57:36.9249352Z Ran 1 test in 5.402s
2022-05-18T04:57:36.9249522Z 
2022-05-18T04:57:36.9249623Z OK
2022-05-18T04:57:36.9249765Z 
2022-05-18T04:57:36.9249904Z Generating XML reports...
2022-05-18T04:57:36.9289996Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045731.xml
2022-05-18T04:57:38.3401224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:57:38.3416095Z 
2022-05-18T04:57:38.3416375Z Running tests...
2022-05-18T04:57:38.3416805Z ----------------------------------------------------------------------
2022-05-18T04:57:38.3459208Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-05-18T04:57:38.3459556Z 
2022-05-18T04:57:38.3459851Z ----------------------------------------------------------------------
2022-05-18T04:57:38.3460484Z Ran 1 test in 0.004s
2022-05-18T04:57:38.3460629Z 
2022-05-18T04:57:38.3460744Z OK (skipped=1)
2022-05-18T04:57:38.3460903Z 
2022-05-18T04:57:38.3461032Z Generating XML reports...
2022-05-18T04:57:38.3499231Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045738.xml
2022-05-18T04:57:39.6019185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:57:39.6033265Z 
2022-05-18T04:57:39.6033745Z Running tests...
2022-05-18T04:57:39.6034241Z ----------------------------------------------------------------------
2022-05-18T04:57:41.1842986Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:41.2241892Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66383
2022-05-18T04:57:41.2349961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66384
2022-05-18T04:57:42.4081139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:57:42.4081696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:57:42.4082501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:42.4083204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:42.4191430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:57:42.5091983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:57:42.7402855Z ok (3.136s)
2022-05-18T04:57:42.7403048Z 
2022-05-18T04:57:42.7403578Z ----------------------------------------------------------------------
2022-05-18T04:57:42.7403907Z Ran 1 test in 3.137s
2022-05-18T04:57:42.7404078Z 
2022-05-18T04:57:42.7404200Z OK
2022-05-18T04:57:42.7404340Z 
2022-05-18T04:57:42.7404474Z Generating XML reports...
2022-05-18T04:57:42.7445172Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045739.xml
2022-05-18T04:57:44.1397684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:57:44.1413076Z 
2022-05-18T04:57:44.1413353Z Running tests...
2022-05-18T04:57:44.1413782Z ----------------------------------------------------------------------
2022-05-18T04:57:45.7190068Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:45.7346589Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.593s)
2022-05-18T04:57:45.7347257Z 
2022-05-18T04:57:45.7347536Z ----------------------------------------------------------------------
2022-05-18T04:57:45.7347855Z Ran 1 test in 1.593s
2022-05-18T04:57:45.7348020Z 
2022-05-18T04:57:45.7348131Z OK (skipped=1)
2022-05-18T04:57:45.7348292Z 
2022-05-18T04:57:45.7348421Z Generating XML reports...
2022-05-18T04:57:45.7382338Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045744.xml
2022-05-18T04:57:47.1166896Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:57:47.1181655Z 
2022-05-18T04:57:47.1182043Z Running tests...
2022-05-18T04:57:47.1182547Z ----------------------------------------------------------------------
2022-05-18T04:57:48.6838609Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:48.7241992Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66526
2022-05-18T04:57:48.7349301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66527
2022-05-18T04:57:49.9197681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:57:49.9198620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:57:49.9199430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:49.9200136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:49.9305134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:57:49.9406172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43g6om11
2022-05-18T04:57:49.9408753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43g6om11/_remote_module_non_scriptable.py
2022-05-18T04:57:50.0208013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:57:50.0311615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lgdn7x6
2022-05-18T04:57:50.0314517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lgdn7x6/_remote_module_non_scriptable.py
2022-05-18T04:57:50.0515599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:57:50.0516108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:57:50.2399540Z ok (3.121s)
2022-05-18T04:57:50.2399776Z 
2022-05-18T04:57:50.2400199Z ----------------------------------------------------------------------
2022-05-18T04:57:50.2400526Z Ran 1 test in 3.122s
2022-05-18T04:57:50.2400700Z 
2022-05-18T04:57:50.2400798Z OK
2022-05-18T04:57:50.2400936Z 
2022-05-18T04:57:50.2401077Z Generating XML reports...
2022-05-18T04:57:50.2442755Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045747.xml
2022-05-18T04:57:51.6455785Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:57:51.6470519Z 
2022-05-18T04:57:51.6470669Z Running tests...
2022-05-18T04:57:51.6471654Z ----------------------------------------------------------------------
2022-05-18T04:57:53.2425268Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:53.2830943Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66639
2022-05-18T04:57:53.2940090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66640
2022-05-18T04:57:54.4784727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:57:54.4785822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:57:54.4787196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:54.4788649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:54.4792730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:57:54.4794624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:57:54.4897010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgyzc7svy
2022-05-18T04:57:54.4899066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgyzc7svy/_remote_module_non_scriptable.py
2022-05-18T04:57:54.4904178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8uakdgk7
2022-05-18T04:57:54.4907039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8uakdgk7/_remote_module_non_scriptable.py
2022-05-18T04:57:54.5113177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:57:54.5114204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:57:54.6990869Z ok (3.052s)
2022-05-18T04:57:54.6991081Z 
2022-05-18T04:57:54.6991480Z ----------------------------------------------------------------------
2022-05-18T04:57:54.6991816Z Ran 1 test in 3.052s
2022-05-18T04:57:54.6991986Z 
2022-05-18T04:57:54.6992085Z OK
2022-05-18T04:57:54.6992222Z 
2022-05-18T04:57:54.6992358Z Generating XML reports...
2022-05-18T04:57:54.7033062Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045751.xml
2022-05-18T04:57:56.1068169Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:57:56.1084515Z 
2022-05-18T04:57:56.1085066Z Running tests...
2022-05-18T04:57:56.1085587Z ----------------------------------------------------------------------
2022-05-18T04:57:57.7023923Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:57:57.7426270Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66752
2022-05-18T04:57:57.7534749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66753
2022-05-18T04:57:58.9333577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:57:58.9334388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:57:58.9335216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:58.9336174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:57:58.9342123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:57:58.9343650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:00.2213307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlbnadum
2022-05-18T04:58:00.2213949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlbnadum/_remote_module_non_scriptable.py
2022-05-18T04:58:00.2439892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo60gln2x
2022-05-18T04:58:00.2443210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo60gln2x/_remote_module_non_scriptable.py
2022-05-18T04:58:00.9946251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:00.9946794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:01.0194386Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:01.0195225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:01.0515222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:01.0515980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:01.0759327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:01.0759909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:01.2053227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:01.2054105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:01.2296562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:01.2297354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:01.6631265Z ok (5.554s)
2022-05-18T04:58:01.6631683Z 
2022-05-18T04:58:01.6632433Z ----------------------------------------------------------------------
2022-05-18T04:58:01.6632822Z Ran 1 test in 5.555s
2022-05-18T04:58:01.6632987Z 
2022-05-18T04:58:01.6633092Z OK
2022-05-18T04:58:01.6633233Z 
2022-05-18T04:58:01.6633366Z Generating XML reports...
2022-05-18T04:58:01.6674265Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045756.xml
2022-05-18T04:58:03.0906365Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:03.0921492Z 
2022-05-18T04:58:03.0921848Z Running tests...
2022-05-18T04:58:03.0922305Z ----------------------------------------------------------------------
2022-05-18T04:58:04.6783797Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:04.7178111Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66867
2022-05-18T04:58:04.7287837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66868
2022-05-18T04:58:05.9176158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:05.9176698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:05.9177517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:05.9178225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:05.9284222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:06.0190812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:07.1859645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5yzpbvy3
2022-05-18T04:58:07.1860253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5yzpbvy3/_remote_module_non_scriptable.py
2022-05-18T04:58:07.2796637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoen9kyh2
2022-05-18T04:58:07.2797776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoen9kyh2/_remote_module_non_scriptable.py
2022-05-18T04:58:07.3045292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:07.3045785Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:07.3218479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:07.3218996Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:07.6366178Z ok (4.544s)
2022-05-18T04:58:07.6366393Z 
2022-05-18T04:58:07.6366781Z ----------------------------------------------------------------------
2022-05-18T04:58:07.6367128Z Ran 1 test in 4.544s
2022-05-18T04:58:07.6367295Z 
2022-05-18T04:58:07.6367394Z OK
2022-05-18T04:58:07.6367530Z 
2022-05-18T04:58:07.6367680Z Generating XML reports...
2022-05-18T04:58:07.6408831Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045803.xml
2022-05-18T04:58:09.0511425Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:09.0526794Z 
2022-05-18T04:58:09.0527335Z Running tests...
2022-05-18T04:58:09.0527831Z ----------------------------------------------------------------------
2022-05-18T04:58:10.6256930Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:10.6646823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66982
2022-05-18T04:58:10.6753656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66983
2022-05-18T04:58:11.8351563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:11.8352161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:11.8352967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:11.8353674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:11.8359519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:11.8360017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:13.0994932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83vsy2db
2022-05-18T04:58:13.0995546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83vsy2db/_remote_module_non_scriptable.py
2022-05-18T04:58:13.1071697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpat4qx3st
2022-05-18T04:58:13.1074474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpat4qx3st/_remote_module_non_scriptable.py
2022-05-18T04:58:13.1331489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:13.1331979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:13.1528222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:13.1528758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:13.4831485Z ok (4.430s)
2022-05-18T04:58:13.4831689Z 
2022-05-18T04:58:13.4832118Z ----------------------------------------------------------------------
2022-05-18T04:58:13.4832481Z Ran 1 test in 4.430s
2022-05-18T04:58:13.4832646Z 
2022-05-18T04:58:13.4832744Z OK
2022-05-18T04:58:13.4832885Z 
2022-05-18T04:58:13.4833004Z Generating XML reports...
2022-05-18T04:58:13.4874652Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045809.xml
2022-05-18T04:58:14.9043615Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:14.9058552Z 
2022-05-18T04:58:14.9058827Z Running tests...
2022-05-18T04:58:14.9059284Z ----------------------------------------------------------------------
2022-05-18T04:58:16.4772809Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:16.5162180Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67097
2022-05-18T04:58:16.5268165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67098
2022-05-18T04:58:17.6841326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:17.6841902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:17.6842719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:17.6843428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:17.6849179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:17.6849682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:18.9474404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0bxx6vul
2022-05-18T04:58:18.9475372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0bxx6vul/_remote_module_non_scriptable.py
2022-05-18T04:58:18.9820520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsz9iovd0
2022-05-18T04:58:18.9823111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsz9iovd0/_remote_module_non_scriptable.py
2022-05-18T04:58:19.0037775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:19.0038601Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:19.5348675Z ok (4.628s)
2022-05-18T04:58:19.5348891Z 
2022-05-18T04:58:19.5349302Z ----------------------------------------------------------------------
2022-05-18T04:58:19.5349648Z Ran 1 test in 4.629s
2022-05-18T04:58:19.5349819Z 
2022-05-18T04:58:19.5349917Z OK
2022-05-18T04:58:19.5350054Z 
2022-05-18T04:58:19.5350172Z Generating XML reports...
2022-05-18T04:58:19.5390420Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045814.xml
2022-05-18T04:58:20.9266779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:20.9282581Z 
2022-05-18T04:58:20.9283016Z Running tests...
2022-05-18T04:58:20.9283567Z ----------------------------------------------------------------------
2022-05-18T04:58:22.4694367Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:22.5092665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67212
2022-05-18T04:58:22.5199608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67213
2022-05-18T04:58:23.7088688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:23.7089230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:23.7090050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:23.7090958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:23.7195709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:23.8102778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:24.9617184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilvl8tsn
2022-05-18T04:58:24.9617809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilvl8tsn/_remote_module_non_scriptable.py
2022-05-18T04:58:25.0347316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_5bbh3k9
2022-05-18T04:58:25.0348574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_5bbh3k9/_remote_module_non_scriptable.py
2022-05-18T04:58:25.7687526Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:25.7688089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:25.7928705Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:25.7929223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:26.1290629Z ok (5.200s)
2022-05-18T04:58:26.1290866Z 
2022-05-18T04:58:26.1291262Z ----------------------------------------------------------------------
2022-05-18T04:58:26.1291590Z Ran 1 test in 5.201s
2022-05-18T04:58:26.1291761Z 
2022-05-18T04:58:26.1291866Z OK
2022-05-18T04:58:26.1292003Z 
2022-05-18T04:58:26.1292144Z Generating XML reports...
2022-05-18T04:58:26.1333067Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045820.xml
2022-05-18T04:58:27.5312232Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:27.5326031Z 
2022-05-18T04:58:27.5326233Z Running tests...
2022-05-18T04:58:27.5326680Z ----------------------------------------------------------------------
2022-05-18T04:58:29.0759494Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:29.1160316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67327
2022-05-18T04:58:29.1265580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67328
2022-05-18T04:58:30.3540207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:30.3540816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:30.3541614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:30.3542341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:30.3648493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:30.4555087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:31.6224741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0jshw_lo
2022-05-18T04:58:31.6225662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0jshw_lo/_remote_module_non_scriptable.py
2022-05-18T04:58:31.6963349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdi8vevsy
2022-05-18T04:58:31.6964465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdi8vevsy/_remote_module_non_scriptable.py
2022-05-18T04:58:31.8936586Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:31.8937153Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:31.9134949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:31.9135697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:32.2348552Z ok (4.702s)
2022-05-18T04:58:32.2348863Z 
2022-05-18T04:58:32.2349328Z ----------------------------------------------------------------------
2022-05-18T04:58:32.2349676Z Ran 1 test in 4.702s
2022-05-18T04:58:32.2349851Z 
2022-05-18T04:58:32.2349949Z OK
2022-05-18T04:58:32.2350087Z 
2022-05-18T04:58:32.2350222Z Generating XML reports...
2022-05-18T04:58:32.2392142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045827.xml
2022-05-18T04:58:33.6345312Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:33.6358774Z 
2022-05-18T04:58:33.6358973Z Running tests...
2022-05-18T04:58:33.6359424Z ----------------------------------------------------------------------
2022-05-18T04:58:35.1727865Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:35.2116880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67442
2022-05-18T04:58:35.2227456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67443
2022-05-18T04:58:36.3605666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:36.3606229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:36.3607026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:36.3607997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:36.3714948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:36.4619881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:37.6316118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwc7jrs3
2022-05-18T04:58:37.6317294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwc7jrs3/_remote_module_non_scriptable.py
2022-05-18T04:58:37.6815697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhdik24f
2022-05-18T04:58:37.6817203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhdik24f/_remote_module_non_scriptable.py
2022-05-18T04:58:37.7053454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:37.7053981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:37.7228694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:37.7229639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:58:38.0302012Z ok (4.394s)
2022-05-18T04:58:38.0302323Z 
2022-05-18T04:58:38.0302842Z ----------------------------------------------------------------------
2022-05-18T04:58:38.0303191Z Ran 1 test in 4.394s
2022-05-18T04:58:38.0303361Z 
2022-05-18T04:58:38.0303459Z OK
2022-05-18T04:58:38.0303596Z 
2022-05-18T04:58:38.0303715Z Generating XML reports...
2022-05-18T04:58:38.0344431Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045833.xml
2022-05-18T04:58:39.4214071Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:39.4227596Z 
2022-05-18T04:58:39.4227721Z Running tests...
2022-05-18T04:58:39.4228480Z ----------------------------------------------------------------------
2022-05-18T04:58:40.9755926Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:40.9905942Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.567s)
2022-05-18T04:58:40.9906677Z 
2022-05-18T04:58:40.9906988Z ----------------------------------------------------------------------
2022-05-18T04:58:40.9907324Z Ran 1 test in 1.568s
2022-05-18T04:58:40.9907500Z 
2022-05-18T04:58:40.9907618Z OK (skipped=1)
2022-05-18T04:58:40.9907785Z 
2022-05-18T04:58:40.9907919Z Generating XML reports...
2022-05-18T04:58:40.9938851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045839.xml
2022-05-18T04:58:42.3695868Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:42.3710606Z 
2022-05-18T04:58:42.3711233Z Running tests...
2022-05-18T04:58:42.3711905Z ----------------------------------------------------------------------
2022-05-18T04:58:43.9399511Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:43.9801375Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67591
2022-05-18T04:58:43.9909438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67592
2022-05-18T04:58:45.1515991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:45.1516572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:45.1517618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:45.1518840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:45.1623884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:45.2527458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:45.4962031Z ok (3.125s)
2022-05-18T04:58:45.4962254Z 
2022-05-18T04:58:45.4962643Z ----------------------------------------------------------------------
2022-05-18T04:58:45.4962967Z Ran 1 test in 3.125s
2022-05-18T04:58:45.4963143Z 
2022-05-18T04:58:45.4963245Z OK
2022-05-18T04:58:45.4963383Z 
2022-05-18T04:58:45.4963518Z Generating XML reports...
2022-05-18T04:58:45.5004561Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045842.xml
2022-05-18T04:58:46.9327952Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:46.9342606Z 
2022-05-18T04:58:46.9343039Z Running tests...
2022-05-18T04:58:46.9343562Z ----------------------------------------------------------------------
2022-05-18T04:58:48.5147305Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:48.5302822Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.596s)
2022-05-18T04:58:48.5303476Z 
2022-05-18T04:58:48.5303760Z ----------------------------------------------------------------------
2022-05-18T04:58:48.5304116Z Ran 1 test in 1.596s
2022-05-18T04:58:48.5304283Z 
2022-05-18T04:58:48.5304395Z OK (skipped=1)
2022-05-18T04:58:48.5304535Z 
2022-05-18T04:58:48.5304668Z Generating XML reports...
2022-05-18T04:58:48.5338436Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045846.xml
2022-05-18T04:58:49.9061792Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:49.9076353Z 
2022-05-18T04:58:49.9076763Z Running tests...
2022-05-18T04:58:49.9077240Z ----------------------------------------------------------------------
2022-05-18T04:58:51.5115336Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:51.5507193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67734
2022-05-18T04:58:51.5617583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67735
2022-05-18T04:58:52.7463670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:52.7464245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:52.7465075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:52.7465778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:52.7472218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:58:52.7472755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:54.3693346Z ok (4.461s)
2022-05-18T04:58:54.3693601Z 
2022-05-18T04:58:54.3694208Z ----------------------------------------------------------------------
2022-05-18T04:58:54.3694568Z Ran 1 test in 4.462s
2022-05-18T04:58:54.3695016Z 
2022-05-18T04:58:54.3695114Z OK
2022-05-18T04:58:54.3695232Z 
2022-05-18T04:58:54.3695368Z Generating XML reports...
2022-05-18T04:58:54.3735239Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045849.xml
2022-05-18T04:58:55.7995668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:55.8011349Z 
2022-05-18T04:58:55.8011813Z Running tests...
2022-05-18T04:58:55.8012304Z ----------------------------------------------------------------------
2022-05-18T04:58:55.8034656Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-05-18T04:58:55.8035187Z 
2022-05-18T04:58:55.8035489Z ----------------------------------------------------------------------
2022-05-18T04:58:55.8035830Z Ran 1 test in 0.002s
2022-05-18T04:58:55.8035997Z 
2022-05-18T04:58:55.8036096Z OK (skipped=1)
2022-05-18T04:58:55.8036258Z 
2022-05-18T04:58:55.8036402Z Generating XML reports...
2022-05-18T04:58:55.8073205Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045855.xml
2022-05-18T04:58:57.0765498Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:58:57.0780082Z 
2022-05-18T04:58:57.0780623Z Running tests...
2022-05-18T04:58:57.0781123Z ----------------------------------------------------------------------
2022-05-18T04:58:57.0798970Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:58:58.6632833Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:58:58.7023392Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67878
2022-05-18T04:58:58.7131421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67879
2022-05-18T04:58:59.9225080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:58:59.9225655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:58:59.9226463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:59.9227378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:58:59.9332736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:58:59.9441728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo47x9rk8
2022-05-18T04:58:59.9444351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo47x9rk8/_remote_module_non_scriptable.py
2022-05-18T04:59:00.0235463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:00.0349005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07g894zv
2022-05-18T04:59:00.0351817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07g894zv/_remote_module_non_scriptable.py
2022-05-18T04:59:00.2184246Z ok (3.140s)
2022-05-18T04:59:00.2184483Z 
2022-05-18T04:59:00.2184910Z ----------------------------------------------------------------------
2022-05-18T04:59:00.2185246Z Ran 1 test in 3.140s
2022-05-18T04:59:00.2185416Z 
2022-05-18T04:59:00.2185520Z OK
2022-05-18T04:59:00.2185658Z 
2022-05-18T04:59:00.2185796Z Generating XML reports...
2022-05-18T04:59:00.2226927Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045857.xml
2022-05-18T04:59:01.6420648Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:01.6435236Z 
2022-05-18T04:59:01.6435556Z Running tests...
2022-05-18T04:59:01.6436274Z ----------------------------------------------------------------------
2022-05-18T04:59:01.6459430Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:59:03.2380623Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:03.2774370Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67991
2022-05-18T04:59:03.2883083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67992
2022-05-18T04:59:04.4443943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:04.4444513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:04.4445285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:04.4445989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:04.4551975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:04.4659135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0c_why7s
2022-05-18T04:59:04.4661487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0c_why7s/_remote_module_non_scriptable.py
2022-05-18T04:59:04.5453721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:04.5563749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbqofd1a
2022-05-18T04:59:04.5566396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbqofd1a/_remote_module_non_scriptable.py
2022-05-18T04:59:04.5815983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:04.5816504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:04.7938830Z ok (3.150s)
2022-05-18T04:59:04.7939072Z 
2022-05-18T04:59:04.7939462Z ----------------------------------------------------------------------
2022-05-18T04:59:04.7939804Z Ran 1 test in 3.150s
2022-05-18T04:59:04.7939978Z 
2022-05-18T04:59:04.7940055Z OK
2022-05-18T04:59:04.7940193Z 
2022-05-18T04:59:04.7940328Z Generating XML reports...
2022-05-18T04:59:04.7981267Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045901.xml
2022-05-18T04:59:06.2188329Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:06.2203330Z 
2022-05-18T04:59:06.2203782Z Running tests...
2022-05-18T04:59:06.2204265Z ----------------------------------------------------------------------
2022-05-18T04:59:06.2227892Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:59:07.8055933Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:07.8451599Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68104
2022-05-18T04:59:07.8562743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68105
2022-05-18T04:59:09.0257854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:09.0261485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:09.0262301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:09.0359663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:09.0366455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:09.0474052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppz2v2olu
2022-05-18T04:59:09.0476853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppz2v2olu/_remote_module_non_scriptable.py
2022-05-18T04:59:09.1274129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:09.1381582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs8zuoa4
2022-05-18T04:59:09.1384254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs8zuoa4/_remote_module_non_scriptable.py
2022-05-18T04:59:09.1628027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:09.1628559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:59:09.3616628Z ok (3.141s)
2022-05-18T04:59:09.3616843Z 
2022-05-18T04:59:09.3617230Z ----------------------------------------------------------------------
2022-05-18T04:59:09.3617556Z Ran 1 test in 3.141s
2022-05-18T04:59:09.3617725Z 
2022-05-18T04:59:09.3617850Z OK
2022-05-18T04:59:09.3617990Z 
2022-05-18T04:59:09.3618129Z Generating XML reports...
2022-05-18T04:59:09.3658785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045906.xml
2022-05-18T04:59:10.7294838Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:10.7308757Z 
2022-05-18T04:59:10.7309110Z Running tests...
2022-05-18T04:59:10.7309564Z ----------------------------------------------------------------------
2022-05-18T04:59:10.7328543Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:59:12.3094846Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:12.3483087Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68217
2022-05-18T04:59:12.3589705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68218
2022-05-18T04:59:13.5528988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:13.5529535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:13.5530555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:13.5531285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:13.5637346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:13.5744260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvc0agdgq
2022-05-18T04:59:13.5746963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvc0agdgq/_remote_module_non_scriptable.py
2022-05-18T04:59:13.6538896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:13.6651793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph85ek5z9
2022-05-18T04:59:13.6654456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph85ek5z9/_remote_module_non_scriptable.py
2022-05-18T04:59:13.9644894Z ok (3.233s)
2022-05-18T04:59:13.9645127Z 
2022-05-18T04:59:13.9645547Z ----------------------------------------------------------------------
2022-05-18T04:59:13.9645895Z Ran 1 test in 3.234s
2022-05-18T04:59:13.9646044Z 
2022-05-18T04:59:13.9646143Z OK
2022-05-18T04:59:13.9646285Z 
2022-05-18T04:59:13.9646428Z Generating XML reports...
2022-05-18T04:59:13.9687091Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045910.xml
2022-05-18T04:59:15.3836382Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:15.3851591Z 
2022-05-18T04:59:15.3852068Z Running tests...
2022-05-18T04:59:15.3852536Z ----------------------------------------------------------------------
2022-05-18T04:59:16.9531997Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:16.9924169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68330
2022-05-18T04:59:17.0030667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68331
2022-05-18T04:59:18.1653231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:18.1654096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:18.1654912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:18.1655609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:18.1661643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:18.1662723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:18.4079347Z ok (3.022s)
2022-05-18T04:59:18.4079750Z 
2022-05-18T04:59:18.4080519Z ----------------------------------------------------------------------
2022-05-18T04:59:18.4081003Z Ran 1 test in 3.023s
2022-05-18T04:59:18.4081174Z 
2022-05-18T04:59:18.4081270Z OK
2022-05-18T04:59:18.4081406Z 
2022-05-18T04:59:18.4081545Z Generating XML reports...
2022-05-18T04:59:18.4121320Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045915.xml
2022-05-18T04:59:19.8154531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:19.8169101Z 
2022-05-18T04:59:19.8169588Z Running tests...
2022-05-18T04:59:19.8170079Z ----------------------------------------------------------------------
2022-05-18T04:59:21.3870922Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:21.4263495Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68443
2022-05-18T04:59:21.4371375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68444
2022-05-18T04:59:22.5869465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:22.5870046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:22.5870828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:22.5871534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:22.5876795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:22.5877301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:22.8424797Z ok (3.025s)
2022-05-18T04:59:22.8425263Z 
2022-05-18T04:59:22.8425984Z ----------------------------------------------------------------------
2022-05-18T04:59:22.8426351Z Ran 1 test in 3.026s
2022-05-18T04:59:22.8426519Z 
2022-05-18T04:59:22.8426622Z OK
2022-05-18T04:59:22.8426761Z 
2022-05-18T04:59:22.8426900Z Generating XML reports...
2022-05-18T04:59:22.8468153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045919.xml
2022-05-18T04:59:24.2537900Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:24.2552897Z 
2022-05-18T04:59:24.2553361Z Running tests...
2022-05-18T04:59:24.2553854Z ----------------------------------------------------------------------
2022-05-18T04:59:25.8399075Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:25.8801458Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68556
2022-05-18T04:59:25.8910440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68557
2022-05-18T04:59:27.0483661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:27.0484226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:27.0485014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:27.0485713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:27.0592330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:27.1493984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:27.1608939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:59:27.1609472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:59:27.1610185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:59:27.1610869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:59:27.3962936Z ok (3.141s)
2022-05-18T04:59:27.3963336Z 
2022-05-18T04:59:27.3964113Z ----------------------------------------------------------------------
2022-05-18T04:59:27.3964588Z Ran 1 test in 3.141s
2022-05-18T04:59:27.3964758Z 
2022-05-18T04:59:27.3964859Z OK
2022-05-18T04:59:27.3964998Z 
2022-05-18T04:59:27.3965136Z Generating XML reports...
2022-05-18T04:59:27.4006325Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045924.xml
2022-05-18T04:59:28.8044776Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:28.8058954Z 
2022-05-18T04:59:28.8059340Z Running tests...
2022-05-18T04:59:28.8060069Z ----------------------------------------------------------------------
2022-05-18T04:59:30.3548664Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:30.3946779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68675
2022-05-18T04:59:30.4053688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68676
2022-05-18T04:59:31.5819473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:31.5820053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:31.5820876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:31.5821580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:31.5928742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:31.6832240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:31.8103107Z skip: Skipped due to small world size. (3.004s)
2022-05-18T04:59:31.8103600Z 
2022-05-18T04:59:31.8104235Z ----------------------------------------------------------------------
2022-05-18T04:59:31.8104879Z Ran 1 test in 3.004s
2022-05-18T04:59:31.8105157Z 
2022-05-18T04:59:31.8105365Z OK (skipped=1)
2022-05-18T04:59:31.8105647Z 
2022-05-18T04:59:31.8105874Z Generating XML reports...
2022-05-18T04:59:31.8147808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045928.xml
2022-05-18T04:59:33.2131715Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:33.2146480Z 
2022-05-18T04:59:33.2146995Z Running tests...
2022-05-18T04:59:33.2147536Z ----------------------------------------------------------------------
2022-05-18T04:59:34.7789377Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:34.8193427Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68784
2022-05-18T04:59:34.8304928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68785
2022-05-18T04:59:36.0055448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:36.0056026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:36.0056864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:36.0057561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:36.0163273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:36.1065578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:36.3356820Z ok (3.121s)
2022-05-18T04:59:36.3357049Z 
2022-05-18T04:59:36.3357427Z ----------------------------------------------------------------------
2022-05-18T04:59:36.3357776Z Ran 1 test in 3.121s
2022-05-18T04:59:36.3357945Z 
2022-05-18T04:59:36.3358043Z OK
2022-05-18T04:59:36.3358532Z 
2022-05-18T04:59:36.3358680Z Generating XML reports...
2022-05-18T04:59:36.3399739Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045933.xml
2022-05-18T04:59:37.7426652Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:37.7441612Z 
2022-05-18T04:59:37.7441766Z Running tests...
2022-05-18T04:59:37.7442475Z ----------------------------------------------------------------------
2022-05-18T04:59:39.3168961Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:39.3559139Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68897
2022-05-18T04:59:39.3666302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68898
2022-05-18T04:59:40.5716089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:40.5716668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:40.5717465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:40.5718433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:40.5826316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:40.6726150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:40.8717356Z ok (3.127s)
2022-05-18T04:59:40.8717717Z 
2022-05-18T04:59:40.8718397Z ----------------------------------------------------------------------
2022-05-18T04:59:40.8718757Z Ran 1 test in 3.128s
2022-05-18T04:59:40.8718906Z 
2022-05-18T04:59:40.8719003Z OK
2022-05-18T04:59:40.8719140Z 
2022-05-18T04:59:40.8719276Z Generating XML reports...
2022-05-18T04:59:40.8759703Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045937.xml
2022-05-18T04:59:42.2801776Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:42.2816284Z 
2022-05-18T04:59:42.2816437Z Running tests...
2022-05-18T04:59:42.2817165Z ----------------------------------------------------------------------
2022-05-18T04:59:43.8504365Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:43.8904662Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69010
2022-05-18T04:59:43.9013180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69011
2022-05-18T04:59:45.0805232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:45.0805786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:45.0806599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:45.0807306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:45.0914656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:45.1816566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:45.4067287Z ok (3.125s)
2022-05-18T04:59:45.4067504Z 
2022-05-18T04:59:45.4067904Z ----------------------------------------------------------------------
2022-05-18T04:59:45.4068261Z Ran 1 test in 3.125s
2022-05-18T04:59:45.4068433Z 
2022-05-18T04:59:45.4068531Z OK
2022-05-18T04:59:45.4068650Z 
2022-05-18T04:59:45.4068790Z Generating XML reports...
2022-05-18T04:59:45.4111454Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045942.xml
2022-05-18T04:59:46.8447892Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:46.8461864Z 
2022-05-18T04:59:46.8462295Z Running tests...
2022-05-18T04:59:46.8462813Z ----------------------------------------------------------------------
2022-05-18T04:59:46.8483239Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-05-18T04:59:46.8483877Z 
2022-05-18T04:59:46.8484224Z ----------------------------------------------------------------------
2022-05-18T04:59:46.8484571Z Ran 1 test in 0.002s
2022-05-18T04:59:46.8484740Z 
2022-05-18T04:59:46.8484852Z OK (skipped=1)
2022-05-18T04:59:46.8484990Z 
2022-05-18T04:59:46.8485118Z Generating XML reports...
2022-05-18T04:59:46.8519594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045946.xml
2022-05-18T04:59:48.1039557Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:48.1054017Z 
2022-05-18T04:59:48.1054401Z Running tests...
2022-05-18T04:59:48.1054909Z ----------------------------------------------------------------------
2022-05-18T04:59:48.1074419Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-05-18T04:59:48.1075140Z 
2022-05-18T04:59:48.1075465Z ----------------------------------------------------------------------
2022-05-18T04:59:48.1075818Z Ran 1 test in 0.002s
2022-05-18T04:59:48.1075967Z 
2022-05-18T04:59:48.1076085Z OK (skipped=1)
2022-05-18T04:59:48.1076246Z 
2022-05-18T04:59:48.1076375Z Generating XML reports...
2022-05-18T04:59:48.1112150Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045948.xml
2022-05-18T04:59:49.3618180Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:49.3632191Z 
2022-05-18T04:59:49.3632546Z Running tests...
2022-05-18T04:59:49.3633049Z ----------------------------------------------------------------------
2022-05-18T04:59:50.9460462Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:50.9863831Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69189
2022-05-18T04:59:50.9974470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69190
2022-05-18T04:59:52.2299993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:52.2300539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:52.2301330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:52.2302034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:52.2410058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:52.3311440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:52.3520807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:59:52.3521345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:59:52.3522281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:59:52.3522972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:59:52.6029082Z ok (3.239s)
2022-05-18T04:59:52.6029307Z 
2022-05-18T04:59:52.6029697Z ----------------------------------------------------------------------
2022-05-18T04:59:52.6030039Z Ran 1 test in 3.240s
2022-05-18T04:59:52.6030186Z 
2022-05-18T04:59:52.6030292Z OK
2022-05-18T04:59:52.6030451Z 
2022-05-18T04:59:52.6030589Z Generating XML reports...
2022-05-18T04:59:52.6071115Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045949.xml
2022-05-18T04:59:53.9893725Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:53.9908764Z 
2022-05-18T04:59:53.9908943Z Running tests...
2022-05-18T04:59:53.9909411Z ----------------------------------------------------------------------
2022-05-18T04:59:55.5861811Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:59:55.6253601Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69308
2022-05-18T04:59:55.6361441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69309
2022-05-18T04:59:56.8042869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:59:56.8043974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:59:56.8045281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:56.8046020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:59:56.8150186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:59:56.9057051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:59:57.0412993Z skip: Skipped due to small world size. (3.050s)
2022-05-18T04:59:57.0413268Z 
2022-05-18T04:59:57.0413667Z ----------------------------------------------------------------------
2022-05-18T04:59:57.0414004Z Ran 1 test in 3.050s
2022-05-18T04:59:57.0414168Z 
2022-05-18T04:59:57.0414286Z OK (skipped=1)
2022-05-18T04:59:57.0414428Z 
2022-05-18T04:59:57.0414848Z Generating XML reports...
2022-05-18T04:59:57.0457327Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045953.xml
2022-05-18T04:59:58.4561971Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:58.4576407Z 
2022-05-18T04:59:58.4576689Z Running tests...
2022-05-18T04:59:58.4577134Z ----------------------------------------------------------------------
2022-05-18T04:59:58.4597547Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-05-18T04:59:58.4597894Z 
2022-05-18T04:59:58.4598459Z ----------------------------------------------------------------------
2022-05-18T04:59:58.4598803Z Ran 1 test in 0.002s
2022-05-18T04:59:58.4598973Z 
2022-05-18T04:59:58.4599084Z OK (skipped=1)
2022-05-18T04:59:58.4599241Z 
2022-05-18T04:59:58.4599351Z Generating XML reports...
2022-05-18T04:59:58.4635184Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045958.xml
2022-05-18T04:59:59.7185174Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:59:59.7199235Z 
2022-05-18T04:59:59.7199779Z Running tests...
2022-05-18T04:59:59.7200288Z ----------------------------------------------------------------------
2022-05-18T04:59:59.7221032Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-05-18T04:59:59.7221383Z 
2022-05-18T04:59:59.7221673Z ----------------------------------------------------------------------
2022-05-18T04:59:59.7222005Z Ran 1 test in 0.002s
2022-05-18T04:59:59.7222167Z 
2022-05-18T04:59:59.7222262Z OK (skipped=1)
2022-05-18T04:59:59.7222419Z 
2022-05-18T04:59:59.7222544Z Generating XML reports...
2022-05-18T04:59:59.7258081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045959.xml
2022-05-18T05:00:00.9830053Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:00.9844771Z 
2022-05-18T05:00:00.9845299Z Running tests...
2022-05-18T05:00:00.9845794Z ----------------------------------------------------------------------
2022-05-18T05:00:02.5797218Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:02.6195154Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69483
2022-05-18T05:00:02.6305248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69484
2022-05-18T05:00:03.8391225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:03.8391781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:03.8392586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:03.8393309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:03.8498896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:03.9402040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:04.1359458Z ok (3.151s)
2022-05-18T05:00:04.1359685Z 
2022-05-18T05:00:04.1360084Z ----------------------------------------------------------------------
2022-05-18T05:00:04.1360424Z Ran 1 test in 3.151s
2022-05-18T05:00:04.1360574Z 
2022-05-18T05:00:04.1360673Z OK
2022-05-18T05:00:04.1360808Z 
2022-05-18T05:00:04.1360939Z Generating XML reports...
2022-05-18T05:00:04.1402189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050000.xml
2022-05-18T05:00:05.5407685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:05.5422180Z 
2022-05-18T05:00:05.5422619Z Running tests...
2022-05-18T05:00:05.5423100Z ----------------------------------------------------------------------
2022-05-18T05:00:07.1197085Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:07.1597426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69592
2022-05-18T05:00:07.1706985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69593
2022-05-18T05:00:08.3688891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:08.3689461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:08.3690244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:08.3690976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:08.3696502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:08.3697017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:08.3909685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:00:08.4012609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:00:08.4013309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:00:08.4014008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:00:08.4155913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:00:08.4156433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:00:08.4157330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:00:08.4158410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:00:08.4279388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T05:00:08.4279892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T05:00:08.4280595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T05:00:08.4281294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T05:00:08.6758447Z ok (3.133s)
2022-05-18T05:00:08.6758641Z 
2022-05-18T05:00:08.6759335Z ----------------------------------------------------------------------
2022-05-18T05:00:08.6759703Z Ran 1 test in 3.134s
2022-05-18T05:00:08.6759852Z 
2022-05-18T05:00:08.6759950Z OK
2022-05-18T05:00:08.6760099Z 
2022-05-18T05:00:08.6760232Z Generating XML reports...
2022-05-18T05:00:08.6802462Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050005.xml
2022-05-18T05:00:10.0776667Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:10.0790955Z 
2022-05-18T05:00:10.0791443Z Running tests...
2022-05-18T05:00:10.0791955Z ----------------------------------------------------------------------
2022-05-18T05:00:11.6735696Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:11.7136323Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69725
2022-05-18T05:00:11.7244397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69726
2022-05-18T05:00:12.9244816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:12.9245397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:12.9246205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:12.9246892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:12.9253264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:12.9253759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:12.9361864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:00:12.9362364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:00:12.9363073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:00:12.9363768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:00:13.1294842Z ok (3.050s)
2022-05-18T05:00:13.1295066Z 
2022-05-18T05:00:13.1295455Z ----------------------------------------------------------------------
2022-05-18T05:00:13.1295804Z Ran 1 test in 3.050s
2022-05-18T05:00:13.1295973Z 
2022-05-18T05:00:13.1296075Z OK
2022-05-18T05:00:13.1296211Z 
2022-05-18T05:00:13.1296351Z Generating XML reports...
2022-05-18T05:00:13.1336532Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050010.xml
2022-05-18T05:00:14.5353235Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:14.5367925Z 
2022-05-18T05:00:14.5368170Z Running tests...
2022-05-18T05:00:14.5368857Z ----------------------------------------------------------------------
2022-05-18T05:00:16.1008730Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:16.1411910Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69844
2022-05-18T05:00:16.1521741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69845
2022-05-18T05:00:17.3729566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:17.3730154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:17.3730959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:17.3731682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:17.3738832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:17.3739308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:17.3846921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:00:17.3847679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:00:17.3848384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:00:17.3849058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:00:17.5570301Z ok (3.020s)
2022-05-18T05:00:17.5570669Z 
2022-05-18T05:00:17.5571210Z ----------------------------------------------------------------------
2022-05-18T05:00:17.5571562Z Ran 1 test in 3.020s
2022-05-18T05:00:17.5571732Z 
2022-05-18T05:00:17.5571811Z OK
2022-05-18T05:00:17.5571966Z 
2022-05-18T05:00:17.5572185Z Generating XML reports...
2022-05-18T05:00:17.5612553Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050014.xml
2022-05-18T05:00:18.9810711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:18.9824658Z 
2022-05-18T05:00:18.9824926Z Running tests...
2022-05-18T05:00:18.9825371Z ----------------------------------------------------------------------
2022-05-18T05:00:20.5466217Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:20.5870139Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69963
2022-05-18T05:00:20.5978252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69964
2022-05-18T05:00:21.7510326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:21.7510920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:21.7511729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:21.7512435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:21.7518239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:21.7518735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:21.7726500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:00:21.7727018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:00:21.7727950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:00:21.7728682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:00:22.0029832Z ok (3.020s)
2022-05-18T05:00:22.0030052Z 
2022-05-18T05:00:22.0030453Z ----------------------------------------------------------------------
2022-05-18T05:00:22.0030778Z Ran 1 test in 3.020s
2022-05-18T05:00:22.0030949Z 
2022-05-18T05:00:22.0031053Z OK
2022-05-18T05:00:22.0031189Z 
2022-05-18T05:00:22.0031326Z Generating XML reports...
2022-05-18T05:00:22.0072641Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050018.xml
2022-05-18T05:00:23.4140496Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:23.4154697Z 
2022-05-18T05:00:23.4154937Z Running tests...
2022-05-18T05:00:23.4155392Z ----------------------------------------------------------------------
2022-05-18T05:00:24.9992867Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:25.0394221Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70082
2022-05-18T05:00:25.0503799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70083
2022-05-18T05:00:26.2454569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:26.2455124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:26.2455936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:26.2456910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:26.2462454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:26.2463036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:26.2670170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:00:26.2670696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:00:26.2671398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:00:26.2672080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:00:26.4555166Z ok (3.040s)
2022-05-18T05:00:26.4555402Z 
2022-05-18T05:00:26.4555816Z ----------------------------------------------------------------------
2022-05-18T05:00:26.4556149Z Ran 1 test in 3.040s
2022-05-18T05:00:26.4556319Z 
2022-05-18T05:00:26.4556419Z OK
2022-05-18T05:00:26.4556557Z 
2022-05-18T05:00:26.4556722Z Generating XML reports...
2022-05-18T05:00:26.4598327Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050023.xml
2022-05-18T05:00:27.8579478Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:27.8594088Z 
2022-05-18T05:00:27.8594477Z Running tests...
2022-05-18T05:00:27.8594921Z ----------------------------------------------------------------------
2022-05-18T05:00:29.4542089Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:29.4947711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70201
2022-05-18T05:00:29.5059791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70202
2022-05-18T05:00:30.7017232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:30.7018043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:30.7018911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:30.7019598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:30.7125964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:30.8031418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:31.0111076Z skip: Skipped due to small world size. (3.151s)
2022-05-18T05:00:31.0111337Z 
2022-05-18T05:00:31.0111722Z ----------------------------------------------------------------------
2022-05-18T05:00:31.0112067Z Ran 1 test in 3.152s
2022-05-18T05:00:31.0112235Z 
2022-05-18T05:00:31.0112345Z OK (skipped=1)
2022-05-18T05:00:31.0112501Z 
2022-05-18T05:00:31.0112647Z Generating XML reports...
2022-05-18T05:00:31.0153667Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050027.xml
2022-05-18T05:00:32.4171552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:32.4186178Z 
2022-05-18T05:00:32.4186700Z Running tests...
2022-05-18T05:00:32.4187212Z ----------------------------------------------------------------------
2022-05-18T05:00:33.9804881Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:34.0197391Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70310
2022-05-18T05:00:34.0305059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70311
2022-05-18T05:00:35.2106592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:35.2107146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:35.2107962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:35.2108664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:35.2213855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:35.3120062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:35.5357065Z skip: Skipped due to small world size. (3.117s)
2022-05-18T05:00:35.5357352Z 
2022-05-18T05:00:35.5357730Z ----------------------------------------------------------------------
2022-05-18T05:00:35.5358324Z Ran 1 test in 3.117s
2022-05-18T05:00:35.5358508Z 
2022-05-18T05:00:35.5358604Z OK (skipped=1)
2022-05-18T05:00:35.5358764Z 
2022-05-18T05:00:35.5358904Z Generating XML reports...
2022-05-18T05:00:35.5400289Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050032.xml
2022-05-18T05:00:36.9339357Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:36.9353919Z 
2022-05-18T05:00:36.9354274Z Running tests...
2022-05-18T05:00:36.9354704Z ----------------------------------------------------------------------
2022-05-18T05:00:38.5216982Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:38.5619596Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70419
2022-05-18T05:00:38.5728214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70420
2022-05-18T05:00:39.7599321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:39.7600119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:39.7600955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:39.7601634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:39.7607842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:39.7608858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:39.9779960Z skip: Skipped due to small world size. (3.042s)
2022-05-18T05:00:39.9780241Z 
2022-05-18T05:00:39.9780650Z ----------------------------------------------------------------------
2022-05-18T05:00:39.9780981Z Ran 1 test in 3.043s
2022-05-18T05:00:39.9781145Z 
2022-05-18T05:00:39.9781255Z OK (skipped=1)
2022-05-18T05:00:39.9781412Z 
2022-05-18T05:00:39.9781548Z Generating XML reports...
2022-05-18T05:00:39.9821374Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050036.xml
2022-05-18T05:00:41.3717104Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:41.3730601Z 
2022-05-18T05:00:41.3730935Z Running tests...
2022-05-18T05:00:41.3731453Z ----------------------------------------------------------------------
2022-05-18T05:00:42.9216527Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:42.9609693Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70528
2022-05-18T05:00:42.9717545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70529
2022-05-18T05:00:44.1667468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:44.1668074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:44.1668884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:44.1669585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:44.1675331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:44.1676134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:44.3768738Z skip: Skipped due to small world size. (3.003s)
2022-05-18T05:00:44.3769030Z 
2022-05-18T05:00:44.3769422Z ----------------------------------------------------------------------
2022-05-18T05:00:44.3769747Z Ran 1 test in 3.004s
2022-05-18T05:00:44.3769910Z 
2022-05-18T05:00:44.3770020Z OK (skipped=1)
2022-05-18T05:00:44.3770174Z 
2022-05-18T05:00:44.3770303Z Generating XML reports...
2022-05-18T05:00:44.3810363Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050041.xml
2022-05-18T05:00:45.7783193Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:45.7797363Z 
2022-05-18T05:00:45.7797721Z Running tests...
2022-05-18T05:00:45.7798663Z ----------------------------------------------------------------------
2022-05-18T05:00:47.3573621Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:47.3964947Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70637
2022-05-18T05:00:47.4072356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70638
2022-05-18T05:00:48.5671761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:48.5672548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:48.5673376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:48.5674079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:48.5680408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:48.5680926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:48.8122197Z ok (3.032s)
2022-05-18T05:00:48.8122416Z 
2022-05-18T05:00:48.8122807Z ----------------------------------------------------------------------
2022-05-18T05:00:48.8123165Z Ran 1 test in 3.032s
2022-05-18T05:00:48.8123333Z 
2022-05-18T05:00:48.8123428Z OK
2022-05-18T05:00:48.8123564Z 
2022-05-18T05:00:48.8123698Z Generating XML reports...
2022-05-18T05:00:48.8163749Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050045.xml
2022-05-18T05:00:50.2175988Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:50.2191942Z 
2022-05-18T05:00:50.2192389Z Running tests...
2022-05-18T05:00:50.2192913Z ----------------------------------------------------------------------
2022-05-18T05:00:51.7986311Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:51.8393880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70750
2022-05-18T05:00:51.8503840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70751
2022-05-18T05:00:52.9940812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:52.9941873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:52.9943027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:52.9944432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:52.9948875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:52.9950752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:53.1552548Z ok (2.936s)
2022-05-18T05:00:53.1552805Z 
2022-05-18T05:00:53.1553415Z ----------------------------------------------------------------------
2022-05-18T05:00:53.1553812Z Ran 1 test in 2.936s
2022-05-18T05:00:53.1553979Z 
2022-05-18T05:00:53.1554096Z OK
2022-05-18T05:00:53.1554234Z 
2022-05-18T05:00:53.1554356Z Generating XML reports...
2022-05-18T05:00:53.1597642Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050050.xml
2022-05-18T05:00:54.5614815Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:54.5629003Z 
2022-05-18T05:00:54.5629255Z Running tests...
2022-05-18T05:00:54.5629699Z ----------------------------------------------------------------------
2022-05-18T05:00:56.1433245Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:00:56.1839011Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70859
2022-05-18T05:00:56.1949911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70860
2022-05-18T05:00:57.3885352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:00:57.3885895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:00:57.3886915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:57.3887641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:00:57.3993207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:00:57.4895580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:00:57.8003705Z ok (3.237s)
2022-05-18T05:00:57.8003930Z 
2022-05-18T05:00:57.8004331Z ----------------------------------------------------------------------
2022-05-18T05:00:57.8004681Z Ran 1 test in 3.237s
2022-05-18T05:00:57.8004859Z 
2022-05-18T05:00:57.8004963Z OK
2022-05-18T05:00:57.8005104Z 
2022-05-18T05:00:57.8005243Z Generating XML reports...
2022-05-18T05:00:57.8045015Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050054.xml
2022-05-18T05:00:59.2235892Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:00:59.2250701Z 
2022-05-18T05:00:59.2251038Z Running tests...
2022-05-18T05:00:59.2251485Z ----------------------------------------------------------------------
2022-05-18T05:01:00.7927512Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:00.8325625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70972
2022-05-18T05:01:00.8432580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70973
2022-05-18T05:01:02.0217756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:02.0218564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:02.0219385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:02.0220083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:02.0325369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:02.1227680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:02.3485290Z ok (3.123s)
2022-05-18T05:01:02.3485511Z 
2022-05-18T05:01:02.3485892Z ----------------------------------------------------------------------
2022-05-18T05:01:02.3486252Z Ran 1 test in 3.123s
2022-05-18T05:01:02.3486418Z 
2022-05-18T05:01:02.3486536Z OK
2022-05-18T05:01:02.3486671Z 
2022-05-18T05:01:02.3486803Z Generating XML reports...
2022-05-18T05:01:02.3527241Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050059.xml
2022-05-18T05:01:03.7731861Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:03.7746632Z 
2022-05-18T05:01:03.7746906Z Running tests...
2022-05-18T05:01:03.7747346Z ----------------------------------------------------------------------
2022-05-18T05:01:05.3699014Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:05.4106239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71085
2022-05-18T05:01:05.4216718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71086
2022-05-18T05:01:06.5626550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:06.5627136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:06.5627946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:06.5628861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:06.5734574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:06.6637953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:06.8266988Z ok (3.052s)
2022-05-18T05:01:06.8267412Z 
2022-05-18T05:01:06.8268112Z ----------------------------------------------------------------------
2022-05-18T05:01:06.8268715Z Ran 1 test in 3.052s
2022-05-18T05:01:06.8269024Z 
2022-05-18T05:01:06.8269199Z OK
2022-05-18T05:01:06.8269420Z 
2022-05-18T05:01:06.8269703Z Generating XML reports...
2022-05-18T05:01:06.8311190Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050103.xml
2022-05-18T05:01:08.2469937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:08.2485335Z 
2022-05-18T05:01:08.2485615Z Running tests...
2022-05-18T05:01:08.2486058Z ----------------------------------------------------------------------
2022-05-18T05:01:09.8208270Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:09.8609068Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71198
2022-05-18T05:01:09.8718694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71199
2022-05-18T05:01:11.0676112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:11.0676674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:11.0677803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:11.0678848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:11.0685966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:11.0686460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:11.2768333Z ok (3.028s)
2022-05-18T05:01:11.2768549Z 
2022-05-18T05:01:11.2768944Z ----------------------------------------------------------------------
2022-05-18T05:01:11.2769299Z Ran 1 test in 3.028s
2022-05-18T05:01:11.2769452Z 
2022-05-18T05:01:11.2769548Z OK
2022-05-18T05:01:11.2769683Z 
2022-05-18T05:01:11.2769819Z Generating XML reports...
2022-05-18T05:01:11.2810317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050108.xml
2022-05-18T05:01:12.6887387Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:12.6902216Z 
2022-05-18T05:01:12.6902501Z Running tests...
2022-05-18T05:01:12.6903249Z ----------------------------------------------------------------------
2022-05-18T05:01:14.2765382Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:14.3172509Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71307
2022-05-18T05:01:14.3283575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71308
2022-05-18T05:01:15.4790130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:15.4790697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:15.4791522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:15.4792219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:15.4897879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:15.5801241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:15.6008968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:01:15.6009492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:01:15.6010227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:15.6010912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:15.8335453Z ok (3.143s)
2022-05-18T05:01:15.8335688Z 
2022-05-18T05:01:15.8336076Z ----------------------------------------------------------------------
2022-05-18T05:01:15.8336416Z Ran 1 test in 3.143s
2022-05-18T05:01:15.8336591Z 
2022-05-18T05:01:15.8336706Z OK
2022-05-18T05:01:15.8336847Z 
2022-05-18T05:01:15.8336965Z Generating XML reports...
2022-05-18T05:01:15.8378096Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050112.xml
2022-05-18T05:01:17.2590029Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:17.2606297Z 
2022-05-18T05:01:17.2606675Z Running tests...
2022-05-18T05:01:17.2607552Z ----------------------------------------------------------------------
2022-05-18T05:01:18.8456573Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:18.8861434Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71426
2022-05-18T05:01:18.8970582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71427
2022-05-18T05:01:20.1345877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:20.1346420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:20.1347209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:20.1347907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:20.1354956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:20.1355963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:20.1563625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:01:20.1564160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:01:20.1564905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:20.1565610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:20.4023527Z ok (3.141s)
2022-05-18T05:01:20.4023937Z 
2022-05-18T05:01:20.4024615Z ----------------------------------------------------------------------
2022-05-18T05:01:20.4025263Z Ran 1 test in 3.142s
2022-05-18T05:01:20.4025562Z 
2022-05-18T05:01:20.4025729Z OK
2022-05-18T05:01:20.4025960Z 
2022-05-18T05:01:20.4026202Z Generating XML reports...
2022-05-18T05:01:20.4068706Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050117.xml
2022-05-18T05:01:21.8199079Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:21.8213473Z 
2022-05-18T05:01:21.8213914Z Running tests...
2022-05-18T05:01:21.8214727Z ----------------------------------------------------------------------
2022-05-18T05:01:23.4030730Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:23.4431086Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71545
2022-05-18T05:01:23.4539272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71546
2022-05-18T05:01:24.6402085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:24.6402635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:24.6403460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:24.6404159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:24.6511976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:24.7413869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:24.7529495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:01:24.7530028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:01:24.7530741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:24.7531419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:24.9592000Z ok (3.137s)
2022-05-18T05:01:24.9592243Z 
2022-05-18T05:01:24.9592674Z ----------------------------------------------------------------------
2022-05-18T05:01:24.9593003Z Ran 1 test in 3.138s
2022-05-18T05:01:24.9593168Z 
2022-05-18T05:01:24.9593265Z OK
2022-05-18T05:01:24.9593417Z 
2022-05-18T05:01:24.9593555Z Generating XML reports...
2022-05-18T05:01:24.9633721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050121.xml
2022-05-18T05:01:26.3629493Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:26.3644319Z 
2022-05-18T05:01:26.3644787Z Running tests...
2022-05-18T05:01:26.3645305Z ----------------------------------------------------------------------
2022-05-18T05:01:27.9449478Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:27.9842904Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71664
2022-05-18T05:01:27.9948808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71665
2022-05-18T05:01:29.1664796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:29.1665331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:29.1666127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:29.1666840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:29.1773911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:29.2675615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:29.2789665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:01:29.2790186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:01:29.2791116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:29.2791849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:01:29.5002246Z ok (3.135s)
2022-05-18T05:01:29.5002469Z 
2022-05-18T05:01:29.5003010Z ----------------------------------------------------------------------
2022-05-18T05:01:29.5003706Z Ran 1 test in 3.136s
2022-05-18T05:01:29.5004067Z 
2022-05-18T05:01:29.5004250Z OK
2022-05-18T05:01:29.5004466Z 
2022-05-18T05:01:29.5004604Z Generating XML reports...
2022-05-18T05:01:29.5045331Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050126.xml
2022-05-18T05:01:30.9079271Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:30.9093558Z 
2022-05-18T05:01:30.9093989Z Running tests...
2022-05-18T05:01:30.9094982Z ----------------------------------------------------------------------
2022-05-18T05:01:32.4946911Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:32.5348371Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71783
2022-05-18T05:01:32.5456896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71784
2022-05-18T05:01:33.7201863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:33.7202763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:33.7203573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:33.7204530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:33.7312303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:33.8215785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:33.9506301Z skip: Skipped due to small world size. (3.041s)
2022-05-18T05:01:33.9506734Z 
2022-05-18T05:01:33.9507521Z ----------------------------------------------------------------------
2022-05-18T05:01:33.9508099Z Ran 1 test in 3.041s
2022-05-18T05:01:33.9508267Z 
2022-05-18T05:01:33.9508379Z OK (skipped=1)
2022-05-18T05:01:33.9508534Z 
2022-05-18T05:01:33.9508661Z Generating XML reports...
2022-05-18T05:01:33.9548754Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050130.xml
2022-05-18T05:01:35.3486571Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:35.3500638Z 
2022-05-18T05:01:35.3500793Z Running tests...
2022-05-18T05:01:35.3501599Z ----------------------------------------------------------------------
2022-05-18T05:01:36.9312704Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:36.9716018Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71892
2022-05-18T05:01:36.9824908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71893
2022-05-18T05:01:38.1713703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:38.1714246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:38.1715065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:38.1715785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:38.1721509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:38.1722205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:38.3874763Z skip: Skipped due to small world size. (3.037s)
2022-05-18T05:01:38.3875034Z 
2022-05-18T05:01:38.3875423Z ----------------------------------------------------------------------
2022-05-18T05:01:38.3875765Z Ran 1 test in 3.037s
2022-05-18T05:01:38.3875929Z 
2022-05-18T05:01:38.3876023Z OK (skipped=1)
2022-05-18T05:01:38.3876178Z 
2022-05-18T05:01:38.3876306Z Generating XML reports...
2022-05-18T05:01:38.3917044Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050135.xml
2022-05-18T05:01:39.7906352Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:39.7920264Z 
2022-05-18T05:01:39.7920534Z Running tests...
2022-05-18T05:01:39.7920976Z ----------------------------------------------------------------------
2022-05-18T05:01:41.3695775Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:41.4097536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72001
2022-05-18T05:01:41.4206485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72002
2022-05-18T05:01:42.6125708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:42.6126316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:42.6127122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:42.6128073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:42.6128599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:42.6129068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:42.8256680Z skip: Skipped due to small world size. (3.033s)
2022-05-18T05:01:42.8256929Z 
2022-05-18T05:01:42.8257331Z ----------------------------------------------------------------------
2022-05-18T05:01:42.8257657Z Ran 1 test in 3.034s
2022-05-18T05:01:42.8257820Z 
2022-05-18T05:01:42.8257934Z OK (skipped=1)
2022-05-18T05:01:42.8258098Z 
2022-05-18T05:01:42.8258233Z Generating XML reports...
2022-05-18T05:01:42.8300140Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050139.xml
2022-05-18T05:01:44.2269536Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:44.2285036Z 
2022-05-18T05:01:44.2285544Z Running tests...
2022-05-18T05:01:44.2286029Z ----------------------------------------------------------------------
2022-05-18T05:01:45.8025481Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:45.8419974Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72110
2022-05-18T05:01:45.8526996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72111
2022-05-18T05:01:47.0266948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:47.0267501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:47.0268295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:47.0269003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:47.0375476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:47.1281682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:47.2576957Z skip: Skipped due to small world size. (3.029s)
2022-05-18T05:01:47.2577227Z 
2022-05-18T05:01:47.2577615Z ----------------------------------------------------------------------
2022-05-18T05:01:47.2577954Z Ran 1 test in 3.029s
2022-05-18T05:01:47.2578127Z 
2022-05-18T05:01:47.2578248Z OK (skipped=1)
2022-05-18T05:01:47.2578387Z 
2022-05-18T05:01:47.2578517Z Generating XML reports...
2022-05-18T05:01:47.2619969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050144.xml
2022-05-18T05:01:48.6625577Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:48.6640456Z 
2022-05-18T05:01:48.6641002Z Running tests...
2022-05-18T05:01:48.6641623Z ----------------------------------------------------------------------
2022-05-18T05:01:50.2468063Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:50.2860989Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72219
2022-05-18T05:01:50.2968832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72220
2022-05-18T05:01:51.4855124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:51.4855698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:51.4856476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:51.4857417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:51.4862789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:51.4864463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:51.8020464Z ok (3.138s)
2022-05-18T05:01:51.8021058Z 
2022-05-18T05:01:51.8021806Z ----------------------------------------------------------------------
2022-05-18T05:01:51.8022479Z Ran 1 test in 3.138s
2022-05-18T05:01:51.8022790Z 
2022-05-18T05:01:51.8022985Z OK
2022-05-18T05:01:51.8023230Z 
2022-05-18T05:01:51.8023460Z Generating XML reports...
2022-05-18T05:01:51.8065132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050148.xml
2022-05-18T05:01:53.1694287Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:53.1709033Z 
2022-05-18T05:01:53.1709386Z Running tests...
2022-05-18T05:01:53.1709849Z ----------------------------------------------------------------------
2022-05-18T05:01:54.7642153Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:54.8044730Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72332
2022-05-18T05:01:54.8153555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72333
2022-05-18T05:01:56.0180995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:01:56.0181567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:01:56.0182357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:56.0183040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:01:56.0289411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:01:56.1192952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:01:56.3205684Z ok (3.149s)
2022-05-18T05:01:56.3206163Z 
2022-05-18T05:01:56.3206578Z ----------------------------------------------------------------------
2022-05-18T05:01:56.3206921Z Ran 1 test in 3.150s
2022-05-18T05:01:56.3207095Z 
2022-05-18T05:01:56.3207173Z OK
2022-05-18T05:01:56.3207308Z 
2022-05-18T05:01:56.3207442Z Generating XML reports...
2022-05-18T05:01:56.3248560Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050153.xml
2022-05-18T05:01:57.6994385Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:01:57.7008711Z 
2022-05-18T05:01:57.7009048Z Running tests...
2022-05-18T05:01:57.7009522Z ----------------------------------------------------------------------
2022-05-18T05:01:59.2338235Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:01:59.2730629Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72445
2022-05-18T05:01:59.2839835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72446
2022-05-18T05:02:00.4460132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:00.4460678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:00.4461479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:00.4462156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:00.4569774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:00.5475060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:02.3919681Z ok (4.691s)
2022-05-18T05:02:02.3919917Z 
2022-05-18T05:02:02.3920361Z ----------------------------------------------------------------------
2022-05-18T05:02:02.3920689Z Ran 1 test in 4.691s
2022-05-18T05:02:02.3920861Z 
2022-05-18T05:02:02.3920960Z OK
2022-05-18T05:02:02.3921096Z 
2022-05-18T05:02:02.3921231Z Generating XML reports...
2022-05-18T05:02:02.3962449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050157.xml
2022-05-18T05:02:03.8450673Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:03.8466010Z 
2022-05-18T05:02:03.8466305Z Running tests...
2022-05-18T05:02:03.8466738Z ----------------------------------------------------------------------
2022-05-18T05:02:05.4243366Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:05.4640355Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72560
2022-05-18T05:02:05.4750961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72561
2022-05-18T05:02:06.6463768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:06.6464328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:06.6465106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:06.6465802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:06.6472104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:06.6472616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:08.4829096Z ok (4.636s)
2022-05-18T05:02:08.4829329Z 
2022-05-18T05:02:08.4829723Z ----------------------------------------------------------------------
2022-05-18T05:02:08.4830309Z Ran 1 test in 4.636s
2022-05-18T05:02:08.4830495Z 
2022-05-18T05:02:08.4830594Z OK
2022-05-18T05:02:08.4830731Z 
2022-05-18T05:02:08.4830868Z Generating XML reports...
2022-05-18T05:02:08.4871172Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050203.xml
2022-05-18T05:02:09.9146207Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:09.9160483Z 
2022-05-18T05:02:09.9160933Z Running tests...
2022-05-18T05:02:09.9161443Z ----------------------------------------------------------------------
2022-05-18T05:02:11.5096134Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:11.5500146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72675
2022-05-18T05:02:11.5610242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72676
2022-05-18T05:02:12.7039760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:12.7040346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:12.7041149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:12.7041846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:12.7047948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:12.7048686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:12.8659607Z ok (2.949s)
2022-05-18T05:02:12.8659836Z 
2022-05-18T05:02:12.8660239Z ----------------------------------------------------------------------
2022-05-18T05:02:12.8660578Z Ran 1 test in 2.950s
2022-05-18T05:02:12.8660747Z 
2022-05-18T05:02:12.8660868Z OK
2022-05-18T05:02:12.8661006Z 
2022-05-18T05:02:12.8661141Z Generating XML reports...
2022-05-18T05:02:12.8702804Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050209.xml
2022-05-18T05:02:14.2725391Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:14.2739252Z 
2022-05-18T05:02:14.2739408Z Running tests...
2022-05-18T05:02:14.2739839Z ----------------------------------------------------------------------
2022-05-18T05:02:15.8592738Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:15.8996027Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72788
2022-05-18T05:02:15.9104409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72789
2022-05-18T05:02:17.1023421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:17.1024001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:17.1024822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:17.1025529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:17.1131259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:17.2036179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:18.7181240Z ok (4.444s)
2022-05-18T05:02:18.7181449Z 
2022-05-18T05:02:18.7181850Z ----------------------------------------------------------------------
2022-05-18T05:02:18.7182196Z Ran 1 test in 4.444s
2022-05-18T05:02:18.7182364Z 
2022-05-18T05:02:18.7182464Z OK
2022-05-18T05:02:18.7183730Z 
2022-05-18T05:02:18.7184580Z Generating XML reports...
2022-05-18T05:02:18.7224979Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050214.xml
2022-05-18T05:02:20.1266983Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:20.1280668Z 
2022-05-18T05:02:20.1281080Z Running tests...
2022-05-18T05:02:20.1281569Z ----------------------------------------------------------------------
2022-05-18T05:02:21.6700322Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:21.7094179Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72899
2022-05-18T05:02:21.7201457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72900
2022-05-18T05:02:22.9087993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:22.9088556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:22.9089378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:22.9090082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:22.9095759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:22.9096821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:23.1252214Z ok (2.997s)
2022-05-18T05:02:23.1252727Z 
2022-05-18T05:02:23.1253138Z ----------------------------------------------------------------------
2022-05-18T05:02:23.1253721Z Ran 1 test in 2.997s
2022-05-18T05:02:23.1253980Z 
2022-05-18T05:02:23.1254146Z OK
2022-05-18T05:02:23.1254375Z 
2022-05-18T05:02:23.1255582Z Generating XML reports...
2022-05-18T05:02:23.1294729Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050220.xml
2022-05-18T05:02:24.5400040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:24.5415494Z 
2022-05-18T05:02:24.5415815Z Running tests...
2022-05-18T05:02:24.5416251Z ----------------------------------------------------------------------
2022-05-18T05:02:26.1132065Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:26.1525519Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73012
2022-05-18T05:02:26.1632981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73013
2022-05-18T05:02:27.3368833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:27.3369768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:27.3371296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:27.3372665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:27.3377692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:27.3378604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:27.5684498Z ok (3.026s)
2022-05-18T05:02:27.5684715Z 
2022-05-18T05:02:27.5685126Z ----------------------------------------------------------------------
2022-05-18T05:02:27.5685478Z Ran 1 test in 3.027s
2022-05-18T05:02:27.5685647Z 
2022-05-18T05:02:27.5685748Z OK
2022-05-18T05:02:27.5685885Z 
2022-05-18T05:02:27.5686020Z Generating XML reports...
2022-05-18T05:02:27.5727508Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050224.xml
2022-05-18T05:02:28.9483774Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:28.9497286Z 
2022-05-18T05:02:28.9497434Z Running tests...
2022-05-18T05:02:28.9498273Z ----------------------------------------------------------------------
2022-05-18T05:02:30.4981879Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:30.5380759Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73125
2022-05-18T05:02:30.5487230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73126
2022-05-18T05:02:31.7000746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:31.7001289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:31.7002119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:31.7003729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:31.7110957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:31.8012492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:32.0539999Z ok (3.104s)
2022-05-18T05:02:32.0540230Z 
2022-05-18T05:02:32.0540617Z ----------------------------------------------------------------------
2022-05-18T05:02:32.0541241Z Ran 1 test in 3.104s
2022-05-18T05:02:32.0541390Z 
2022-05-18T05:02:32.0541489Z OK
2022-05-18T05:02:32.0541634Z 
2022-05-18T05:02:32.0541769Z Generating XML reports...
2022-05-18T05:02:32.0581802Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050228.xml
2022-05-18T05:02:33.4697003Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:33.4711342Z 
2022-05-18T05:02:33.4711749Z Running tests...
2022-05-18T05:02:33.4712256Z ----------------------------------------------------------------------
2022-05-18T05:02:35.0510326Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:35.0911770Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73238
2022-05-18T05:02:35.1020044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73239
2022-05-18T05:02:36.2953852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:36.2954439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:36.2962463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:36.2963175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:36.2963688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:36.2964176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:40.2136585Z ok (6.742s)
2022-05-18T05:02:40.2137265Z 
2022-05-18T05:02:40.2137692Z ----------------------------------------------------------------------
2022-05-18T05:02:40.2138062Z Ran 1 test in 6.743s
2022-05-18T05:02:40.2138232Z 
2022-05-18T05:02:40.2138353Z OK
2022-05-18T05:02:40.2138490Z 
2022-05-18T05:02:40.2138609Z Generating XML reports...
2022-05-18T05:02:40.2180292Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050233.xml
2022-05-18T05:02:41.6335602Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:41.6349885Z 
2022-05-18T05:02:41.6350254Z Running tests...
2022-05-18T05:02:41.6350706Z ----------------------------------------------------------------------
2022-05-18T05:02:43.1963527Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:43.2364273Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73355
2022-05-18T05:02:43.2472609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73356
2022-05-18T05:02:44.4619710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:44.4620270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:44.4621075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:44.4621786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:44.4627488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:44.4628624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:48.3591947Z ok (6.724s)
2022-05-18T05:02:48.3592134Z 
2022-05-18T05:02:48.3592545Z ----------------------------------------------------------------------
2022-05-18T05:02:48.3592892Z Ran 1 test in 6.724s
2022-05-18T05:02:48.3593060Z 
2022-05-18T05:02:48.3593155Z OK
2022-05-18T05:02:48.3593293Z 
2022-05-18T05:02:48.3593684Z Generating XML reports...
2022-05-18T05:02:48.3634757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050241.xml
2022-05-18T05:02:49.7706166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:49.7719885Z 
2022-05-18T05:02:49.7720451Z Running tests...
2022-05-18T05:02:49.7720956Z ----------------------------------------------------------------------
2022-05-18T05:02:51.3015127Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:02:51.3407624Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73472
2022-05-18T05:02:51.3515009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73473
2022-05-18T05:02:52.5168339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:02:52.5168986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:02:52.5169825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:52.5170534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:02:52.5276454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:02:52.6182811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:02:56.4633241Z ok (6.691s)
2022-05-18T05:02:56.4633578Z 
2022-05-18T05:02:56.4634120Z ----------------------------------------------------------------------
2022-05-18T05:02:56.4634475Z Ran 1 test in 6.691s
2022-05-18T05:02:56.4634623Z 
2022-05-18T05:02:56.4634779Z OK
2022-05-18T05:02:56.4635036Z 
2022-05-18T05:02:56.4635274Z Generating XML reports...
2022-05-18T05:02:56.4674766Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050249.xml
2022-05-18T05:02:57.8812871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:57.8826498Z 
2022-05-18T05:02:57.8826823Z Running tests...
2022-05-18T05:02:57.8827561Z ----------------------------------------------------------------------
2022-05-18T05:02:57.8846885Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T05:02:57.8847192Z 
2022-05-18T05:02:57.8847471Z ----------------------------------------------------------------------
2022-05-18T05:02:57.8847819Z Ran 1 test in 0.002s
2022-05-18T05:02:57.8847984Z 
2022-05-18T05:02:57.8848099Z OK (skipped=1)
2022-05-18T05:02:57.8848265Z 
2022-05-18T05:02:57.8848396Z Generating XML reports...
2022-05-18T05:02:57.8883460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050257.xml
2022-05-18T05:02:59.1122703Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:02:59.1137090Z 
2022-05-18T05:02:59.1137240Z Running tests...
2022-05-18T05:02:59.1138176Z ----------------------------------------------------------------------
2022-05-18T05:02:59.1156450Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T05:02:59.1157114Z 
2022-05-18T05:02:59.1157422Z ----------------------------------------------------------------------
2022-05-18T05:02:59.1157780Z Ran 1 test in 0.002s
2022-05-18T05:02:59.1157952Z 
2022-05-18T05:02:59.1158349Z OK (skipped=1)
2022-05-18T05:02:59.1158524Z 
2022-05-18T05:02:59.1158635Z Generating XML reports...
2022-05-18T05:02:59.1194317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050259.xml
2022-05-18T05:03:00.3726430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:00.3741366Z 
2022-05-18T05:03:00.3741779Z Running tests...
2022-05-18T05:03:00.3742306Z ----------------------------------------------------------------------
2022-05-18T05:03:00.3761861Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T05:03:00.3762365Z 
2022-05-18T05:03:00.3762667Z ----------------------------------------------------------------------
2022-05-18T05:03:00.3763002Z Ran 1 test in 0.002s
2022-05-18T05:03:00.3763172Z 
2022-05-18T05:03:00.3763287Z OK (skipped=1)
2022-05-18T05:03:00.3763446Z 
2022-05-18T05:03:00.3763575Z Generating XML reports...
2022-05-18T05:03:00.3799300Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050300.xml
2022-05-18T05:03:01.6405144Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:01.6419751Z 
2022-05-18T05:03:01.6419928Z Running tests...
2022-05-18T05:03:01.6420687Z ----------------------------------------------------------------------
2022-05-18T05:03:01.6441066Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T05:03:01.6441404Z 
2022-05-18T05:03:01.6441711Z ----------------------------------------------------------------------
2022-05-18T05:03:01.6442050Z Ran 1 test in 0.002s
2022-05-18T05:03:01.6442217Z 
2022-05-18T05:03:01.6442338Z OK (skipped=1)
2022-05-18T05:03:01.6442499Z 
2022-05-18T05:03:01.6442608Z Generating XML reports...
2022-05-18T05:03:01.6479064Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050301.xml
2022-05-18T05:03:02.9121532Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:02.9136535Z 
2022-05-18T05:03:02.9136846Z Running tests...
2022-05-18T05:03:02.9137280Z ----------------------------------------------------------------------
2022-05-18T05:03:02.9156587Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T05:03:02.9157122Z 
2022-05-18T05:03:02.9157424Z ----------------------------------------------------------------------
2022-05-18T05:03:02.9158036Z Ran 1 test in 0.002s
2022-05-18T05:03:02.9158584Z 
2022-05-18T05:03:02.9158681Z OK (skipped=1)
2022-05-18T05:03:02.9158838Z 
2022-05-18T05:03:02.9158966Z Generating XML reports...
2022-05-18T05:03:02.9194063Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050302.xml
2022-05-18T05:03:04.1797685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:04.1812379Z 
2022-05-18T05:03:04.1812696Z Running tests...
2022-05-18T05:03:04.1813151Z ----------------------------------------------------------------------
2022-05-18T05:03:04.1833129Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T05:03:04.1833675Z 
2022-05-18T05:03:04.1834115Z ----------------------------------------------------------------------
2022-05-18T05:03:04.1834438Z Ran 1 test in 0.002s
2022-05-18T05:03:04.1834611Z 
2022-05-18T05:03:04.1834739Z OK (skipped=1)
2022-05-18T05:03:04.1834902Z 
2022-05-18T05:03:04.1835031Z Generating XML reports...
2022-05-18T05:03:04.1870190Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050304.xml
2022-05-18T05:03:05.4443975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:05.4458238Z 
2022-05-18T05:03:05.4458582Z Running tests...
2022-05-18T05:03:05.4459025Z ----------------------------------------------------------------------
2022-05-18T05:03:05.4477591Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-05-18T05:03:05.4478489Z 
2022-05-18T05:03:05.4478796Z ----------------------------------------------------------------------
2022-05-18T05:03:05.4479132Z Ran 1 test in 0.002s
2022-05-18T05:03:05.4479300Z 
2022-05-18T05:03:05.4479412Z OK (skipped=1)
2022-05-18T05:03:05.4479571Z 
2022-05-18T05:03:05.4479680Z Generating XML reports...
2022-05-18T05:03:05.4514839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050305.xml
2022-05-18T05:03:06.6744054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:06.6758509Z 
2022-05-18T05:03:06.6758791Z Running tests...
2022-05-18T05:03:06.6759223Z ----------------------------------------------------------------------
2022-05-18T05:03:06.6779361Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:03:06.6779692Z 
2022-05-18T05:03:06.6779977Z ----------------------------------------------------------------------
2022-05-18T05:03:06.6780339Z Ran 1 test in 0.002s
2022-05-18T05:03:06.6780505Z 
2022-05-18T05:03:06.6780597Z OK (skipped=1)
2022-05-18T05:03:06.6780757Z 
2022-05-18T05:03:06.6780885Z Generating XML reports...
2022-05-18T05:03:06.6816722Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050306.xml
2022-05-18T05:03:07.9356574Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:07.9371676Z 
2022-05-18T05:03:07.9372207Z Running tests...
2022-05-18T05:03:07.9372718Z ----------------------------------------------------------------------
2022-05-18T05:03:07.9390433Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:03:07.9390772Z 
2022-05-18T05:03:07.9391061Z ----------------------------------------------------------------------
2022-05-18T05:03:07.9391405Z Ran 1 test in 0.002s
2022-05-18T05:03:07.9391571Z 
2022-05-18T05:03:07.9391684Z OK (skipped=1)
2022-05-18T05:03:07.9391844Z 
2022-05-18T05:03:07.9391971Z Generating XML reports...
2022-05-18T05:03:07.9427920Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050307.xml
2022-05-18T05:03:09.1956490Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:09.1971157Z 
2022-05-18T05:03:09.1971456Z Running tests...
2022-05-18T05:03:09.1971910Z ----------------------------------------------------------------------
2022-05-18T05:03:09.1990604Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:03:09.1991045Z 
2022-05-18T05:03:09.1991439Z ----------------------------------------------------------------------
2022-05-18T05:03:09.1991810Z Ran 1 test in 0.002s
2022-05-18T05:03:09.1991972Z 
2022-05-18T05:03:09.1992091Z OK (skipped=1)
2022-05-18T05:03:09.1992251Z 
2022-05-18T05:03:09.1992379Z Generating XML reports...
2022-05-18T05:03:09.2027923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050309.xml
2022-05-18T05:03:10.4316027Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:10.4331786Z 
2022-05-18T05:03:10.4332168Z Running tests...
2022-05-18T05:03:10.4332717Z ----------------------------------------------------------------------
2022-05-18T05:03:10.4353382Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:03:10.4353748Z 
2022-05-18T05:03:10.4354050Z ----------------------------------------------------------------------
2022-05-18T05:03:10.4354366Z Ran 1 test in 0.002s
2022-05-18T05:03:10.4354532Z 
2022-05-18T05:03:10.4354643Z OK (skipped=1)
2022-05-18T05:03:10.4355111Z 
2022-05-18T05:03:10.4355240Z Generating XML reports...
2022-05-18T05:03:10.4390869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050310.xml
2022-05-18T05:03:11.6975725Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:11.6990059Z 
2022-05-18T05:03:11.6990428Z Running tests...
2022-05-18T05:03:11.6990879Z ----------------------------------------------------------------------
2022-05-18T05:03:11.7011348Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:03:11.7011699Z 
2022-05-18T05:03:11.7011985Z ----------------------------------------------------------------------
2022-05-18T05:03:11.7012303Z Ran 1 test in 0.002s
2022-05-18T05:03:11.7012468Z 
2022-05-18T05:03:11.7012582Z OK (skipped=1)
2022-05-18T05:03:11.7012741Z 
2022-05-18T05:03:11.7012868Z Generating XML reports...
2022-05-18T05:03:11.7048702Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050311.xml
2022-05-18T05:03:12.9157540Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:12.9171929Z 
2022-05-18T05:03:12.9172733Z Running tests...
2022-05-18T05:03:12.9173227Z ----------------------------------------------------------------------
2022-05-18T05:03:12.9191020Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:03:12.9191580Z 
2022-05-18T05:03:12.9191954Z ----------------------------------------------------------------------
2022-05-18T05:03:12.9192300Z Ran 1 test in 0.002s
2022-05-18T05:03:12.9192449Z 
2022-05-18T05:03:12.9192568Z OK (skipped=1)
2022-05-18T05:03:12.9192736Z 
2022-05-18T05:03:12.9192864Z Generating XML reports...
2022-05-18T05:03:12.9228032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050312.xml
2022-05-18T05:03:14.1799107Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:14.1813861Z 
2022-05-18T05:03:14.1814320Z Running tests...
2022-05-18T05:03:14.1815094Z ----------------------------------------------------------------------
2022-05-18T05:03:14.1834692Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:03:14.1835055Z 
2022-05-18T05:03:14.1835351Z ----------------------------------------------------------------------
2022-05-18T05:03:14.1835672Z Ran 1 test in 0.002s
2022-05-18T05:03:14.1835839Z 
2022-05-18T05:03:14.1835952Z OK (skipped=1)
2022-05-18T05:03:14.1837354Z 
2022-05-18T05:03:14.1839211Z Generating XML reports...
2022-05-18T05:03:14.1871772Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050314.xml
2022-05-18T05:03:15.4051234Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:15.4065126Z 
2022-05-18T05:03:15.4065411Z Running tests...
2022-05-18T05:03:15.4066117Z ----------------------------------------------------------------------
2022-05-18T05:03:15.4085754Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:03:15.4086392Z 
2022-05-18T05:03:15.4087021Z ----------------------------------------------------------------------
2022-05-18T05:03:15.4087691Z Ran 1 test in 0.002s
2022-05-18T05:03:15.4087862Z 
2022-05-18T05:03:15.4087974Z OK (skipped=1)
2022-05-18T05:03:15.4088134Z 
2022-05-18T05:03:15.4088243Z Generating XML reports...
2022-05-18T05:03:15.4123325Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050315.xml
2022-05-18T05:03:16.6256927Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:16.6271134Z 
2022-05-18T05:03:16.6271567Z Running tests...
2022-05-18T05:03:16.6272117Z ----------------------------------------------------------------------
2022-05-18T05:03:16.6293532Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:03:16.6294088Z 
2022-05-18T05:03:16.6294623Z ----------------------------------------------------------------------
2022-05-18T05:03:16.6294986Z Ran 1 test in 0.002s
2022-05-18T05:03:16.6295132Z 
2022-05-18T05:03:16.6295252Z OK (skipped=1)
2022-05-18T05:03:16.6295410Z 
2022-05-18T05:03:16.6295537Z Generating XML reports...
2022-05-18T05:03:16.6331483Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050316.xml
2022-05-18T05:03:17.8698976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:17.8713200Z 
2022-05-18T05:03:17.8713425Z Running tests...
2022-05-18T05:03:17.8713854Z ----------------------------------------------------------------------
2022-05-18T05:03:17.8736524Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:03:17.8737037Z 
2022-05-18T05:03:17.8737428Z ----------------------------------------------------------------------
2022-05-18T05:03:17.8737748Z Ran 1 test in 0.002s
2022-05-18T05:03:17.8737920Z 
2022-05-18T05:03:17.8738031Z OK (skipped=1)
2022-05-18T05:03:17.8738188Z 
2022-05-18T05:03:17.8738315Z Generating XML reports...
2022-05-18T05:03:17.8772886Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050317.xml
2022-05-18T05:03:19.0979919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:19.0994331Z 
2022-05-18T05:03:19.0994660Z Running tests...
2022-05-18T05:03:19.0995097Z ----------------------------------------------------------------------
2022-05-18T05:03:19.1014465Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:03:19.1014918Z 
2022-05-18T05:03:19.1015522Z ----------------------------------------------------------------------
2022-05-18T05:03:19.1015886Z Ran 1 test in 0.002s
2022-05-18T05:03:19.1016051Z 
2022-05-18T05:03:19.1016163Z OK (skipped=1)
2022-05-18T05:03:19.1016322Z 
2022-05-18T05:03:19.1016431Z Generating XML reports...
2022-05-18T05:03:19.1050616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050319.xml
2022-05-18T05:03:20.3526267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:20.3540893Z 
2022-05-18T05:03:20.3541055Z Running tests...
2022-05-18T05:03:20.3541519Z ----------------------------------------------------------------------
2022-05-18T05:03:20.3562868Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:03:20.3563210Z 
2022-05-18T05:03:20.3563511Z ----------------------------------------------------------------------
2022-05-18T05:03:20.3563853Z Ran 1 test in 0.002s
2022-05-18T05:03:20.3564021Z 
2022-05-18T05:03:20.3564120Z OK (skipped=1)
2022-05-18T05:03:20.3564281Z 
2022-05-18T05:03:20.3564409Z Generating XML reports...
2022-05-18T05:03:20.3601020Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050320.xml
2022-05-18T05:03:21.5959345Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:21.5974200Z 
2022-05-18T05:03:21.5974525Z Running tests...
2022-05-18T05:03:21.5974962Z ----------------------------------------------------------------------
2022-05-18T05:03:21.5995710Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:03:21.5996063Z 
2022-05-18T05:03:21.5996354Z ----------------------------------------------------------------------
2022-05-18T05:03:21.5996683Z Ran 1 test in 0.002s
2022-05-18T05:03:21.5996854Z 
2022-05-18T05:03:21.5996973Z OK (skipped=1)
2022-05-18T05:03:21.5997133Z 
2022-05-18T05:03:21.5997261Z Generating XML reports...
2022-05-18T05:03:21.6033933Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050321.xml
2022-05-18T05:03:22.8667510Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:22.8682805Z 
2022-05-18T05:03:22.8683395Z Running tests...
2022-05-18T05:03:22.8683879Z ----------------------------------------------------------------------
2022-05-18T05:03:22.8702208Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:03:22.8702724Z 
2022-05-18T05:03:22.8703133Z ----------------------------------------------------------------------
2022-05-18T05:03:22.8703468Z Ran 1 test in 0.002s
2022-05-18T05:03:22.8703625Z 
2022-05-18T05:03:22.8703739Z OK (skipped=1)
2022-05-18T05:03:22.8703900Z 
2022-05-18T05:03:22.8704028Z Generating XML reports...
2022-05-18T05:03:22.8739349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050322.xml
2022-05-18T05:03:24.1298402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:24.1312782Z 
2022-05-18T05:03:24.1313119Z Running tests...
2022-05-18T05:03:24.1313846Z ----------------------------------------------------------------------
2022-05-18T05:03:24.1334420Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:03:24.1335114Z 
2022-05-18T05:03:24.1335463Z ----------------------------------------------------------------------
2022-05-18T05:03:24.1335811Z Ran 1 test in 0.002s
2022-05-18T05:03:24.1335978Z 
2022-05-18T05:03:24.1336094Z OK (skipped=1)
2022-05-18T05:03:24.1336529Z 
2022-05-18T05:03:24.1336655Z Generating XML reports...
2022-05-18T05:03:24.1371104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050324.xml
2022-05-18T05:03:25.3868833Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:25.3883153Z 
2022-05-18T05:03:25.3883455Z Running tests...
2022-05-18T05:03:25.3883891Z ----------------------------------------------------------------------
2022-05-18T05:03:25.3903170Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T05:03:25.3903829Z 
2022-05-18T05:03:25.3904184Z ----------------------------------------------------------------------
2022-05-18T05:03:25.3904525Z Ran 1 test in 0.002s
2022-05-18T05:03:25.3904672Z 
2022-05-18T05:03:25.3904790Z OK (skipped=1)
2022-05-18T05:03:25.3904948Z 
2022-05-18T05:03:25.3905084Z Generating XML reports...
2022-05-18T05:03:25.3940076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050325.xml
2022-05-18T05:03:26.6450990Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:26.6464704Z 
2022-05-18T05:03:26.6464835Z Running tests...
2022-05-18T05:03:26.6465754Z ----------------------------------------------------------------------
2022-05-18T05:03:26.6485860Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T05:03:26.6486597Z 
2022-05-18T05:03:26.6486899Z ----------------------------------------------------------------------
2022-05-18T05:03:26.6487257Z Ran 1 test in 0.002s
2022-05-18T05:03:26.6487432Z 
2022-05-18T05:03:26.6487546Z OK (skipped=1)
2022-05-18T05:03:26.6487714Z 
2022-05-18T05:03:26.6487846Z Generating XML reports...
2022-05-18T05:03:26.6522937Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050326.xml
2022-05-18T05:03:27.9025870Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:27.9039987Z 
2022-05-18T05:03:27.9040436Z Running tests...
2022-05-18T05:03:27.9040949Z ----------------------------------------------------------------------
2022-05-18T05:03:29.4986731Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:29.5387232Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74381
2022-05-18T05:03:29.5495661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74382
2022-05-18T05:03:30.7247022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:30.7247759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:30.7248583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:30.7249265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:30.7255266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:30.7255769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:32.9759451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:03:32.9760169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:03:32.9760959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:03:32.9761885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:03:33.3599693Z ok (5.456s)
2022-05-18T05:03:33.3599898Z 
2022-05-18T05:03:33.3600703Z ----------------------------------------------------------------------
2022-05-18T05:03:33.3601096Z Ran 1 test in 5.456s
2022-05-18T05:03:33.3601247Z 
2022-05-18T05:03:33.3601346Z OK
2022-05-18T05:03:33.3601492Z 
2022-05-18T05:03:33.3601635Z Generating XML reports...
2022-05-18T05:03:33.3641858Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050327.xml
2022-05-18T05:03:34.7563239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:34.7576658Z 
2022-05-18T05:03:34.7576884Z Running tests...
2022-05-18T05:03:34.7577750Z ----------------------------------------------------------------------
2022-05-18T05:03:36.2999819Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:36.3393926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74504
2022-05-18T05:03:36.3500170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74505
2022-05-18T05:03:37.5112711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:37.5113288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:37.5114077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:37.5115032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:37.5222160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:37.6126212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:37.7548981Z skip: Need at least 3 CUDA devices (2.997s)
2022-05-18T05:03:37.7549432Z 
2022-05-18T05:03:37.7549921Z ----------------------------------------------------------------------
2022-05-18T05:03:37.7550310Z Ran 1 test in 2.997s
2022-05-18T05:03:37.7550485Z 
2022-05-18T05:03:37.7550602Z OK (skipped=1)
2022-05-18T05:03:37.7550763Z 
2022-05-18T05:03:37.7550895Z Generating XML reports...
2022-05-18T05:03:37.7592464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050334.xml
2022-05-18T05:03:39.1688903Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:39.1703390Z 
2022-05-18T05:03:39.1703541Z Running tests...
2022-05-18T05:03:39.1704283Z ----------------------------------------------------------------------
2022-05-18T05:03:39.1724315Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-05-18T05:03:39.1724637Z 
2022-05-18T05:03:39.1724929Z ----------------------------------------------------------------------
2022-05-18T05:03:39.1725261Z Ran 1 test in 0.002s
2022-05-18T05:03:39.1725429Z 
2022-05-18T05:03:39.1725543Z OK (skipped=1)
2022-05-18T05:03:39.1725702Z 
2022-05-18T05:03:39.1725830Z Generating XML reports...
2022-05-18T05:03:39.1760944Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050339.xml
2022-05-18T05:03:40.3963612Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:40.3977937Z 
2022-05-18T05:03:40.3978182Z Running tests...
2022-05-18T05:03:40.3978745Z ----------------------------------------------------------------------
2022-05-18T05:03:41.9820272Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:42.0222158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74646
2022-05-18T05:03:42.0332326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74647
2022-05-18T05:03:43.2190906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:43.2191475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:43.2192278Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:43.2192965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:43.2199184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:43.2199681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:44.2396838Z ok (3.841s)
2022-05-18T05:03:44.2397062Z 
2022-05-18T05:03:44.2397470Z ----------------------------------------------------------------------
2022-05-18T05:03:44.2397802Z Ran 1 test in 3.842s
2022-05-18T05:03:44.2397972Z 
2022-05-18T05:03:44.2398296Z OK
2022-05-18T05:03:44.2398443Z 
2022-05-18T05:03:44.2398581Z Generating XML reports...
2022-05-18T05:03:44.2438290Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050340.xml
2022-05-18T05:03:45.6424784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:45.6438860Z 
2022-05-18T05:03:45.6439121Z Running tests...
2022-05-18T05:03:45.6439562Z ----------------------------------------------------------------------
2022-05-18T05:03:47.1844542Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:47.2234167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74755
2022-05-18T05:03:47.2344994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74756
2022-05-18T05:03:48.4289607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:48.4290174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:48.4290982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:48.4291672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:48.4297901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:48.4298755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:50.8432389Z ok (5.199s)
2022-05-18T05:03:50.8432720Z 
2022-05-18T05:03:50.8433316Z ----------------------------------------------------------------------
2022-05-18T05:03:50.8433695Z Ran 1 test in 5.199s
2022-05-18T05:03:50.8433865Z 
2022-05-18T05:03:50.8433964Z OK
2022-05-18T05:03:50.8434104Z 
2022-05-18T05:03:50.8434222Z Generating XML reports...
2022-05-18T05:03:50.8473851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050345.xml
2022-05-18T05:03:52.2404919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:52.2418851Z 
2022-05-18T05:03:52.2419124Z Running tests...
2022-05-18T05:03:52.2419584Z ----------------------------------------------------------------------
2022-05-18T05:03:53.7810777Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:53.8204345Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74866
2022-05-18T05:03:53.8314577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74867
2022-05-18T05:03:55.0454134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:03:55.0454737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:03:55.0455560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:55.0456263Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:03:55.0561998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:03:55.1464828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:03:55.1578713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:03:55.1579652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:03:55.1580673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:03:55.1581381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:03:56.1380234Z ok (3.896s)
2022-05-18T05:03:56.1380485Z 
2022-05-18T05:03:56.1380884Z ----------------------------------------------------------------------
2022-05-18T05:03:56.1381212Z Ran 1 test in 3.896s
2022-05-18T05:03:56.1381381Z 
2022-05-18T05:03:56.1381479Z OK
2022-05-18T05:03:56.1381615Z 
2022-05-18T05:03:56.1381756Z Generating XML reports...
2022-05-18T05:03:56.1422762Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050352.xml
2022-05-18T05:03:57.5616066Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:03:57.5631002Z 
2022-05-18T05:03:57.5631266Z Running tests...
2022-05-18T05:03:57.5632004Z ----------------------------------------------------------------------
2022-05-18T05:03:59.1430182Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:03:59.1832353Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74981
2022-05-18T05:03:59.1942632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74982
2022-05-18T05:04:00.3998746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:00.3999619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:00.4000457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:00.4001140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:00.4106879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:00.5012975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:00.6994225Z skip: Skipped due to small world size. (3.136s)
2022-05-18T05:04:00.6994495Z 
2022-05-18T05:04:00.6994908Z ----------------------------------------------------------------------
2022-05-18T05:04:00.6995253Z Ran 1 test in 3.136s
2022-05-18T05:04:00.6995402Z 
2022-05-18T05:04:00.6995516Z OK (skipped=1)
2022-05-18T05:04:00.6995674Z 
2022-05-18T05:04:00.6995804Z Generating XML reports...
2022-05-18T05:04:00.7037425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050357.xml
2022-05-18T05:04:02.1104025Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:02.1118753Z 
2022-05-18T05:04:02.1119092Z Running tests...
2022-05-18T05:04:02.1119843Z ----------------------------------------------------------------------
2022-05-18T05:04:03.6854290Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:03.7246812Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75090
2022-05-18T05:04:03.7354514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75091
2022-05-18T05:04:04.9182545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:04.9183185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:04.9184002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:04.9184702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:04.9289770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:05.0195776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:05.2407296Z skip: Skipped due to small world size. (3.128s)
2022-05-18T05:04:05.2407551Z 
2022-05-18T05:04:05.2407938Z ----------------------------------------------------------------------
2022-05-18T05:04:05.2408261Z Ran 1 test in 3.129s
2022-05-18T05:04:05.2408427Z 
2022-05-18T05:04:05.2408546Z OK (skipped=1)
2022-05-18T05:04:05.2408703Z 
2022-05-18T05:04:05.2408830Z Generating XML reports...
2022-05-18T05:04:05.2449121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050402.xml
2022-05-18T05:04:06.6254637Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:06.6268995Z 
2022-05-18T05:04:06.6269156Z Running tests...
2022-05-18T05:04:06.6269964Z ----------------------------------------------------------------------
2022-05-18T05:04:08.1546998Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:08.1938357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75199
2022-05-18T05:04:08.2046791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75200
2022-05-18T05:04:09.4008354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:09.4008918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:09.4009953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:09.4010692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:09.4115945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:09.5021938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:09.7098130Z skip: Skipped due to small world size. (3.082s)
2022-05-18T05:04:09.7098391Z 
2022-05-18T05:04:09.7098783Z ----------------------------------------------------------------------
2022-05-18T05:04:09.7099127Z Ran 1 test in 3.083s
2022-05-18T05:04:09.7099291Z 
2022-05-18T05:04:09.7099405Z OK (skipped=1)
2022-05-18T05:04:09.7099561Z 
2022-05-18T05:04:09.7099691Z Generating XML reports...
2022-05-18T05:04:09.7139836Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050406.xml
2022-05-18T05:04:11.1131546Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:11.1146591Z 
2022-05-18T05:04:11.1146799Z Running tests...
2022-05-18T05:04:11.1148173Z ----------------------------------------------------------------------
2022-05-18T05:04:12.6954938Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:12.7348772Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75308
2022-05-18T05:04:12.7458116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75309
2022-05-18T05:04:13.9038817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:13.9039424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:13.9040226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:13.9040936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:13.9146411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:14.0050283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:14.0160974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:04:14.0161778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:04:14.0162459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:04:14.0163150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:04:15.2530275Z ok (4.138s)
2022-05-18T05:04:15.2530516Z 
2022-05-18T05:04:15.2530906Z ----------------------------------------------------------------------
2022-05-18T05:04:15.2531254Z Ran 1 test in 4.138s
2022-05-18T05:04:15.2531403Z 
2022-05-18T05:04:15.2531508Z OK
2022-05-18T05:04:15.2531643Z 
2022-05-18T05:04:15.2531796Z Generating XML reports...
2022-05-18T05:04:15.2580507Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050411.xml
2022-05-18T05:04:16.6773876Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:16.6788123Z 
2022-05-18T05:04:16.6788547Z Running tests...
2022-05-18T05:04:16.6789043Z ----------------------------------------------------------------------
2022-05-18T05:04:18.2613224Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:18.3017098Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75423
2022-05-18T05:04:18.3127237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75424
2022-05-18T05:04:19.4606197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:19.4606769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:19.4607639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:19.4608323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:19.4614285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:19.4614772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:19.5038721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:19.5039238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:19.5040136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:19.5040916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:20.7197161Z ok (4.040s)
2022-05-18T05:04:20.7197396Z 
2022-05-18T05:04:20.7197767Z ----------------------------------------------------------------------
2022-05-18T05:04:20.7198379Z Ran 1 test in 4.041s
2022-05-18T05:04:20.7198556Z 
2022-05-18T05:04:20.7198657Z OK
2022-05-18T05:04:20.7198798Z 
2022-05-18T05:04:20.7198939Z Generating XML reports...
2022-05-18T05:04:20.7247896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050416.xml
2022-05-18T05:04:22.1504252Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:22.1518213Z 
2022-05-18T05:04:22.1519011Z Running tests...
2022-05-18T05:04:22.1519521Z ----------------------------------------------------------------------
2022-05-18T05:04:23.7386180Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:23.7786904Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75538
2022-05-18T05:04:23.7895801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75539
2022-05-18T05:04:24.9712489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:24.9713033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:24.9713838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:24.9714799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:24.9721088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:24.9721746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:25.1947603Z skip: Skipped due to small world size. (3.043s)
2022-05-18T05:04:25.1947852Z 
2022-05-18T05:04:25.1948252Z ----------------------------------------------------------------------
2022-05-18T05:04:25.1948611Z Ran 1 test in 3.043s
2022-05-18T05:04:25.1948784Z 
2022-05-18T05:04:25.1948878Z OK (skipped=1)
2022-05-18T05:04:25.1949038Z 
2022-05-18T05:04:25.1949170Z Generating XML reports...
2022-05-18T05:04:25.1997240Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050422.xml
2022-05-18T05:04:26.5975605Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:26.5990861Z 
2022-05-18T05:04:26.5991001Z Running tests...
2022-05-18T05:04:26.5992143Z ----------------------------------------------------------------------
2022-05-18T05:04:28.1756405Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:28.2149148Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75647
2022-05-18T05:04:28.2256899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75648
2022-05-18T05:04:29.3856744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:29.3857285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:29.3858080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:29.3858794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:29.3964628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:29.4869545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:29.7309432Z ok (3.131s)
2022-05-18T05:04:29.7309644Z 
2022-05-18T05:04:29.7310035Z ----------------------------------------------------------------------
2022-05-18T05:04:29.7310377Z Ran 1 test in 3.132s
2022-05-18T05:04:29.7310548Z 
2022-05-18T05:04:29.7310642Z OK
2022-05-18T05:04:29.7312069Z 
2022-05-18T05:04:29.7312551Z Generating XML reports...
2022-05-18T05:04:29.7352415Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050426.xml
2022-05-18T05:04:31.1519909Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:31.1535069Z 
2022-05-18T05:04:31.1535458Z Running tests...
2022-05-18T05:04:31.1535958Z ----------------------------------------------------------------------
2022-05-18T05:04:32.7249575Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:32.7656860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75756
2022-05-18T05:04:32.7767343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75757
2022-05-18T05:04:33.9410761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:33.9411331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:33.9412152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:33.9412860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:33.9418791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:33.9419294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:34.1819526Z ok (3.028s)
2022-05-18T05:04:34.1819738Z 
2022-05-18T05:04:34.1820147Z ----------------------------------------------------------------------
2022-05-18T05:04:34.1820471Z Ran 1 test in 3.028s
2022-05-18T05:04:34.1820636Z 
2022-05-18T05:04:34.1820740Z OK
2022-05-18T05:04:34.1820878Z 
2022-05-18T05:04:34.1821023Z Generating XML reports...
2022-05-18T05:04:34.1861972Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050431.xml
2022-05-18T05:04:35.5844989Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:35.5859098Z 
2022-05-18T05:04:35.5859551Z Running tests...
2022-05-18T05:04:35.5860025Z ----------------------------------------------------------------------
2022-05-18T05:04:35.5886054Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T05:04:35.5886722Z 
2022-05-18T05:04:35.5887377Z ----------------------------------------------------------------------
2022-05-18T05:04:35.5887884Z Ran 1 test in 0.003s
2022-05-18T05:04:35.5888051Z 
2022-05-18T05:04:35.5888144Z OK (skipped=1)
2022-05-18T05:04:35.5888300Z 
2022-05-18T05:04:35.5888427Z Generating XML reports...
2022-05-18T05:04:35.5923099Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050435.xml
2022-05-18T05:04:36.8356551Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:36.8371105Z 
2022-05-18T05:04:36.8371522Z Running tests...
2022-05-18T05:04:36.8372510Z ----------------------------------------------------------------------
2022-05-18T05:04:36.8399995Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T05:04:36.8400615Z 
2022-05-18T05:04:36.8401237Z ----------------------------------------------------------------------
2022-05-18T05:04:36.8402135Z Ran 1 test in 0.003s
2022-05-18T05:04:36.8402329Z 
2022-05-18T05:04:36.8402442Z OK (skipped=1)
2022-05-18T05:04:36.8402598Z 
2022-05-18T05:04:36.8402708Z Generating XML reports...
2022-05-18T05:04:36.8437001Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050436.xml
2022-05-18T05:04:38.0928550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:38.0942388Z 
2022-05-18T05:04:38.0942710Z Running tests...
2022-05-18T05:04:38.0943398Z ----------------------------------------------------------------------
2022-05-18T05:04:38.0971991Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T05:04:38.0972324Z 
2022-05-18T05:04:38.0972625Z ----------------------------------------------------------------------
2022-05-18T05:04:38.0973241Z Ran 1 test in 0.003s
2022-05-18T05:04:38.0973397Z 
2022-05-18T05:04:38.0973519Z OK (skipped=1)
2022-05-18T05:04:38.0973676Z 
2022-05-18T05:04:38.0973803Z Generating XML reports...
2022-05-18T05:04:38.1008967Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050438.xml
2022-05-18T05:04:39.3641624Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:39.3656282Z 
2022-05-18T05:04:39.3656803Z Running tests...
2022-05-18T05:04:39.3657280Z ----------------------------------------------------------------------
2022-05-18T05:04:39.3679832Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T05:04:39.3680403Z 
2022-05-18T05:04:39.3680698Z ----------------------------------------------------------------------
2022-05-18T05:04:39.3681010Z Ran 1 test in 0.002s
2022-05-18T05:04:39.3681178Z 
2022-05-18T05:04:39.3681289Z OK (skipped=1)
2022-05-18T05:04:39.3681444Z 
2022-05-18T05:04:39.3681583Z Generating XML reports...
2022-05-18T05:04:39.3716873Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050439.xml
2022-05-18T05:04:40.6310001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:40.6326549Z 
2022-05-18T05:04:40.6326950Z Running tests...
2022-05-18T05:04:40.6327463Z ----------------------------------------------------------------------
2022-05-18T05:04:40.6348539Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T05:04:40.6348861Z 
2022-05-18T05:04:40.6349264Z ----------------------------------------------------------------------
2022-05-18T05:04:40.6349762Z Ran 1 test in 0.002s
2022-05-18T05:04:40.6349929Z 
2022-05-18T05:04:40.6350041Z OK (skipped=1)
2022-05-18T05:04:40.6350198Z 
2022-05-18T05:04:40.6350326Z Generating XML reports...
2022-05-18T05:04:40.6385297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050440.xml
2022-05-18T05:04:41.8974414Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:41.8988717Z 
2022-05-18T05:04:41.8989157Z Running tests...
2022-05-18T05:04:41.8989662Z ----------------------------------------------------------------------
2022-05-18T05:04:41.9015742Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T05:04:41.9016087Z 
2022-05-18T05:04:41.9016380Z ----------------------------------------------------------------------
2022-05-18T05:04:41.9016735Z Ran 1 test in 0.003s
2022-05-18T05:04:41.9016901Z 
2022-05-18T05:04:41.9017010Z OK (skipped=1)
2022-05-18T05:04:41.9017150Z 
2022-05-18T05:04:41.9017274Z Generating XML reports...
2022-05-18T05:04:41.9054007Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050441.xml
2022-05-18T05:04:43.1178877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:43.1193288Z 
2022-05-18T05:04:43.1193765Z Running tests...
2022-05-18T05:04:43.1194263Z ----------------------------------------------------------------------
2022-05-18T05:04:43.1220779Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-05-18T05:04:43.1221120Z 
2022-05-18T05:04:43.1221416Z ----------------------------------------------------------------------
2022-05-18T05:04:43.1221754Z Ran 1 test in 0.003s
2022-05-18T05:04:43.1221902Z 
2022-05-18T05:04:43.1222034Z OK (skipped=1)
2022-05-18T05:04:43.1222191Z 
2022-05-18T05:04:43.1222319Z Generating XML reports...
2022-05-18T05:04:43.1257584Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050443.xml
2022-05-18T05:04:44.3766258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:44.3780264Z 
2022-05-18T05:04:44.3780522Z Running tests...
2022-05-18T05:04:44.3780954Z ----------------------------------------------------------------------
2022-05-18T05:04:44.3803980Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T05:04:44.3804679Z 
2022-05-18T05:04:44.3805305Z ----------------------------------------------------------------------
2022-05-18T05:04:44.3805691Z Ran 1 test in 0.002s
2022-05-18T05:04:44.3805858Z 
2022-05-18T05:04:44.3805954Z OK (skipped=1)
2022-05-18T05:04:44.3806110Z 
2022-05-18T05:04:44.3806501Z Generating XML reports...
2022-05-18T05:04:44.3840819Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050444.xml
2022-05-18T05:04:45.6332911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:45.6346171Z 
2022-05-18T05:04:45.6346607Z Running tests...
2022-05-18T05:04:45.6347563Z ----------------------------------------------------------------------
2022-05-18T05:04:47.2270934Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:47.2671992Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76129
2022-05-18T05:04:47.2781532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76130
2022-05-18T05:04:48.4447323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:48.4447970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:48.4448809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:48.4449501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:48.4554890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:48.5457973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:48.7833771Z ok (3.148s)
2022-05-18T05:04:48.7833965Z 
2022-05-18T05:04:48.7834495Z ----------------------------------------------------------------------
2022-05-18T05:04:48.7834994Z Ran 1 test in 3.149s
2022-05-18T05:04:48.7835165Z 
2022-05-18T05:04:48.7835262Z OK
2022-05-18T05:04:48.7835408Z 
2022-05-18T05:04:48.7835523Z Generating XML reports...
2022-05-18T05:04:48.7876954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050445.xml
2022-05-18T05:04:50.1928391Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:50.1942582Z 
2022-05-18T05:04:50.1942857Z Running tests...
2022-05-18T05:04:50.1943560Z ----------------------------------------------------------------------
2022-05-18T05:04:51.7839836Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:51.8243648Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76242
2022-05-18T05:04:51.8353735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76243
2022-05-18T05:04:53.0392696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:53.0393259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:53.0394112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:53.0394827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:53.0501156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:53.1406839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:55.0436779Z ok (4.849s)
2022-05-18T05:04:55.0437004Z 
2022-05-18T05:04:55.0437399Z ----------------------------------------------------------------------
2022-05-18T05:04:55.0437764Z Ran 1 test in 4.849s
2022-05-18T05:04:55.0437936Z 
2022-05-18T05:04:55.0438015Z OK
2022-05-18T05:04:55.0438533Z 
2022-05-18T05:04:55.0438675Z Generating XML reports...
2022-05-18T05:04:55.0481542Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050450.xml
2022-05-18T05:04:56.4634028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:04:56.4648534Z 
2022-05-18T05:04:56.4649019Z Running tests...
2022-05-18T05:04:56.4649519Z ----------------------------------------------------------------------
2022-05-18T05:04:58.0480357Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:04:58.0880143Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76357
2022-05-18T05:04:58.0988402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76358
2022-05-18T05:04:59.2501057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:04:59.2501877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:04:59.2502668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:59.2503381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:04:59.2510193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:04:59.2510909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:04:59.2617027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:04:59.2617785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:04:59.2618749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:04:59.2619451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:04:59.5039422Z ok (3.039s)
2022-05-18T05:04:59.5039790Z 
2022-05-18T05:04:59.5040223Z ----------------------------------------------------------------------
2022-05-18T05:04:59.5040557Z Ran 1 test in 3.039s
2022-05-18T05:04:59.5040724Z 
2022-05-18T05:04:59.5040823Z OK
2022-05-18T05:04:59.5040962Z 
2022-05-18T05:04:59.5041386Z Generating XML reports...
2022-05-18T05:04:59.5082651Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050456.xml
2022-05-18T05:05:00.9077447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:00.9092095Z 
2022-05-18T05:05:00.9092358Z Running tests...
2022-05-18T05:05:00.9092822Z ----------------------------------------------------------------------
2022-05-18T05:05:02.5037964Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:02.5440175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76476
2022-05-18T05:05:02.5549009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76477
2022-05-18T05:05:03.7405644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:03.7406228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:03.7407032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:03.7407739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:03.7413099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:03.7413828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:03.9597847Z skip: Skipped due to small world size. (3.050s)
2022-05-18T05:05:03.9598866Z 
2022-05-18T05:05:03.9599271Z ----------------------------------------------------------------------
2022-05-18T05:05:03.9599624Z Ran 1 test in 3.051s
2022-05-18T05:05:03.9599790Z 
2022-05-18T05:05:03.9599901Z OK (skipped=1)
2022-05-18T05:05:03.9600038Z 
2022-05-18T05:05:03.9600165Z Generating XML reports...
2022-05-18T05:05:03.9640656Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050500.xml
2022-05-18T05:05:05.3646547Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:05.3661258Z 
2022-05-18T05:05:05.3661514Z Running tests...
2022-05-18T05:05:05.3661970Z ----------------------------------------------------------------------
2022-05-18T05:05:06.9703958Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:07.0106592Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76585
2022-05-18T05:05:07.0215330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76586
2022-05-18T05:05:08.2389529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:08.2390096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:08.2390901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:08.2391602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:08.2397196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:08.2397675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:09.8291963Z ok (4.463s)
2022-05-18T05:05:09.8292188Z 
2022-05-18T05:05:09.8292584Z ----------------------------------------------------------------------
2022-05-18T05:05:09.8292957Z Ran 1 test in 4.463s
2022-05-18T05:05:09.8293132Z 
2022-05-18T05:05:09.8293210Z OK
2022-05-18T05:05:09.8293346Z 
2022-05-18T05:05:09.8293480Z Generating XML reports...
2022-05-18T05:05:09.8334521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050505.xml
2022-05-18T05:05:11.2257502Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:11.2271527Z 
2022-05-18T05:05:11.2272025Z Running tests...
2022-05-18T05:05:11.2272531Z ----------------------------------------------------------------------
2022-05-18T05:05:12.7755142Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:12.8164779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76696
2022-05-18T05:05:12.8274922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76697
2022-05-18T05:05:14.0239495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:14.0240096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:14.0240910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:14.0241613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:14.0347344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:14.1249626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:14.3326658Z ok (3.105s)
2022-05-18T05:05:14.3326859Z 
2022-05-18T05:05:14.3327841Z ----------------------------------------------------------------------
2022-05-18T05:05:14.3328445Z Ran 1 test in 3.105s
2022-05-18T05:05:14.3328627Z 
2022-05-18T05:05:14.3328722Z OK
2022-05-18T05:05:14.3328859Z 
2022-05-18T05:05:14.3328994Z Generating XML reports...
2022-05-18T05:05:14.3369227Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050511.xml
2022-05-18T05:05:15.7418623Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:15.7433011Z 
2022-05-18T05:05:15.7433318Z Running tests...
2022-05-18T05:05:15.7433767Z ----------------------------------------------------------------------
2022-05-18T05:05:17.3322327Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:17.3724642Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76805
2022-05-18T05:05:17.3835036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76806
2022-05-18T05:05:18.5582550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:18.5583121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:18.5583963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:18.5584652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:18.5690067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:18.6593345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:18.6800725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:05:18.6801225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:05:18.6801953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:05:18.6802648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:05:18.6909035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:05:18.6909578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:05:18.6910244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:05:18.6910935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:05:19.9681288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ze8hlf6
2022-05-18T05:05:19.9682250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ze8hlf6/_remote_module_non_scriptable.py
2022-05-18T05:05:19.9916570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdy0e8edw
2022-05-18T05:05:19.9919042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdy0e8edw/_remote_module_non_scriptable.py
2022-05-18T05:05:20.2913012Z ok (4.548s)
2022-05-18T05:05:20.2913197Z 
2022-05-18T05:05:20.2913583Z ----------------------------------------------------------------------
2022-05-18T05:05:20.2913933Z Ran 1 test in 4.548s
2022-05-18T05:05:20.2914081Z 
2022-05-18T05:05:20.2914180Z OK
2022-05-18T05:05:20.2914318Z 
2022-05-18T05:05:20.2914454Z Generating XML reports...
2022-05-18T05:05:20.2956213Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050515.xml
2022-05-18T05:05:21.7013752Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:21.7027669Z 
2022-05-18T05:05:21.7027936Z Running tests...
2022-05-18T05:05:21.7028388Z ----------------------------------------------------------------------
2022-05-18T05:05:23.2473073Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:23.2874483Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76928
2022-05-18T05:05:23.2982027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76929
2022-05-18T05:05:24.4852388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:24.4852962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:24.4853769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:24.4854494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:24.4960067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:24.5864684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:24.5975590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:05:24.5976117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:05:24.5976805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:05:24.5977506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:05:24.6286108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:05:24.6286635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:05:24.6287295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:05:24.6288206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:05:25.9104421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkd2yrdho
2022-05-18T05:05:25.9105069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkd2yrdho/_remote_module_non_scriptable.py
2022-05-18T05:05:25.9500436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx2pf3iq7
2022-05-18T05:05:25.9502439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx2pf3iq7/_remote_module_non_scriptable.py
2022-05-18T05:05:26.3063111Z ok (4.603s)
2022-05-18T05:05:26.3063975Z 
2022-05-18T05:05:26.3064702Z ----------------------------------------------------------------------
2022-05-18T05:05:26.3065351Z Ran 1 test in 4.603s
2022-05-18T05:05:26.3065662Z 
2022-05-18T05:05:26.3065830Z OK
2022-05-18T05:05:26.3066081Z 
2022-05-18T05:05:26.3066314Z Generating XML reports...
2022-05-18T05:05:26.3107836Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050521.xml
2022-05-18T05:05:27.7091575Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:27.7105041Z 
2022-05-18T05:05:27.7105281Z Running tests...
2022-05-18T05:05:27.7105859Z ----------------------------------------------------------------------
2022-05-18T05:05:29.2626255Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:29.3024598Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77051
2022-05-18T05:05:29.3131882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77052
2022-05-18T05:05:30.5059835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:30.5060389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:30.5061210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:30.5061896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:30.5068014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:30.5068508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:31.7878156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiomh5zqk
2022-05-18T05:05:31.7879003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiomh5zqk/_remote_module_non_scriptable.py
2022-05-18T05:05:31.8387735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptu4rsnyk
2022-05-18T05:05:31.8388756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptu4rsnyk/_remote_module_non_scriptable.py
2022-05-18T05:05:32.0279273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:32.0279828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:32.3209943Z ok (4.610s)
2022-05-18T05:05:32.3210126Z 
2022-05-18T05:05:32.3210843Z ----------------------------------------------------------------------
2022-05-18T05:05:32.3211228Z Ran 1 test in 4.610s
2022-05-18T05:05:32.3211401Z 
2022-05-18T05:05:32.3211499Z OK
2022-05-18T05:05:32.3211620Z 
2022-05-18T05:05:32.3211759Z Generating XML reports...
2022-05-18T05:05:32.3252265Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050527.xml
2022-05-18T05:05:33.7441764Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:33.7456191Z 
2022-05-18T05:05:33.7456651Z Running tests...
2022-05-18T05:05:33.7457436Z ----------------------------------------------------------------------
2022-05-18T05:05:35.3343982Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:35.3747830Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77166
2022-05-18T05:05:35.3857361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77167
2022-05-18T05:05:36.5644744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:36.5645314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:36.5646146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:36.5646850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:36.5652852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:36.5653350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:37.8488007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphaj8prsn
2022-05-18T05:05:37.8488658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphaj8prsn/_remote_module_non_scriptable.py
2022-05-18T05:05:37.8615887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1g4d6i1w
2022-05-18T05:05:37.8618813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1g4d6i1w/_remote_module_non_scriptable.py
2022-05-18T05:05:38.0551710Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:38.0552248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:38.0575333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:38.0575848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:38.3934364Z ok (4.647s)
2022-05-18T05:05:38.3934579Z 
2022-05-18T05:05:38.3934971Z ----------------------------------------------------------------------
2022-05-18T05:05:38.3935327Z Ran 1 test in 4.648s
2022-05-18T05:05:38.3935496Z 
2022-05-18T05:05:38.3935594Z OK
2022-05-18T05:05:38.3935733Z 
2022-05-18T05:05:38.3935848Z Generating XML reports...
2022-05-18T05:05:38.3976492Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050533.xml
2022-05-18T05:05:39.8192824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:39.8207552Z 
2022-05-18T05:05:39.8207942Z Running tests...
2022-05-18T05:05:39.8208450Z ----------------------------------------------------------------------
2022-05-18T05:05:41.3906804Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:41.4309111Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77281
2022-05-18T05:05:41.4418655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77282
2022-05-18T05:05:42.6246938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:42.6247500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:42.6248292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:42.6249015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:42.6354950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:42.7261061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:43.8867521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7u7zdrr
2022-05-18T05:05:43.8868357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7u7zdrr/_remote_module_non_scriptable.py
2022-05-18T05:05:43.9748992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptl4cbx5u
2022-05-18T05:05:43.9750248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptl4cbx5u/_remote_module_non_scriptable.py
2022-05-18T05:05:44.1730256Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:44.1730844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:44.1747311Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:44.1747824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:44.1908381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:44.1908899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:44.1922439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:44.1922941Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:05:44.4499059Z ok (4.629s)
2022-05-18T05:05:44.4499288Z 
2022-05-18T05:05:44.4499829Z ----------------------------------------------------------------------
2022-05-18T05:05:44.4500558Z Ran 1 test in 4.629s
2022-05-18T05:05:44.4500723Z 
2022-05-18T05:05:44.4500821Z OK
2022-05-18T05:05:44.4501720Z 
2022-05-18T05:05:44.4502391Z Generating XML reports...
2022-05-18T05:05:44.4541354Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050539.xml
2022-05-18T05:05:45.8419697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:45.8434709Z 
2022-05-18T05:05:45.8434948Z Running tests...
2022-05-18T05:05:45.8435544Z ----------------------------------------------------------------------
2022-05-18T05:05:47.4329316Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:47.4485967Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.605s)
2022-05-18T05:05:47.4486629Z 
2022-05-18T05:05:47.4486925Z ----------------------------------------------------------------------
2022-05-18T05:05:47.4487261Z Ran 1 test in 1.605s
2022-05-18T05:05:47.4487408Z 
2022-05-18T05:05:47.4487529Z OK (skipped=1)
2022-05-18T05:05:47.4487688Z 
2022-05-18T05:05:47.4487817Z Generating XML reports...
2022-05-18T05:05:47.4520758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050545.xml
2022-05-18T05:05:48.7944142Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:48.7958739Z 
2022-05-18T05:05:48.7958974Z Running tests...
2022-05-18T05:05:48.7959600Z ----------------------------------------------------------------------
2022-05-18T05:05:50.3659542Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:50.4058358Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77430
2022-05-18T05:05:50.4166370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77431
2022-05-18T05:05:51.5778855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:51.5779441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:51.5780249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:51.5780933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:51.5886327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:51.6792411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:52.8358194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvnbo1ir
2022-05-18T05:05:52.8359080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvnbo1ir/_remote_module_non_scriptable.py
2022-05-18T05:05:52.9450289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5nd6ekie
2022-05-18T05:05:52.9451621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5nd6ekie/_remote_module_non_scriptable.py
2022-05-18T05:05:52.9660306Z <class 'torch.nn.parameter.Parameter'>
2022-05-18T05:05:53.2243532Z ok (4.428s)
2022-05-18T05:05:53.2243748Z 
2022-05-18T05:05:53.2244139Z ----------------------------------------------------------------------
2022-05-18T05:05:53.2244479Z Ran 1 test in 4.428s
2022-05-18T05:05:53.2244649Z 
2022-05-18T05:05:53.2244727Z OK
2022-05-18T05:05:53.2244863Z 
2022-05-18T05:05:53.2244999Z Generating XML reports...
2022-05-18T05:05:53.2285312Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050548.xml
2022-05-18T05:05:54.6308640Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:05:54.6322850Z 
2022-05-18T05:05:54.6323177Z Running tests...
2022-05-18T05:05:54.6323635Z ----------------------------------------------------------------------
2022-05-18T05:05:56.1840302Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:05:56.2233696Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77541
2022-05-18T05:05:56.2340682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77542
2022-05-18T05:05:57.4402208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:05:57.4402767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:05:57.4403604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:57.4404285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:05:57.4411272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:05:57.4412067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:05:58.7360522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjrdme7t
2022-05-18T05:05:58.7361604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjrdme7t/_remote_module_non_scriptable.py
2022-05-18T05:05:58.7689008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptiggg0_e
2022-05-18T05:05:58.7691688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptiggg0_e/_remote_module_non_scriptable.py
2022-05-18T05:05:59.0416403Z ok (4.409s)
2022-05-18T05:05:59.0416593Z 
2022-05-18T05:05:59.0416950Z ----------------------------------------------------------------------
2022-05-18T05:05:59.0417291Z Ran 1 test in 4.409s
2022-05-18T05:05:59.0417464Z 
2022-05-18T05:05:59.0417561Z OK
2022-05-18T05:05:59.0417696Z 
2022-05-18T05:05:59.0418019Z Generating XML reports...
2022-05-18T05:05:59.0458542Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050554.xml
2022-05-18T05:06:00.4362218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:00.4376915Z 
2022-05-18T05:06:00.4377358Z Running tests...
2022-05-18T05:06:00.4377854Z ----------------------------------------------------------------------
2022-05-18T05:06:02.0258740Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:02.0673403Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77652
2022-05-18T05:06:02.0787961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77653
2022-05-18T05:06:03.2816809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:03.2817397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:03.2818195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:03.2818892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:03.2825189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:03.2826034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:04.5829471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2f7g1_1
2022-05-18T05:06:04.5830800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2f7g1_1/_remote_module_non_scriptable.py
2022-05-18T05:06:04.5925676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_1vb_uz
2022-05-18T05:06:04.5928571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_1vb_uz/_remote_module_non_scriptable.py
2022-05-18T05:06:04.7948385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:04.7948942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:05.0867466Z ok (4.649s)
2022-05-18T05:06:05.0867745Z 
2022-05-18T05:06:05.0868304Z ----------------------------------------------------------------------
2022-05-18T05:06:05.0868656Z Ran 1 test in 4.649s
2022-05-18T05:06:05.0868823Z 
2022-05-18T05:06:05.0868919Z OK
2022-05-18T05:06:05.0869056Z 
2022-05-18T05:06:05.0869195Z Generating XML reports...
2022-05-18T05:06:05.0909551Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050600.xml
2022-05-18T05:06:06.4832258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:06.4846498Z 
2022-05-18T05:06:06.4846780Z Running tests...
2022-05-18T05:06:06.4847372Z ----------------------------------------------------------------------
2022-05-18T05:06:08.0333411Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:08.0725121Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77767
2022-05-18T05:06:08.0831498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77768
2022-05-18T05:06:09.2348134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:09.2348691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:09.2349476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:09.2350340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:09.2356119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:09.2356927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:10.5248736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjuoj60s
2022-05-18T05:06:10.5249347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjuoj60s/_remote_module_non_scriptable.py
2022-05-18T05:06:10.5311386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujo71t3j
2022-05-18T05:06:10.5314299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujo71t3j/_remote_module_non_scriptable.py
2022-05-18T05:06:10.7189946Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:06:11.0912490Z ok (4.606s)
2022-05-18T05:06:11.0912711Z 
2022-05-18T05:06:11.0913130Z ----------------------------------------------------------------------
2022-05-18T05:06:11.0913458Z Ran 1 test in 4.607s
2022-05-18T05:06:11.0913623Z 
2022-05-18T05:06:11.0913721Z OK
2022-05-18T05:06:11.0913858Z 
2022-05-18T05:06:11.0914268Z Generating XML reports...
2022-05-18T05:06:11.0954421Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050606.xml
2022-05-18T05:06:12.5081202Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:12.5095439Z 
2022-05-18T05:06:12.5095924Z Running tests...
2022-05-18T05:06:12.5096428Z ----------------------------------------------------------------------
2022-05-18T05:06:14.1013606Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:14.1403296Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77882
2022-05-18T05:06:14.1511185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77883
2022-05-18T05:06:15.3441256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:15.3441882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:15.3442720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:15.3443406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:15.3449579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:15.3450294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:16.6328273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyu5y9gtj
2022-05-18T05:06:16.6329286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyu5y9gtj/_remote_module_non_scriptable.py
2022-05-18T05:06:16.6673623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzuua1hw
2022-05-18T05:06:16.6675711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzuua1hw/_remote_module_non_scriptable.py
2022-05-18T05:06:16.8549216Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:06:16.8550948Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:06:17.1589654Z ok (4.649s)
2022-05-18T05:06:17.1590001Z 
2022-05-18T05:06:17.1590423Z ----------------------------------------------------------------------
2022-05-18T05:06:17.1590776Z Ran 1 test in 4.649s
2022-05-18T05:06:17.1590947Z 
2022-05-18T05:06:17.1591046Z OK
2022-05-18T05:06:17.1591186Z 
2022-05-18T05:06:17.1591310Z Generating XML reports...
2022-05-18T05:06:17.1630986Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050612.xml
2022-05-18T05:06:18.5820522Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:18.5834898Z 
2022-05-18T05:06:18.5835341Z Running tests...
2022-05-18T05:06:18.5835819Z ----------------------------------------------------------------------
2022-05-18T05:06:20.1669207Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:20.2071411Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77997
2022-05-18T05:06:20.2181171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77998
2022-05-18T05:06:21.3778800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:21.3779631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:21.3780482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:21.3781418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:21.3886679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:21.3980856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpesx_eue5
2022-05-18T05:06:21.3983315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpesx_eue5/_remote_module_non_scriptable.py
2022-05-18T05:06:21.4789312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:21.4889038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwgoj9_4
2022-05-18T05:06:21.4891671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwgoj9_4/_remote_module_non_scriptable.py
2022-05-18T05:06:21.5035293Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:06:21.5037099Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:06:21.5041096Z /opt/conda/lib/python3.9/site-packages/torch/autograd/__init__.py:173: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T05:06:21.5042121Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-05-18T05:06:21.5043538Z /opt/conda/lib/python3.9/site-packages/torch/autograd/__init__.py:173: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T05:06:21.5044801Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-05-18T05:06:21.5045877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:21.5046363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:21.5049729Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:06:21.5051758Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:06:21.5055993Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:06:21.5058071Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:06:21.5061479Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:06:21.5063511Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:06:21.5067040Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:06:21.5069115Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:06:21.5072957Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:06:21.5074870Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T05:06:21.7239575Z ok (3.140s)
2022-05-18T05:06:21.7239793Z 
2022-05-18T05:06:21.7240377Z ----------------------------------------------------------------------
2022-05-18T05:06:21.7241033Z Ran 1 test in 3.140s
2022-05-18T05:06:21.7241216Z 
2022-05-18T05:06:21.7241313Z OK
2022-05-18T05:06:21.7241454Z 
2022-05-18T05:06:21.7241593Z Generating XML reports...
2022-05-18T05:06:21.7283043Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050618.xml
2022-05-18T05:06:23.1560448Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:23.1574687Z 
2022-05-18T05:06:23.1575072Z Running tests...
2022-05-18T05:06:23.1575641Z ----------------------------------------------------------------------
2022-05-18T05:06:24.7576724Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:24.7979372Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78110
2022-05-18T05:06:24.8088268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78111
2022-05-18T05:06:26.0262239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:26.0262862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:26.0263661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:26.0264377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:26.0372537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:26.1275543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:27.3069850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5ok5_4l
2022-05-18T05:06:27.3070473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5ok5_4l/_remote_module_non_scriptable.py
2022-05-18T05:06:27.3658071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxw5hc3i
2022-05-18T05:06:27.3659894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxw5hc3i/_remote_module_non_scriptable.py
2022-05-18T05:06:27.5551817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:27.5552675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:27.9171452Z ok (4.759s)
2022-05-18T05:06:27.9171693Z 
2022-05-18T05:06:27.9172397Z ----------------------------------------------------------------------
2022-05-18T05:06:27.9172781Z Ran 1 test in 4.760s
2022-05-18T05:06:27.9172952Z 
2022-05-18T05:06:27.9173030Z OK
2022-05-18T05:06:27.9173170Z 
2022-05-18T05:06:27.9173317Z Generating XML reports...
2022-05-18T05:06:27.9213838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050623.xml
2022-05-18T05:06:29.3386077Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:29.3400207Z 
2022-05-18T05:06:29.3400570Z Running tests...
2022-05-18T05:06:29.3401254Z ----------------------------------------------------------------------
2022-05-18T05:06:30.9186173Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:30.9579090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78225
2022-05-18T05:06:30.9686540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78226
2022-05-18T05:06:32.1448004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:32.1448847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:32.1449672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:32.1450653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:32.1555389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:32.2462044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:33.4308974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbb5xjye
2022-05-18T05:06:33.4312690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbb5xjye/_remote_module_non_scriptable.py
2022-05-18T05:06:33.5075402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusbs3uyg
2022-05-18T05:06:33.5076050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusbs3uyg/_remote_module_non_scriptable.py
2022-05-18T05:06:33.5284869Z /opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T05:06:33.5286734Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T05:06:33.5288933Z /opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py:1053: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-05-18T05:06:33.5290696Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-05-18T05:06:33.9767475Z ok (4.636s)
2022-05-18T05:06:33.9767696Z 
2022-05-18T05:06:33.9768084Z ----------------------------------------------------------------------
2022-05-18T05:06:33.9768436Z Ran 1 test in 4.637s
2022-05-18T05:06:33.9768604Z 
2022-05-18T05:06:33.9768703Z OK
2022-05-18T05:06:33.9768840Z 
2022-05-18T05:06:33.9768972Z Generating XML reports...
2022-05-18T05:06:33.9810132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050629.xml
2022-05-18T05:06:35.4072857Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:35.4087434Z 
2022-05-18T05:06:35.4087885Z Running tests...
2022-05-18T05:06:35.4088400Z ----------------------------------------------------------------------
2022-05-18T05:06:37.0006967Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:37.0409896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78370
2022-05-18T05:06:37.0520048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78371
2022-05-18T05:06:38.2303118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:38.2303679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:38.2304464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:38.2305174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:38.2311405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:38.2311915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:39.5095357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcu3_afp
2022-05-18T05:06:39.5096609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcu3_afp/_remote_module_non_scriptable.py
2022-05-18T05:06:39.5595497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppasd_vva
2022-05-18T05:06:39.5597686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppasd_vva/_remote_module_non_scriptable.py
2022-05-18T05:06:39.7480853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:39.7481385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:39.7705191Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpxcu3_afp'>
2022-05-18T05:06:39.7705702Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:06:39.7706570Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmppasd_vva'>
2022-05-18T05:06:39.7707041Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:06:40.0599973Z ok (4.651s)
2022-05-18T05:06:40.0600227Z 
2022-05-18T05:06:40.0600843Z ----------------------------------------------------------------------
2022-05-18T05:06:40.0601181Z Ran 1 test in 4.651s
2022-05-18T05:06:40.0601347Z 
2022-05-18T05:06:40.0601441Z OK
2022-05-18T05:06:40.0601577Z 
2022-05-18T05:06:40.0601798Z Generating XML reports...
2022-05-18T05:06:40.0643145Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050635.xml
2022-05-18T05:06:41.4922405Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:41.4936602Z 
2022-05-18T05:06:41.4937010Z Running tests...
2022-05-18T05:06:41.4937683Z ----------------------------------------------------------------------
2022-05-18T05:06:43.0692779Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:43.0846407Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.591s)
2022-05-18T05:06:43.0847085Z 
2022-05-18T05:06:43.0847537Z ----------------------------------------------------------------------
2022-05-18T05:06:43.0848136Z Ran 1 test in 1.591s
2022-05-18T05:06:43.0848377Z 
2022-05-18T05:06:43.0848582Z OK (skipped=1)
2022-05-18T05:06:43.0848853Z 
2022-05-18T05:06:43.0848983Z Generating XML reports...
2022-05-18T05:06:43.0880110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050641.xml
2022-05-18T05:06:44.4619689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:44.4634294Z 
2022-05-18T05:06:44.4634443Z Running tests...
2022-05-18T05:06:44.4635092Z ----------------------------------------------------------------------
2022-05-18T05:06:46.0525284Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:46.0928046Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78519
2022-05-18T05:06:46.1035475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78520
2022-05-18T05:06:47.2577865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:47.2578441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:47.2579249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:47.2579955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:47.2586480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:47.2587233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:47.2697267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:06:47.2697793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:06:47.2698506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:06:47.2699203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:06:48.5601414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgnkxqri
2022-05-18T05:06:48.5602348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgnkxqri/_remote_module_non_scriptable.py
2022-05-18T05:06:48.5702092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpht2fc707
2022-05-18T05:06:48.5704974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpht2fc707/_remote_module_non_scriptable.py
2022-05-18T05:06:48.7636735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:48.7637730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:48.7654056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:48.7654926Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:49.3120589Z ok (4.848s)
2022-05-18T05:06:49.3120946Z 
2022-05-18T05:06:49.3121436Z ----------------------------------------------------------------------
2022-05-18T05:06:49.3121826Z Ran 1 test in 4.849s
2022-05-18T05:06:49.3121999Z 
2022-05-18T05:06:49.3122097Z OK
2022-05-18T05:06:49.3122219Z 
2022-05-18T05:06:49.3122355Z Generating XML reports...
2022-05-18T05:06:49.3164153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050644.xml
2022-05-18T05:06:50.7452570Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:50.7466405Z 
2022-05-18T05:06:50.7466647Z Running tests...
2022-05-18T05:06:50.7467373Z ----------------------------------------------------------------------
2022-05-18T05:06:52.3187486Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:52.3582497Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78640
2022-05-18T05:06:52.3691433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78641
2022-05-18T05:06:53.5803797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:06:53.5804393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:06:53.5805201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:53.5805904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:06:53.5913430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:06:53.5914983Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T05:06:53.6818037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:06:53.6818629Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T05:06:54.8656661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxes9x6tu
2022-05-18T05:06:54.8657832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxes9x6tu/_remote_module_non_scriptable.py
2022-05-18T05:06:54.9261378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ir8bikw
2022-05-18T05:06:54.9262450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ir8bikw/_remote_module_non_scriptable.py
2022-05-18T05:06:55.1183300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:55.1183859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:55.1199323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:55.1200272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:55.1449494Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T05:06:55.1450100Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T05:06:55.3930694Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T05:06:55.3931285Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-05-18T05:06:55.4009004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:55.4009786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:55.4026441Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:55.4026992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:55.4266178Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T05:06:55.4266775Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-05-18T05:06:55.5776816Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T05:06:55.5777611Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-05-18T05:06:55.5853068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:55.5853592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:55.5870589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:55.5871095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:06:56.1782499Z ok (5.431s)
2022-05-18T05:06:56.1783679Z 
2022-05-18T05:06:56.1784108Z ----------------------------------------------------------------------
2022-05-18T05:06:56.1784495Z Ran 1 test in 5.432s
2022-05-18T05:06:56.1784663Z 
2022-05-18T05:06:56.1784741Z OK
2022-05-18T05:06:56.1784879Z 
2022-05-18T05:06:56.1785014Z Generating XML reports...
2022-05-18T05:06:56.1824578Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050650.xml
2022-05-18T05:06:57.6025625Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:06:57.6040095Z 
2022-05-18T05:06:57.6040446Z Running tests...
2022-05-18T05:06:57.6040885Z ----------------------------------------------------------------------
2022-05-18T05:06:59.1964170Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:06:59.2367962Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78755
2022-05-18T05:06:59.2477295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78756
2022-05-18T05:07:00.3968786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:00.3969339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:00.3970173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:00.3970878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:00.3977044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:00.3977538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:00.3979989Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T05:07:00.3981190Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T05:07:01.6796389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xex5cyl
2022-05-18T05:07:01.6797016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xex5cyl/_remote_module_non_scriptable.py
2022-05-18T05:07:01.6912962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprntk75v_
2022-05-18T05:07:01.6915717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprntk75v_/_remote_module_non_scriptable.py
2022-05-18T05:07:01.8824981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:01.8825545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:01.8841852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:01.8842675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:01.8848632Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T05:07:01.8849199Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T05:07:01.8880733Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T05:07:01.8881364Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T05:07:01.8882651Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T05:07:01.8883328Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T05:07:01.8883976Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T05:07:01.8884635Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-05-18T05:07:02.2261991Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T05:07:02.2263537Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T05:07:02.2344398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:02.2344914Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:02.2361955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:02.2362455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:02.2367639Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T05:07:02.2368229Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-05-18T05:07:02.2399007Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T05:07:02.2399633Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-05-18T05:07:02.2400457Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T05:07:02.2401125Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-05-18T05:07:02.8571002Z ok (5.253s)
2022-05-18T05:07:02.8571409Z 
2022-05-18T05:07:02.8572043Z ----------------------------------------------------------------------
2022-05-18T05:07:02.8572726Z Ran 1 test in 5.253s
2022-05-18T05:07:02.8573035Z 
2022-05-18T05:07:02.8573193Z OK
2022-05-18T05:07:02.8573432Z 
2022-05-18T05:07:02.8573680Z Generating XML reports...
2022-05-18T05:07:02.8616247Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050657.xml
2022-05-18T05:07:04.2899258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:04.2913555Z 
2022-05-18T05:07:04.2913801Z Running tests...
2022-05-18T05:07:04.2914252Z ----------------------------------------------------------------------
2022-05-18T05:07:05.8555641Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:05.8950241Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78870
2022-05-18T05:07:05.9058988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78871
2022-05-18T05:07:07.0696355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:07.0696919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:07.0697738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:07.0698415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:07.0703928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:07.0704421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:08.3757541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_x1ncqn
2022-05-18T05:07:08.3758917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_x1ncqn/_remote_module_non_scriptable.py
2022-05-18T05:07:08.3819242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawuemmwy
2022-05-18T05:07:08.3821987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawuemmwy/_remote_module_non_scriptable.py
2022-05-18T05:07:08.6193079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:08.6193737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:08.6907449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:08.6908080Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:09.1140406Z ok (4.822s)
2022-05-18T05:07:09.1140708Z 
2022-05-18T05:07:09.1141450Z ----------------------------------------------------------------------
2022-05-18T05:07:09.1142129Z Ran 1 test in 4.823s
2022-05-18T05:07:09.1142346Z 
2022-05-18T05:07:09.1142457Z OK
2022-05-18T05:07:09.1142596Z 
2022-05-18T05:07:09.1142728Z Generating XML reports...
2022-05-18T05:07:09.1183748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050704.xml
2022-05-18T05:07:10.5086945Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:10.5100382Z 
2022-05-18T05:07:10.5100717Z Running tests...
2022-05-18T05:07:10.5101679Z ----------------------------------------------------------------------
2022-05-18T05:07:12.0472327Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:12.0864172Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79015
2022-05-18T05:07:12.0973847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79016
2022-05-18T05:07:13.2705555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:13.2706632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:13.2707407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:13.2708313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:13.2815713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:13.3719337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:14.5314989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgp32hu8
2022-05-18T05:07:14.5315599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgp32hu8/_remote_module_non_scriptable.py
2022-05-18T05:07:14.6615142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph22llmi9
2022-05-18T05:07:14.6615784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph22llmi9/_remote_module_non_scriptable.py
2022-05-18T05:07:14.9005526Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:14.9006093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:14.9697532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:14.9698091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:15.3055189Z ok (4.795s)
2022-05-18T05:07:15.3055708Z 
2022-05-18T05:07:15.3056284Z ----------------------------------------------------------------------
2022-05-18T05:07:15.3056613Z Ran 1 test in 4.795s
2022-05-18T05:07:15.3056781Z 
2022-05-18T05:07:15.3056882Z OK
2022-05-18T05:07:15.3057027Z 
2022-05-18T05:07:15.3057155Z Generating XML reports...
2022-05-18T05:07:15.3097959Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050710.xml
2022-05-18T05:07:16.7000901Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:16.7014821Z 
2022-05-18T05:07:16.7015468Z Running tests...
2022-05-18T05:07:16.7016060Z ----------------------------------------------------------------------
2022-05-18T05:07:18.2490218Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:18.2885032Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79160
2022-05-18T05:07:18.2990428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79161
2022-05-18T05:07:19.4992504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:19.4993096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:19.4993892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:19.4994585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:19.5000888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:19.5002102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:20.7799381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw2emq91s
2022-05-18T05:07:20.7800015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw2emq91s/_remote_module_non_scriptable.py
2022-05-18T05:07:20.8209392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpepkznvb0
2022-05-18T05:07:20.8211542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpepkznvb0/_remote_module_non_scriptable.py
2022-05-18T05:07:21.0610484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:21.0611319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:21.1313438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:21.1314025Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:21.5072071Z ok (4.805s)
2022-05-18T05:07:21.5072292Z 
2022-05-18T05:07:21.5072699Z ----------------------------------------------------------------------
2022-05-18T05:07:21.5073026Z Ran 1 test in 4.806s
2022-05-18T05:07:21.5073198Z 
2022-05-18T05:07:21.5073297Z OK
2022-05-18T05:07:21.5073436Z 
2022-05-18T05:07:21.5073569Z Generating XML reports...
2022-05-18T05:07:21.5113556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050716.xml
2022-05-18T05:07:22.9203872Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:22.9218410Z 
2022-05-18T05:07:22.9218679Z Running tests...
2022-05-18T05:07:22.9219125Z ----------------------------------------------------------------------
2022-05-18T05:07:24.5010619Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:24.5412416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79305
2022-05-18T05:07:24.5522305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79306
2022-05-18T05:07:25.7149364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:25.7149927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:25.7150966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:25.7151659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:25.7256821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:25.8164631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:26.9850991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpctxwkv6d
2022-05-18T05:07:26.9852349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpctxwkv6d/_remote_module_non_scriptable.py
2022-05-18T05:07:27.1124217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqy06j7p2
2022-05-18T05:07:27.1125571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqy06j7p2/_remote_module_non_scriptable.py
2022-05-18T05:07:27.3474275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:27.3474838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:27.4153983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:27.4154535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:27.7604839Z ok (4.838s)
2022-05-18T05:07:27.7605104Z 
2022-05-18T05:07:27.7605501Z ----------------------------------------------------------------------
2022-05-18T05:07:27.7605846Z Ran 1 test in 4.839s
2022-05-18T05:07:27.7606018Z 
2022-05-18T05:07:27.7606097Z OK
2022-05-18T05:07:27.7606231Z 
2022-05-18T05:07:27.7606367Z Generating XML reports...
2022-05-18T05:07:27.7647460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050722.xml
2022-05-18T05:07:29.1788180Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:29.1802354Z 
2022-05-18T05:07:29.1802593Z Running tests...
2022-05-18T05:07:29.1803522Z ----------------------------------------------------------------------
2022-05-18T05:07:30.7633758Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:30.8039237Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79450
2022-05-18T05:07:30.8149568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79451
2022-05-18T05:07:31.9734956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:31.9735525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:31.9736341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:31.9737046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:31.9742718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:31.9743214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:33.2933576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq886gymb
2022-05-18T05:07:33.2934543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq886gymb/_remote_module_non_scriptable.py
2022-05-18T05:07:33.3022759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrx1lska
2022-05-18T05:07:33.3025531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrx1lska/_remote_module_non_scriptable.py
2022-05-18T05:07:33.5522305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:33.5522897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:33.6217038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:33.6217598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:34.0234610Z ok (4.843s)
2022-05-18T05:07:34.0234853Z 
2022-05-18T05:07:34.0235257Z ----------------------------------------------------------------------
2022-05-18T05:07:34.0235605Z Ran 1 test in 4.843s
2022-05-18T05:07:34.0235773Z 
2022-05-18T05:07:34.0235868Z OK
2022-05-18T05:07:34.0236003Z 
2022-05-18T05:07:34.0238989Z Generating XML reports...
2022-05-18T05:07:34.0277027Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050729.xml
2022-05-18T05:07:35.4190198Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:35.4205622Z 
2022-05-18T05:07:35.4205896Z Running tests...
2022-05-18T05:07:35.4206325Z ----------------------------------------------------------------------
2022-05-18T05:07:36.9971520Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:37.0363431Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79595
2022-05-18T05:07:37.0471477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79596
2022-05-18T05:07:38.2019473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:38.2020539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:38.2021873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:38.2022589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:38.2129415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:38.3034246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:39.4783925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5q_9dwch
2022-05-18T05:07:39.4785107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5q_9dwch/_remote_module_non_scriptable.py
2022-05-18T05:07:39.5784242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmzpxi16
2022-05-18T05:07:39.5785420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmzpxi16/_remote_module_non_scriptable.py
2022-05-18T05:07:39.8262189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:39.8262761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:39.8940693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:39.8941289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:40.2556431Z ok (4.835s)
2022-05-18T05:07:40.2556664Z 
2022-05-18T05:07:40.2557068Z ----------------------------------------------------------------------
2022-05-18T05:07:40.2557397Z Ran 1 test in 4.835s
2022-05-18T05:07:40.2557570Z 
2022-05-18T05:07:40.2557668Z OK
2022-05-18T05:07:40.2560286Z 
2022-05-18T05:07:40.2560631Z Generating XML reports...
2022-05-18T05:07:40.2599219Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050735.xml
2022-05-18T05:07:41.6930665Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:41.6945481Z 
2022-05-18T05:07:41.6945756Z Running tests...
2022-05-18T05:07:41.6946207Z ----------------------------------------------------------------------
2022-05-18T05:07:43.2782844Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:43.3188209Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79740
2022-05-18T05:07:43.3297121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79741
2022-05-18T05:07:44.5908664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:44.5909224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:44.5910013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:44.5910736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:44.5916670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:44.5917169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:45.8864574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqt2bwela
2022-05-18T05:07:45.8865472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqt2bwela/_remote_module_non_scriptable.py
2022-05-18T05:07:45.9060664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjo815lye
2022-05-18T05:07:45.9062047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjo815lye/_remote_module_non_scriptable.py
2022-05-18T05:07:46.1406666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:46.1407707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:46.2113483Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:46.2114929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:46.6384937Z ok (4.944s)
2022-05-18T05:07:46.6385148Z 
2022-05-18T05:07:46.6385551Z ----------------------------------------------------------------------
2022-05-18T05:07:46.6385909Z Ran 1 test in 4.944s
2022-05-18T05:07:46.6386074Z 
2022-05-18T05:07:46.6386154Z OK
2022-05-18T05:07:46.6386289Z 
2022-05-18T05:07:46.6386434Z Generating XML reports...
2022-05-18T05:07:46.6427447Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050741.xml
2022-05-18T05:07:48.0215044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:48.0230807Z 
2022-05-18T05:07:48.0231459Z Running tests...
2022-05-18T05:07:48.0232339Z ----------------------------------------------------------------------
2022-05-18T05:07:49.5987656Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:49.6393253Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79885
2022-05-18T05:07:49.6502533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79886
2022-05-18T05:07:50.8775347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:50.8775909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:50.8776690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:50.8777657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:50.8883060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:50.9790141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:52.1319425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt41rd9xr
2022-05-18T05:07:52.1320069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt41rd9xr/_remote_module_non_scriptable.py
2022-05-18T05:07:52.2797122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcf7bk2_e
2022-05-18T05:07:52.2799051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcf7bk2_e/_remote_module_non_scriptable.py
2022-05-18T05:07:52.5134378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:52.5134969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:52.5793409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:52.5793970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:52.9588288Z ok (4.935s)
2022-05-18T05:07:52.9588517Z 
2022-05-18T05:07:52.9588916Z ----------------------------------------------------------------------
2022-05-18T05:07:52.9589241Z Ran 1 test in 4.936s
2022-05-18T05:07:52.9589409Z 
2022-05-18T05:07:52.9589511Z OK
2022-05-18T05:07:52.9589647Z 
2022-05-18T05:07:52.9589782Z Generating XML reports...
2022-05-18T05:07:52.9630494Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050748.xml
2022-05-18T05:07:54.3801271Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:07:54.3815539Z 
2022-05-18T05:07:54.3816067Z Running tests...
2022-05-18T05:07:54.3816566Z ----------------------------------------------------------------------
2022-05-18T05:07:55.9495917Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:07:55.9900291Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80030
2022-05-18T05:07:56.0010732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80031
2022-05-18T05:07:57.1803656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:07:57.1804209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:07:57.1805000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:57.1805718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:07:57.1811691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:07:57.1812223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:07:58.4912372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbm77zutp
2022-05-18T05:07:58.4913330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbm77zutp/_remote_module_non_scriptable.py
2022-05-18T05:07:58.4941453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjh2lh6ws
2022-05-18T05:07:58.4944311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjh2lh6ws/_remote_module_non_scriptable.py
2022-05-18T05:07:58.7429188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:58.7430163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:58.8112017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:58.8112564Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:07:59.2093898Z ok (4.827s)
2022-05-18T05:07:59.2094138Z 
2022-05-18T05:07:59.2094548Z ----------------------------------------------------------------------
2022-05-18T05:07:59.2094891Z Ran 1 test in 4.828s
2022-05-18T05:07:59.2095063Z 
2022-05-18T05:07:59.2095158Z OK
2022-05-18T05:07:59.2095298Z 
2022-05-18T05:07:59.2095415Z Generating XML reports...
2022-05-18T05:07:59.2136580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050754.xml
2022-05-18T05:08:00.6314015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:00.6328838Z 
2022-05-18T05:08:00.6329212Z Running tests...
2022-05-18T05:08:00.6329661Z ----------------------------------------------------------------------
2022-05-18T05:08:02.2195929Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:02.2602789Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80175
2022-05-18T05:08:02.2712968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80176
2022-05-18T05:08:03.4826679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:03.4827237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:03.4828093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:03.4828813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:03.4934869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:03.5841347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:04.7793801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxv8qirrx
2022-05-18T05:08:04.7794789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxv8qirrx/_remote_module_non_scriptable.py
2022-05-18T05:08:04.8644511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzey4xug9
2022-05-18T05:08:04.8645119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzey4xug9/_remote_module_non_scriptable.py
2022-05-18T05:08:05.1140229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:05.1140795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:05.1819906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:05.1820425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:05.5800027Z ok (4.947s)
2022-05-18T05:08:05.5800257Z 
2022-05-18T05:08:05.5800673Z ----------------------------------------------------------------------
2022-05-18T05:08:05.5801214Z Ran 1 test in 4.947s
2022-05-18T05:08:05.5801503Z 
2022-05-18T05:08:05.5801668Z OK
2022-05-18T05:08:05.5801889Z 
2022-05-18T05:08:05.5802136Z Generating XML reports...
2022-05-18T05:08:05.5843663Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050800.xml
2022-05-18T05:08:07.0023789Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:07.0037530Z 
2022-05-18T05:08:07.0038212Z Running tests...
2022-05-18T05:08:07.0039534Z ----------------------------------------------------------------------
2022-05-18T05:08:08.5984279Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:08.6389742Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80320
2022-05-18T05:08:08.6498810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80321
2022-05-18T05:08:09.8140676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:09.8141222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:09.8142031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:09.8142731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:09.8249993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:09.9154778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:11.0818342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4uxzxp6d
2022-05-18T05:08:11.0819325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4uxzxp6d/_remote_module_non_scriptable.py
2022-05-18T05:08:11.2135604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdcj4l32
2022-05-18T05:08:11.2136537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdcj4l32/_remote_module_non_scriptable.py
2022-05-18T05:08:11.4483809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:11.4484371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:11.5135095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:11.5135628Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:11.8582351Z ok (4.854s)
2022-05-18T05:08:11.8582561Z 
2022-05-18T05:08:11.8583192Z ----------------------------------------------------------------------
2022-05-18T05:08:11.8583566Z Ran 1 test in 4.854s
2022-05-18T05:08:11.8583735Z 
2022-05-18T05:08:11.8583813Z OK
2022-05-18T05:08:11.8583952Z 
2022-05-18T05:08:11.8584087Z Generating XML reports...
2022-05-18T05:08:11.8624735Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050806.xml
2022-05-18T05:08:13.2514981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:13.2528683Z 
2022-05-18T05:08:13.2529032Z Running tests...
2022-05-18T05:08:13.2529481Z ----------------------------------------------------------------------
2022-05-18T05:08:14.7824822Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:14.8221543Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80465
2022-05-18T05:08:14.8331948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80466
2022-05-18T05:08:16.0437116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:16.0437690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:16.0438780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:16.0439469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:16.0546819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:16.1451426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:17.3020390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsditwidd
2022-05-18T05:08:17.3021040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsditwidd/_remote_module_non_scriptable.py
2022-05-18T05:08:17.4248312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbthxdxk
2022-05-18T05:08:17.4249342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbthxdxk/_remote_module_non_scriptable.py
2022-05-18T05:08:17.6595117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:17.6595678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:17.7268296Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:17.7268842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:18.1416772Z ok (4.888s)
2022-05-18T05:08:18.1417120Z 
2022-05-18T05:08:18.1417587Z ----------------------------------------------------------------------
2022-05-18T05:08:18.1417957Z Ran 1 test in 4.889s
2022-05-18T05:08:18.1418154Z 
2022-05-18T05:08:18.1418323Z OK
2022-05-18T05:08:18.1418576Z 
2022-05-18T05:08:18.1418740Z Generating XML reports...
2022-05-18T05:08:18.1458325Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050813.xml
2022-05-18T05:08:19.5582526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:19.5596606Z 
2022-05-18T05:08:19.5596922Z Running tests...
2022-05-18T05:08:19.5597374Z ----------------------------------------------------------------------
2022-05-18T05:08:21.1248283Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:21.1396438Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.580s)
2022-05-18T05:08:21.1397084Z 
2022-05-18T05:08:21.1397390Z ----------------------------------------------------------------------
2022-05-18T05:08:21.1397743Z Ran 1 test in 1.580s
2022-05-18T05:08:21.1397922Z 
2022-05-18T05:08:21.1398036Z OK (skipped=1)
2022-05-18T05:08:21.1398537Z 
2022-05-18T05:08:21.1398656Z Generating XML reports...
2022-05-18T05:08:21.1428677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050819.xml
2022-05-18T05:08:22.5006788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:22.5020732Z 
2022-05-18T05:08:22.5020991Z Running tests...
2022-05-18T05:08:22.5021470Z ----------------------------------------------------------------------
2022-05-18T05:08:24.0775849Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:24.1179725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80644
2022-05-18T05:08:24.1289353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80645
2022-05-18T05:08:25.3072537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:25.3073118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:25.3073915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:25.3074600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:25.3182127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:25.4087071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:26.5478352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0z3xlpm7
2022-05-18T05:08:26.5478997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0z3xlpm7/_remote_module_non_scriptable.py
2022-05-18T05:08:26.6452342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppa0v77y6
2022-05-18T05:08:26.6453424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppa0v77y6/_remote_module_non_scriptable.py
2022-05-18T05:08:27.7379538Z ok (5.235s)
2022-05-18T05:08:27.7379950Z 
2022-05-18T05:08:27.7380726Z ----------------------------------------------------------------------
2022-05-18T05:08:27.7381250Z Ran 1 test in 5.236s
2022-05-18T05:08:27.7381445Z 
2022-05-18T05:08:27.7381542Z OK
2022-05-18T05:08:27.7383190Z 
2022-05-18T05:08:27.7383457Z Generating XML reports...
2022-05-18T05:08:27.7422606Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050822.xml
2022-05-18T05:08:29.1364751Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:29.1378761Z 
2022-05-18T05:08:29.1379098Z Running tests...
2022-05-18T05:08:29.1379553Z ----------------------------------------------------------------------
2022-05-18T05:08:30.6853029Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:30.7246975Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80755
2022-05-18T05:08:30.7355940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80756
2022-05-18T05:08:31.9022602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:31.9023177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:31.9023972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:31.9024879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:31.9130205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:32.0036978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:33.3274577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2t46h9lb
2022-05-18T05:08:33.3275478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2t46h9lb/_remote_module_non_scriptable.py
2022-05-18T05:08:33.4135771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24ab_03w
2022-05-18T05:08:33.4136972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24ab_03w/_remote_module_non_scriptable.py
2022-05-18T05:08:33.4318642Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:33.4319644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:33.4426435Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp24ab_03w'>
2022-05-18T05:08:33.4426944Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:08:33.4446062Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp2t46h9lb'>
2022-05-18T05:08:33.4446564Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:08:33.7436187Z ok (4.605s)
2022-05-18T05:08:33.7436445Z 
2022-05-18T05:08:33.7436834Z ----------------------------------------------------------------------
2022-05-18T05:08:33.7437486Z Ran 1 test in 4.606s
2022-05-18T05:08:33.7437665Z 
2022-05-18T05:08:33.7437763Z OK
2022-05-18T05:08:33.7437894Z 
2022-05-18T05:08:33.7438036Z Generating XML reports...
2022-05-18T05:08:33.7478330Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050829.xml
2022-05-18T05:08:35.1659271Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:35.1674583Z 
2022-05-18T05:08:35.1674820Z Running tests...
2022-05-18T05:08:35.1675262Z ----------------------------------------------------------------------
2022-05-18T05:08:36.7486984Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:36.7879421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80870
2022-05-18T05:08:36.7986606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80871
2022-05-18T05:08:38.0183517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:38.0184075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:38.0184900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:38.0185604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:38.0291152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:38.0392623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpgn1qqw
2022-05-18T05:08:38.0395482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpgn1qqw/_remote_module_non_scriptable.py
2022-05-18T05:08:38.1193305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:38.1299426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08iiulz9
2022-05-18T05:08:38.1302455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08iiulz9/_remote_module_non_scriptable.py
2022-05-18T05:08:38.1472006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:38.1472545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:38.4041018Z ok (3.236s)
2022-05-18T05:08:38.4041274Z 
2022-05-18T05:08:38.4041895Z ----------------------------------------------------------------------
2022-05-18T05:08:38.4042361Z Ran 1 test in 3.237s
2022-05-18T05:08:38.4042531Z 
2022-05-18T05:08:38.4042637Z OK
2022-05-18T05:08:38.4042755Z 
2022-05-18T05:08:38.4042891Z Generating XML reports...
2022-05-18T05:08:38.4083047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050835.xml
2022-05-18T05:08:39.8031620Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:39.8046178Z 
2022-05-18T05:08:39.8046707Z Running tests...
2022-05-18T05:08:39.8047328Z ----------------------------------------------------------------------
2022-05-18T05:08:41.3921771Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:41.4329616Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81013
2022-05-18T05:08:41.4438623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81014
2022-05-18T05:08:42.5838468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:42.5839299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:42.5840091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:42.5841098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:42.5846257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:42.5847748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:43.8680650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbpyoiz3
2022-05-18T05:08:43.8681732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbpyoiz3/_remote_module_non_scriptable.py
2022-05-18T05:08:43.8887803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kpcgn73
2022-05-18T05:08:43.8890664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kpcgn73/_remote_module_non_scriptable.py
2022-05-18T05:08:44.0796162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:44.0796738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:08:44.4518086Z ok (4.647s)
2022-05-18T05:08:44.4518625Z 
2022-05-18T05:08:44.4519058Z ----------------------------------------------------------------------
2022-05-18T05:08:44.4519411Z Ran 1 test in 4.647s
2022-05-18T05:08:44.4519566Z 
2022-05-18T05:08:44.4519669Z OK
2022-05-18T05:08:44.4519806Z 
2022-05-18T05:08:44.4522450Z Generating XML reports...
2022-05-18T05:08:44.4560443Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050839.xml
2022-05-18T05:08:45.8744417Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:45.8758823Z 
2022-05-18T05:08:45.8759113Z Running tests...
2022-05-18T05:08:45.8759538Z ----------------------------------------------------------------------
2022-05-18T05:08:47.4631502Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:47.5034124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81128
2022-05-18T05:08:47.5144360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81129
2022-05-18T05:08:48.6774145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:48.6774717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:48.6775498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:48.6776176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:48.6884408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:48.7786077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:48.7902280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:08:48.7902836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:08:48.7903542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:08:48.7904246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:08:48.8010899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:08:48.8011402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:08:48.8012093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:08:48.8012978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:08:50.0830948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpix0jjt7_
2022-05-18T05:08:50.0831574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpix0jjt7_/_remote_module_non_scriptable.py
2022-05-18T05:08:50.1078678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0h_2iyay
2022-05-18T05:08:50.1081499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0h_2iyay/_remote_module_non_scriptable.py
2022-05-18T05:08:50.4221206Z ok (4.546s)
2022-05-18T05:08:50.4221424Z 
2022-05-18T05:08:50.4222011Z ----------------------------------------------------------------------
2022-05-18T05:08:50.4222718Z Ran 1 test in 4.546s
2022-05-18T05:08:50.4223065Z 
2022-05-18T05:08:50.4223263Z OK
2022-05-18T05:08:50.4223416Z 
2022-05-18T05:08:50.4223553Z Generating XML reports...
2022-05-18T05:08:50.4264354Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050845.xml
2022-05-18T05:08:51.8633293Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:08:51.8647637Z 
2022-05-18T05:08:51.8648079Z Running tests...
2022-05-18T05:08:51.8649038Z ----------------------------------------------------------------------
2022-05-18T05:08:53.4332719Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:08:53.4725039Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81251
2022-05-18T05:08:53.4832303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81252
2022-05-18T05:08:54.6928014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:08:54.6928599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:08:54.6929396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:54.6930284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:08:54.6936154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:08:54.6937219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:08:54.7144267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:08:54.7144792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:08:54.7145468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:08:54.7146173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:08:54.7352904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:08:54.7353429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:08:54.7354092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:08:54.7354786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:08:56.0168092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzce6k6sq
2022-05-18T05:08:56.0168936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzce6k6sq/_remote_module_non_scriptable.py
2022-05-18T05:08:56.0248252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzy81iu4z
2022-05-18T05:08:56.0251489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzy81iu4z/_remote_module_non_scriptable.py
2022-05-18T05:09:06.4082143Z ok (14.543s)
2022-05-18T05:09:06.4082530Z 
2022-05-18T05:09:06.4083415Z ----------------------------------------------------------------------
2022-05-18T05:09:06.4084158Z Ran 1 test in 14.543s
2022-05-18T05:09:06.4084349Z 
2022-05-18T05:09:06.4085039Z OK
2022-05-18T05:09:06.4085239Z 
2022-05-18T05:09:06.4085384Z Generating XML reports...
2022-05-18T05:09:06.4124655Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050851.xml
2022-05-18T05:09:07.8369124Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:07.8383394Z 
2022-05-18T05:09:07.8383682Z Running tests...
2022-05-18T05:09:07.8384145Z ----------------------------------------------------------------------
2022-05-18T05:09:09.4194800Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:09.4598515Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81374
2022-05-18T05:09:09.4707489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81375
2022-05-18T05:09:10.6680386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:10.6681347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:10.6682153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:10.6682861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:10.6788089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:10.7694430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:11.9605210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstksm8t5
2022-05-18T05:09:11.9606734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstksm8t5/_remote_module_non_scriptable.py
2022-05-18T05:09:12.0385840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9k9249e5
2022-05-18T05:09:12.0386931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9k9249e5/_remote_module_non_scriptable.py
2022-05-18T05:09:13.1798023Z ok (5.341s)
2022-05-18T05:09:13.1798680Z 
2022-05-18T05:09:13.1799193Z ----------------------------------------------------------------------
2022-05-18T05:09:13.1799527Z Ran 1 test in 5.341s
2022-05-18T05:09:13.1799698Z 
2022-05-18T05:09:13.1799796Z OK
2022-05-18T05:09:13.1799951Z 
2022-05-18T05:09:13.1800146Z Generating XML reports...
2022-05-18T05:09:13.1840689Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050907.xml
2022-05-18T05:09:14.5952568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:14.5967291Z 
2022-05-18T05:09:14.5967569Z Running tests...
2022-05-18T05:09:14.5967999Z ----------------------------------------------------------------------
2022-05-18T05:09:16.1652637Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:16.2045642Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81489
2022-05-18T05:09:16.2154493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81490
2022-05-18T05:09:17.3714867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:17.3715718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:17.3716522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:17.3717216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:17.3722818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:17.3723630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:18.6430016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2tpyoha
2022-05-18T05:09:18.6430622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2tpyoha/_remote_module_non_scriptable.py
2022-05-18T05:09:18.6733614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1cdhsvf
2022-05-18T05:09:18.6736281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1cdhsvf/_remote_module_non_scriptable.py
2022-05-18T05:09:19.8245292Z ok (5.227s)
2022-05-18T05:09:19.8245598Z 
2022-05-18T05:09:19.8246161Z ----------------------------------------------------------------------
2022-05-18T05:09:19.8246516Z Ran 1 test in 5.228s
2022-05-18T05:09:19.8246689Z 
2022-05-18T05:09:19.8246785Z OK
2022-05-18T05:09:19.8246921Z 
2022-05-18T05:09:19.8247058Z Generating XML reports...
2022-05-18T05:09:19.8287190Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050914.xml
2022-05-18T05:09:21.2513304Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:21.2528591Z 
2022-05-18T05:09:21.2528871Z Running tests...
2022-05-18T05:09:21.2529319Z ----------------------------------------------------------------------
2022-05-18T05:09:22.8258855Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:22.8664070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81604
2022-05-18T05:09:22.8773183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81605
2022-05-18T05:09:24.0799325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:24.0799955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:24.0800781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:24.0801488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:24.0906806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:24.1813468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:25.3172907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmo27iui9
2022-05-18T05:09:25.3173513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmo27iui9/_remote_module_non_scriptable.py
2022-05-18T05:09:25.4335867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_crine7
2022-05-18T05:09:25.4336731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_crine7/_remote_module_non_scriptable.py
2022-05-18T05:09:25.9854644Z ok (4.732s)
2022-05-18T05:09:25.9854854Z 
2022-05-18T05:09:25.9855237Z ----------------------------------------------------------------------
2022-05-18T05:09:25.9855588Z Ran 1 test in 4.733s
2022-05-18T05:09:25.9855759Z 
2022-05-18T05:09:25.9855855Z OK
2022-05-18T05:09:25.9855990Z 
2022-05-18T05:09:25.9856127Z Generating XML reports...
2022-05-18T05:09:25.9898685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050921.xml
2022-05-18T05:09:27.4023635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:27.4037332Z 
2022-05-18T05:09:27.4037755Z Running tests...
2022-05-18T05:09:27.4038545Z ----------------------------------------------------------------------
2022-05-18T05:09:28.9582820Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:28.9979009Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81715
2022-05-18T05:09:29.0087110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81716
2022-05-18T05:09:30.1939845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:30.1940424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:30.1941231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:30.1941960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:30.1948842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:30.1949354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:31.4594177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6qiu57f
2022-05-18T05:09:31.4594779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6qiu57f/_remote_module_non_scriptable.py
2022-05-18T05:09:31.4741956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnks2ln_p
2022-05-18T05:09:31.4744496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnks2ln_p/_remote_module_non_scriptable.py
2022-05-18T05:09:31.6520636Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:09:31.6562022Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:09:31.6657649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:31.6658476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:32.0165990Z ok (4.612s)
2022-05-18T05:09:32.0166207Z 
2022-05-18T05:09:32.0166587Z ----------------------------------------------------------------------
2022-05-18T05:09:32.0166933Z Ran 1 test in 4.613s
2022-05-18T05:09:32.0167102Z 
2022-05-18T05:09:32.0167207Z OK
2022-05-18T05:09:32.0167349Z 
2022-05-18T05:09:32.0167484Z Generating XML reports...
2022-05-18T05:09:32.0210334Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050927.xml
2022-05-18T05:09:33.4127348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:33.4141024Z 
2022-05-18T05:09:33.4141443Z Running tests...
2022-05-18T05:09:33.4141904Z ----------------------------------------------------------------------
2022-05-18T05:09:34.9508217Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:34.9906593Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81830
2022-05-18T05:09:35.0017775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81831
2022-05-18T05:09:36.1613790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:36.1614336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:36.1615152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:36.1615851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:36.1723881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:36.2628265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:37.3928928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9jkcm6ha
2022-05-18T05:09:37.3929555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9jkcm6ha/_remote_module_non_scriptable.py
2022-05-18T05:09:37.5332715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2fwzotvy
2022-05-18T05:09:37.5334474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2fwzotvy/_remote_module_non_scriptable.py
2022-05-18T05:09:37.5519264Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:09:37.5520680Z   warnings.warn(
2022-05-18T05:09:37.5521967Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:09:37.5522704Z   warnings.warn(
2022-05-18T05:09:38.0096409Z ok (4.595s)
2022-05-18T05:09:38.0096617Z 
2022-05-18T05:09:38.0097012Z ----------------------------------------------------------------------
2022-05-18T05:09:38.0097359Z Ran 1 test in 4.596s
2022-05-18T05:09:38.0097535Z 
2022-05-18T05:09:38.0097633Z OK
2022-05-18T05:09:38.0097770Z 
2022-05-18T05:09:38.0100353Z Generating XML reports...
2022-05-18T05:09:38.0138798Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050933.xml
2022-05-18T05:09:39.4107285Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:39.4121044Z 
2022-05-18T05:09:39.4121380Z Running tests...
2022-05-18T05:09:39.4121838Z ----------------------------------------------------------------------
2022-05-18T05:09:40.9489904Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:40.9637953Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.551s)
2022-05-18T05:09:40.9638835Z 
2022-05-18T05:09:40.9639130Z ----------------------------------------------------------------------
2022-05-18T05:09:40.9639792Z Ran 1 test in 1.552s
2022-05-18T05:09:40.9639970Z 
2022-05-18T05:09:40.9640085Z OK (skipped=1)
2022-05-18T05:09:40.9640250Z 
2022-05-18T05:09:40.9640361Z Generating XML reports...
2022-05-18T05:09:40.9670832Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050939.xml
2022-05-18T05:09:42.3327364Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:42.3342286Z 
2022-05-18T05:09:42.3342450Z Running tests...
2022-05-18T05:09:42.3342885Z ----------------------------------------------------------------------
2022-05-18T05:09:43.9017736Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:43.9421705Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81979
2022-05-18T05:09:43.9532251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81980
2022-05-18T05:09:45.1085758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:45.1086316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:45.1087142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:45.1087825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:45.1193012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:45.2100630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:46.3514614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz6jmq04q
2022-05-18T05:09:46.3515246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz6jmq04q/_remote_module_non_scriptable.py
2022-05-18T05:09:46.4608873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mi0kv55
2022-05-18T05:09:46.4610132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mi0kv55/_remote_module_non_scriptable.py
2022-05-18T05:09:48.1153737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:48.1154348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:09:48.6504445Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:09:48.6871984Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:09:49.9666263Z ok (7.632s)
2022-05-18T05:09:49.9666496Z 
2022-05-18T05:09:49.9667110Z ----------------------------------------------------------------------
2022-05-18T05:09:49.9667461Z Ran 1 test in 7.632s
2022-05-18T05:09:49.9667610Z 
2022-05-18T05:09:49.9667708Z OK
2022-05-18T05:09:49.9667842Z 
2022-05-18T05:09:49.9667979Z Generating XML reports...
2022-05-18T05:09:49.9708399Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050942.xml
2022-05-18T05:09:51.3866925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:51.3880963Z 
2022-05-18T05:09:51.3881120Z Running tests...
2022-05-18T05:09:51.3881575Z ----------------------------------------------------------------------
2022-05-18T05:09:52.9497537Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:52.9902174Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82098
2022-05-18T05:09:53.0011713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82099
2022-05-18T05:09:54.1855843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:09:54.1856407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:09:54.1857214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:54.1857916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:09:54.1963694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:09:54.2870219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:09:55.4290784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_y19138
2022-05-18T05:09:55.4291401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_y19138/_remote_module_non_scriptable.py
2022-05-18T05:09:55.5268963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgag86jof
2022-05-18T05:09:55.5269865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgag86jof/_remote_module_non_scriptable.py
2022-05-18T05:09:55.8088282Z ok (4.420s)
2022-05-18T05:09:55.8088508Z 
2022-05-18T05:09:55.8088903Z ----------------------------------------------------------------------
2022-05-18T05:09:55.8089250Z Ran 1 test in 4.421s
2022-05-18T05:09:55.8089425Z 
2022-05-18T05:09:55.8089525Z OK
2022-05-18T05:09:55.8089910Z 
2022-05-18T05:09:55.8090071Z Generating XML reports...
2022-05-18T05:09:55.8130075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050951.xml
2022-05-18T05:09:57.2169854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:09:57.2183306Z 
2022-05-18T05:09:57.2183644Z Running tests...
2022-05-18T05:09:57.2184091Z ----------------------------------------------------------------------
2022-05-18T05:09:58.7708097Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:09:58.8104025Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82209
2022-05-18T05:09:58.8209881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82210
2022-05-18T05:10:00.0389752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:00.0390325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:00.0391130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:00.0391829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:00.0398424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:00.0399186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:01.3210909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdy1cqcp7
2022-05-18T05:10:01.3211887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdy1cqcp7/_remote_module_non_scriptable.py
2022-05-18T05:10:01.3237016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdactqptd
2022-05-18T05:10:01.3239961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdactqptd/_remote_module_non_scriptable.py
2022-05-18T05:10:01.3398816Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:10:01.3399591Z   warnings.warn(
2022-05-18T05:10:01.3400642Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:10:01.3401382Z   warnings.warn(
2022-05-18T05:10:01.5102083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:01.5102622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:01.5170377Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:10:01.5172295Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:10:01.5307081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:01.5307844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:01.5388610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:01.5389403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:01.8286699Z ok (4.610s)
2022-05-18T05:10:01.8286895Z 
2022-05-18T05:10:01.8287309Z ----------------------------------------------------------------------
2022-05-18T05:10:01.8287666Z Ran 1 test in 4.610s
2022-05-18T05:10:01.8287822Z 
2022-05-18T05:10:01.8287922Z OK
2022-05-18T05:10:01.8288061Z 
2022-05-18T05:10:01.8288192Z Generating XML reports...
2022-05-18T05:10:01.8329329Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050957.xml
2022-05-18T05:10:03.2751630Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:03.2766391Z 
2022-05-18T05:10:03.2766714Z Running tests...
2022-05-18T05:10:03.2767316Z ----------------------------------------------------------------------
2022-05-18T05:10:04.8540009Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:04.8938692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82324
2022-05-18T05:10:04.9049609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82325
2022-05-18T05:10:06.0959682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:06.0960244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:06.0961019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:06.0961728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:06.0969444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:06.0970477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:07.3936106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4x7ry3sq
2022-05-18T05:10:07.3937357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4x7ry3sq/_remote_module_non_scriptable.py
2022-05-18T05:10:07.4205995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hq39p_0
2022-05-18T05:10:07.4208915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hq39p_0/_remote_module_non_scriptable.py
2022-05-18T05:10:07.4373902Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:10:07.4374793Z   warnings.warn(
2022-05-18T05:10:07.4376071Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T05:10:07.4376813Z   warnings.warn(
2022-05-18T05:10:07.6115428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:07.6115989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:07.9129367Z ok (4.636s)
2022-05-18T05:10:07.9129582Z 
2022-05-18T05:10:07.9129973Z ----------------------------------------------------------------------
2022-05-18T05:10:07.9130298Z Ran 1 test in 4.636s
2022-05-18T05:10:07.9130466Z 
2022-05-18T05:10:07.9130585Z OK
2022-05-18T05:10:07.9130720Z 
2022-05-18T05:10:07.9130858Z Generating XML reports...
2022-05-18T05:10:07.9172077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051003.xml
2022-05-18T05:10:09.3361873Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:09.3377032Z 
2022-05-18T05:10:09.3377472Z Running tests...
2022-05-18T05:10:09.3377966Z ----------------------------------------------------------------------
2022-05-18T05:10:10.9353992Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:10.9509702Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.613s)
2022-05-18T05:10:10.9510667Z 
2022-05-18T05:10:10.9510971Z ----------------------------------------------------------------------
2022-05-18T05:10:10.9511321Z Ran 1 test in 1.613s
2022-05-18T05:10:10.9511493Z 
2022-05-18T05:10:10.9511611Z OK (skipped=1)
2022-05-18T05:10:10.9511778Z 
2022-05-18T05:10:10.9511890Z Generating XML reports...
2022-05-18T05:10:10.9544845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051009.xml
2022-05-18T05:10:12.3115744Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:12.3130294Z 
2022-05-18T05:10:12.3130718Z Running tests...
2022-05-18T05:10:12.3131222Z ----------------------------------------------------------------------
2022-05-18T05:10:13.8567253Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:13.8961504Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82473
2022-05-18T05:10:13.9068013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82474
2022-05-18T05:10:15.1325787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:15.1326410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:15.1327210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:15.1327913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:15.1436117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:15.2339946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:16.4133458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpte2ce8py
2022-05-18T05:10:16.4134106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpte2ce8py/_remote_module_non_scriptable.py
2022-05-18T05:10:16.4914392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5v23ryl
2022-05-18T05:10:16.4915906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5v23ryl/_remote_module_non_scriptable.py
2022-05-18T05:10:16.7979450Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:16.7979992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:17.9182837Z ok (5.605s)
2022-05-18T05:10:17.9183069Z 
2022-05-18T05:10:17.9183458Z ----------------------------------------------------------------------
2022-05-18T05:10:17.9183784Z Ran 1 test in 5.605s
2022-05-18T05:10:17.9183949Z 
2022-05-18T05:10:17.9184045Z OK
2022-05-18T05:10:17.9184180Z 
2022-05-18T05:10:17.9184315Z Generating XML reports...
2022-05-18T05:10:17.9226055Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051012.xml
2022-05-18T05:10:19.3551051Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:19.3565728Z 
2022-05-18T05:10:19.3566166Z Running tests...
2022-05-18T05:10:19.3566672Z ----------------------------------------------------------------------
2022-05-18T05:10:20.9377470Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:20.9781419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82592
2022-05-18T05:10:20.9891182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82593
2022-05-18T05:10:22.1639584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:22.1640647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:22.1642172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:22.1642874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:22.1747050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:22.2655017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:23.3968319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvu4f2y9m
2022-05-18T05:10:23.3969512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvu4f2y9m/_remote_module_non_scriptable.py
2022-05-18T05:10:23.5168017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoolkq3su
2022-05-18T05:10:23.5169164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoolkq3su/_remote_module_non_scriptable.py
2022-05-18T05:10:23.7968960Z ok (4.440s)
2022-05-18T05:10:23.7969184Z 
2022-05-18T05:10:23.7969592Z ----------------------------------------------------------------------
2022-05-18T05:10:23.7969933Z Ran 1 test in 4.440s
2022-05-18T05:10:23.7970102Z 
2022-05-18T05:10:23.7970186Z OK
2022-05-18T05:10:23.7970349Z 
2022-05-18T05:10:23.7970485Z Generating XML reports...
2022-05-18T05:10:23.8009895Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051019.xml
2022-05-18T05:10:25.1839535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:25.1854579Z 
2022-05-18T05:10:25.1854813Z Running tests...
2022-05-18T05:10:25.1855471Z ----------------------------------------------------------------------
2022-05-18T05:10:26.7755597Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:26.8161891Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82703
2022-05-18T05:10:26.8272669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82704
2022-05-18T05:10:27.9840884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:27.9841716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:27.9842524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:27.9843230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:27.9849357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:27.9850154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:29.2669758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4j_5xri
2022-05-18T05:10:29.2670634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4j_5xri/_remote_module_non_scriptable.py
2022-05-18T05:10:29.2809614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppb2co3mk
2022-05-18T05:10:29.2812190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppb2co3mk/_remote_module_non_scriptable.py
2022-05-18T05:10:29.5346035Z ok (4.349s)
2022-05-18T05:10:29.5346272Z 
2022-05-18T05:10:29.5346673Z ----------------------------------------------------------------------
2022-05-18T05:10:29.5347020Z Ran 1 test in 4.349s
2022-05-18T05:10:29.5347186Z 
2022-05-18T05:10:29.5347270Z OK
2022-05-18T05:10:29.5347405Z 
2022-05-18T05:10:29.5347537Z Generating XML reports...
2022-05-18T05:10:29.5388373Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051025.xml
2022-05-18T05:10:30.9617524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:30.9632019Z 
2022-05-18T05:10:30.9632291Z Running tests...
2022-05-18T05:10:30.9632746Z ----------------------------------------------------------------------
2022-05-18T05:10:32.5463684Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:32.5867199Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82814
2022-05-18T05:10:32.5977004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82815
2022-05-18T05:10:33.7913484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:33.7914063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:33.7914859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:33.7915590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:33.7933824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:33.7934401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:35.0582620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw2ybiuuv
2022-05-18T05:10:35.0583256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw2ybiuuv/_remote_module_non_scriptable.py
2022-05-18T05:10:35.0773671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_63admr1
2022-05-18T05:10:35.0776253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_63admr1/_remote_module_non_scriptable.py
2022-05-18T05:10:35.2654300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:35.2654877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:35.6058362Z ok (4.642s)
2022-05-18T05:10:35.6058734Z 
2022-05-18T05:10:35.6059149Z ----------------------------------------------------------------------
2022-05-18T05:10:35.6059840Z Ran 1 test in 4.643s
2022-05-18T05:10:35.6060181Z 
2022-05-18T05:10:35.6060307Z OK
2022-05-18T05:10:35.6060444Z 
2022-05-18T05:10:35.6060562Z Generating XML reports...
2022-05-18T05:10:35.6101358Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051030.xml
2022-05-18T05:10:37.0299753Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:37.0314130Z 
2022-05-18T05:10:37.0314607Z Running tests...
2022-05-18T05:10:37.0315120Z ----------------------------------------------------------------------
2022-05-18T05:10:38.6268494Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:38.6423405Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.611s)
2022-05-18T05:10:38.6424076Z 
2022-05-18T05:10:38.6424361Z ----------------------------------------------------------------------
2022-05-18T05:10:38.6424680Z Ran 1 test in 1.611s
2022-05-18T05:10:38.6424847Z 
2022-05-18T05:10:38.6424957Z OK (skipped=1)
2022-05-18T05:10:38.6425114Z 
2022-05-18T05:10:38.6425241Z Generating XML reports...
2022-05-18T05:10:38.6457801Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051037.xml
2022-05-18T05:10:40.0268695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:40.0284017Z 
2022-05-18T05:10:40.0284490Z Running tests...
2022-05-18T05:10:40.0285493Z ----------------------------------------------------------------------
2022-05-18T05:10:41.5966206Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:41.6372700Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82963
2022-05-18T05:10:41.6481749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82964
2022-05-18T05:10:42.8420386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:42.8421443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:42.8422738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:42.8423920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:42.8429325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:42.8430605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:44.1246120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6voq3b1p
2022-05-18T05:10:44.1247291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6voq3b1p/_remote_module_non_scriptable.py
2022-05-18T05:10:44.1422994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6uihkrjc
2022-05-18T05:10:44.1425321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6uihkrjc/_remote_module_non_scriptable.py
2022-05-18T05:10:44.3335565Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:44.3336557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:44.3681144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:44.3681841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:10:44.3846315Z /opt/conda/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp6voq3b1p'>
2022-05-18T05:10:44.3847382Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T05:10:44.6574665Z ok (4.629s)
2022-05-18T05:10:44.6574914Z 
2022-05-18T05:10:44.6575311Z ----------------------------------------------------------------------
2022-05-18T05:10:44.6575653Z Ran 1 test in 4.629s
2022-05-18T05:10:44.6575820Z 
2022-05-18T05:10:44.6575918Z OK
2022-05-18T05:10:44.6576053Z 
2022-05-18T05:10:44.6577131Z Generating XML reports...
2022-05-18T05:10:44.6616885Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051040.xml
2022-05-18T05:10:46.0741416Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:46.0756357Z 
2022-05-18T05:10:46.0756655Z Running tests...
2022-05-18T05:10:46.0757103Z ----------------------------------------------------------------------
2022-05-18T05:10:47.6386492Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:47.6792093Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83078
2022-05-18T05:10:47.6902485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83079
2022-05-18T05:10:48.9132450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:48.9133051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:48.9133845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:48.9134809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:48.9240883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:49.0147195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:50.1936006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zg_zf69
2022-05-18T05:10:50.1936607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zg_zf69/_remote_module_non_scriptable.py
2022-05-18T05:10:50.2598114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rz5shm7
2022-05-18T05:10:50.2599399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rz5shm7/_remote_module_non_scriptable.py
2022-05-18T05:10:50.7983721Z ok (4.722s)
2022-05-18T05:10:50.7984209Z 
2022-05-18T05:10:50.7984953Z ----------------------------------------------------------------------
2022-05-18T05:10:50.7985350Z Ran 1 test in 4.723s
2022-05-18T05:10:50.7985518Z 
2022-05-18T05:10:50.7985596Z OK
2022-05-18T05:10:50.7985735Z 
2022-05-18T05:10:50.7985879Z Generating XML reports...
2022-05-18T05:10:50.8026465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051046.xml
2022-05-18T05:10:52.2266950Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:52.2281679Z 
2022-05-18T05:10:52.2281958Z Running tests...
2022-05-18T05:10:52.2282399Z ----------------------------------------------------------------------
2022-05-18T05:10:53.8084280Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:53.8489073Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83193
2022-05-18T05:10:53.8599525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83194
2022-05-18T05:10:55.0333654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:55.0334211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:55.0335235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:55.0335950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:55.0341443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:55.0342407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:55.0450540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:10:55.0451084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:10:55.0451821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:10:55.0452522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:10:55.2650463Z ok (3.036s)
2022-05-18T05:10:55.2650673Z 
2022-05-18T05:10:55.2651051Z ----------------------------------------------------------------------
2022-05-18T05:10:55.2651392Z Ran 1 test in 3.037s
2022-05-18T05:10:55.2651556Z 
2022-05-18T05:10:55.2651661Z OK
2022-05-18T05:10:55.2651798Z 
2022-05-18T05:10:55.2651926Z Generating XML reports...
2022-05-18T05:10:55.2692508Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051052.xml
2022-05-18T05:10:56.6629649Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:10:56.6644110Z 
2022-05-18T05:10:56.6644483Z Running tests...
2022-05-18T05:10:56.6644921Z ----------------------------------------------------------------------
2022-05-18T05:10:58.2494033Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:10:58.2897657Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83308
2022-05-18T05:10:58.3007181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83309
2022-05-18T05:10:59.4971650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:10:59.4972232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:10:59.4973021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:59.4973722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:10:59.5079633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:10:59.5984504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:10:59.6192160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:10:59.6192690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:10:59.6193379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:10:59.6194080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:10:59.9062538Z ok (3.241s)
2022-05-18T05:10:59.9062738Z 
2022-05-18T05:10:59.9063761Z ----------------------------------------------------------------------
2022-05-18T05:10:59.9064163Z Ran 1 test in 3.242s
2022-05-18T05:10:59.9064340Z 
2022-05-18T05:10:59.9064420Z OK
2022-05-18T05:10:59.9064560Z 
2022-05-18T05:10:59.9064696Z Generating XML reports...
2022-05-18T05:10:59.9105573Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051056.xml
2022-05-18T05:11:01.3288745Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:01.3303073Z 
2022-05-18T05:11:01.3303540Z Running tests...
2022-05-18T05:11:01.3304032Z ----------------------------------------------------------------------
2022-05-18T05:11:02.9151899Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:02.9556653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83423
2022-05-18T05:11:02.9667024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83424
2022-05-18T05:11:04.1912491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:04.1913057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:04.1913867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:04.1914546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:04.2021715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:04.2926654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:05.4903532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbi1plqm
2022-05-18T05:11:05.4904137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbi1plqm/_remote_module_non_scriptable.py
2022-05-18T05:11:05.5505368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_evy4a14
2022-05-18T05:11:05.5506524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_evy4a14/_remote_module_non_scriptable.py
2022-05-18T05:11:05.7479303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:05.7479910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:05.7571592Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:11:06.0749421Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:11:06.0750381Z ok (4.744s)
2022-05-18T05:11:06.0750537Z 
2022-05-18T05:11:06.0750945Z ----------------------------------------------------------------------
2022-05-18T05:11:06.0751267Z Ran 1 test in 4.744s
2022-05-18T05:11:06.0751431Z 
2022-05-18T05:11:06.0751526Z OK
2022-05-18T05:11:06.0751662Z 
2022-05-18T05:11:06.0751788Z Generating XML reports...
2022-05-18T05:11:06.0790522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051101.xml
2022-05-18T05:11:07.4958813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:07.4973078Z 
2022-05-18T05:11:07.4973713Z Running tests...
2022-05-18T05:11:07.4974658Z ----------------------------------------------------------------------
2022-05-18T05:11:09.0677865Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:09.1085965Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83538
2022-05-18T05:11:09.1196932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83539
2022-05-18T05:11:10.2881328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:10.2881895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:10.2882681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:10.2883388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:10.2889240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:10.2889732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:11.5679847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1ppt1gs
2022-05-18T05:11:11.5680509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1ppt1gs/_remote_module_non_scriptable.py
2022-05-18T05:11:11.5821039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp86wa1i_u
2022-05-18T05:11:11.5823605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp86wa1i_u/_remote_module_non_scriptable.py
2022-05-18T05:11:11.7641611Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:11:11.7845183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:11.7845725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:11:12.1276801Z ok (4.630s)
2022-05-18T05:11:12.1289592Z 
2022-05-18T05:11:12.1290021Z ----------------------------------------------------------------------
2022-05-18T05:11:12.1290403Z Ran 1 test in 4.630s
2022-05-18T05:11:12.1290571Z 
2022-05-18T05:11:12.1290648Z OK
2022-05-18T05:11:12.1290786Z 
2022-05-18T05:11:12.1290926Z Generating XML reports...
2022-05-18T05:11:12.1318732Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051107.xml
2022-05-18T05:11:13.5218553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:13.5231613Z 
2022-05-18T05:11:13.5232029Z Running tests...
2022-05-18T05:11:13.5232996Z ----------------------------------------------------------------------
2022-05-18T05:11:15.0731267Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:15.1124823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83653
2022-05-18T05:11:15.1231789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83654
2022-05-18T05:11:16.3013872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:16.3014418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:16.3015431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:16.3016201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:16.3021616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:16.3022088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:16.5284583Z ok (3.005s)
2022-05-18T05:11:16.5284821Z 
2022-05-18T05:11:16.5285228Z ----------------------------------------------------------------------
2022-05-18T05:11:16.5285557Z Ran 1 test in 3.005s
2022-05-18T05:11:16.5285747Z 
2022-05-18T05:11:16.5285844Z OK
2022-05-18T05:11:16.5285981Z 
2022-05-18T05:11:16.5286117Z Generating XML reports...
2022-05-18T05:11:16.5325480Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051113.xml
2022-05-18T05:11:17.9293054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:17.9307444Z 
2022-05-18T05:11:17.9307615Z Running tests...
2022-05-18T05:11:17.9308617Z ----------------------------------------------------------------------
2022-05-18T05:11:19.5035441Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:19.5441376Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83762
2022-05-18T05:11:19.5552176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83763
2022-05-18T05:11:20.7545208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:20.7546227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:20.7547018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:20.7547727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:20.7652882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:20.8556989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:21.0603764Z ok (3.129s)
2022-05-18T05:11:21.0604090Z 
2022-05-18T05:11:21.0604641Z ----------------------------------------------------------------------
2022-05-18T05:11:21.0604973Z Ran 1 test in 3.130s
2022-05-18T05:11:21.0605140Z 
2022-05-18T05:11:21.0605236Z OK
2022-05-18T05:11:21.0605394Z 
2022-05-18T05:11:21.0605533Z Generating XML reports...
2022-05-18T05:11:21.0646076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051117.xml
2022-05-18T05:11:22.4690899Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:22.4705525Z 
2022-05-18T05:11:22.4705956Z Running tests...
2022-05-18T05:11:22.4706459Z ----------------------------------------------------------------------
2022-05-18T05:11:24.0376571Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:24.0771762Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83875
2022-05-18T05:11:24.0880235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83876
2022-05-18T05:11:25.2769170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:25.2769735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:25.2770538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:25.2771458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:25.2878956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:25.3779189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:25.5932528Z ok (3.122s)
2022-05-18T05:11:25.5932738Z 
2022-05-18T05:11:25.5933134Z ----------------------------------------------------------------------
2022-05-18T05:11:25.5933469Z Ran 1 test in 3.123s
2022-05-18T05:11:25.5933634Z 
2022-05-18T05:11:25.5933731Z OK
2022-05-18T05:11:25.5933871Z 
2022-05-18T05:11:25.5934012Z Generating XML reports...
2022-05-18T05:11:25.5974671Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051122.xml
2022-05-18T05:11:27.0028528Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:27.0043705Z 
2022-05-18T05:11:27.0044059Z Running tests...
2022-05-18T05:11:27.0044531Z ----------------------------------------------------------------------
2022-05-18T05:11:27.0068796Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T05:11:27.0069103Z 
2022-05-18T05:11:27.0069391Z ----------------------------------------------------------------------
2022-05-18T05:11:27.0069704Z Ran 1 test in 0.003s
2022-05-18T05:11:27.0069869Z 
2022-05-18T05:11:27.0069984Z OK (skipped=1)
2022-05-18T05:11:27.0070140Z 
2022-05-18T05:11:27.0070268Z Generating XML reports...
2022-05-18T05:11:27.0113788Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051126.xml
2022-05-18T05:11:28.2448746Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:28.2461986Z 
2022-05-18T05:11:28.2462198Z Running tests...
2022-05-18T05:11:28.2462640Z ----------------------------------------------------------------------
2022-05-18T05:11:29.7776916Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:29.8167471Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84017
2022-05-18T05:11:29.8277809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84018
2022-05-18T05:11:31.0284338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:31.0284903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:31.0285712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:31.0286421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:31.0293062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:31.0293889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:31.0400864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:11:31.0401397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:11:31.0402115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:11:31.0402812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:11:31.3329082Z ok (3.086s)
2022-05-18T05:11:31.3329654Z 
2022-05-18T05:11:31.3330085Z ----------------------------------------------------------------------
2022-05-18T05:11:31.3330433Z Ran 1 test in 3.087s
2022-05-18T05:11:31.3330589Z 
2022-05-18T05:11:31.3330690Z OK
2022-05-18T05:11:31.3330829Z 
2022-05-18T05:11:31.3331157Z Generating XML reports...
2022-05-18T05:11:31.3372153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051128.xml
2022-05-18T05:11:32.7312190Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:32.7327487Z 
2022-05-18T05:11:32.7327902Z Running tests...
2022-05-18T05:11:32.7328408Z ----------------------------------------------------------------------
2022-05-18T05:11:34.3175066Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:34.3580222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84136
2022-05-18T05:11:34.3692680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84137
2022-05-18T05:11:35.5319389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:35.5320310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:35.5321115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:35.5321807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:35.5426509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:35.6333479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:35.7744541Z skip: Skipped due to small world size. (3.041s)
2022-05-18T05:11:35.7745417Z 
2022-05-18T05:11:35.7746136Z ----------------------------------------------------------------------
2022-05-18T05:11:35.7746586Z Ran 1 test in 3.042s
2022-05-18T05:11:35.7746754Z 
2022-05-18T05:11:35.7746868Z OK (skipped=1)
2022-05-18T05:11:35.7747033Z 
2022-05-18T05:11:35.7747163Z Generating XML reports...
2022-05-18T05:11:35.7787453Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051132.xml
2022-05-18T05:11:37.1801279Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:37.1816209Z 
2022-05-18T05:11:37.1816462Z Running tests...
2022-05-18T05:11:37.1816913Z ----------------------------------------------------------------------
2022-05-18T05:11:38.7542688Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:38.7945571Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84245
2022-05-18T05:11:38.8055577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84246
2022-05-18T05:11:40.0291018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:40.0291615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:40.0292404Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:40.0293110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:40.0400885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:40.1303056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:40.3107247Z ok (3.129s)
2022-05-18T05:11:40.3107591Z 
2022-05-18T05:11:40.3108158Z ----------------------------------------------------------------------
2022-05-18T05:11:40.3108618Z Ran 1 test in 3.129s
2022-05-18T05:11:40.3108789Z 
2022-05-18T05:11:40.3108886Z OK
2022-05-18T05:11:40.3109024Z 
2022-05-18T05:11:40.3109140Z Generating XML reports...
2022-05-18T05:11:40.3157441Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051137.xml
2022-05-18T05:11:41.7003124Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:41.7016489Z 
2022-05-18T05:11:41.7016754Z Running tests...
2022-05-18T05:11:41.7017185Z ----------------------------------------------------------------------
2022-05-18T05:11:43.2388366Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:43.2783916Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84354
2022-05-18T05:11:43.2894954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84355
2022-05-18T05:11:44.4700706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:44.4701248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:44.4702074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:44.4702777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:44.4808146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:44.5712466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:44.6031696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:11:44.6133925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:11:44.6134956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:11:44.6135701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:11:44.6482717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:11:44.6483243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:11:44.6483945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:11:44.6484640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:11:44.6705247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T05:11:44.6705782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T05:11:44.6706475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T05:11:44.6707173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T05:11:44.8948404Z ok (3.193s)
2022-05-18T05:11:44.8948669Z 
2022-05-18T05:11:44.8949057Z ----------------------------------------------------------------------
2022-05-18T05:11:44.8949379Z Ran 1 test in 3.193s
2022-05-18T05:11:44.8949544Z 
2022-05-18T05:11:44.8949640Z OK
2022-05-18T05:11:44.8949782Z 
2022-05-18T05:11:44.8949917Z Generating XML reports...
2022-05-18T05:11:44.8990873Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051141.xml
2022-05-18T05:11:46.3145477Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:46.3159983Z 
2022-05-18T05:11:46.3160469Z Running tests...
2022-05-18T05:11:46.3160983Z ----------------------------------------------------------------------
2022-05-18T05:11:47.8793291Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:47.9200246Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84487
2022-05-18T05:11:47.9309971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84488
2022-05-18T05:11:49.1227129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:49.1227677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:49.1228467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:49.1229184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:49.1235768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:49.1236273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:49.1342625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:11:49.1343143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:11:49.1343834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:11:49.1344533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:11:49.3359970Z ok (3.020s)
2022-05-18T05:11:49.3360439Z 
2022-05-18T05:11:49.3360836Z ----------------------------------------------------------------------
2022-05-18T05:11:49.3361200Z Ran 1 test in 3.020s
2022-05-18T05:11:49.3361376Z 
2022-05-18T05:11:49.3361472Z OK
2022-05-18T05:11:49.3361614Z 
2022-05-18T05:11:49.3361751Z Generating XML reports...
2022-05-18T05:11:49.3409997Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051146.xml
2022-05-18T05:11:50.7414437Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:50.7428699Z 
2022-05-18T05:11:50.7429138Z Running tests...
2022-05-18T05:11:50.7429615Z ----------------------------------------------------------------------
2022-05-18T05:11:52.2926203Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:52.3330083Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84602
2022-05-18T05:11:52.3439779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84603
2022-05-18T05:11:53.5551302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:53.5551864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:53.5552663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:53.5553364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:53.5559780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:53.5560272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:53.7490446Z ok (3.006s)
2022-05-18T05:11:53.7490670Z 
2022-05-18T05:11:53.7491069Z ----------------------------------------------------------------------
2022-05-18T05:11:53.7491419Z Ran 1 test in 3.006s
2022-05-18T05:11:53.7491586Z 
2022-05-18T05:11:53.7491686Z OK
2022-05-18T05:11:53.7491825Z 
2022-05-18T05:11:53.7491961Z Generating XML reports...
2022-05-18T05:11:53.7532323Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051150.xml
2022-05-18T05:11:55.1182253Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:55.1195963Z 
2022-05-18T05:11:55.1196457Z Running tests...
2022-05-18T05:11:55.1196984Z ----------------------------------------------------------------------
2022-05-18T05:11:56.6983687Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:11:56.7389798Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84711
2022-05-18T05:11:56.7501172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84712
2022-05-18T05:11:57.9019264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:11:57.9020330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:11:57.9021584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:57.9022982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:11:57.9027470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:11:57.9028851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:11:58.0548736Z ok (2.935s)
2022-05-18T05:11:58.0548951Z 
2022-05-18T05:11:58.0549344Z ----------------------------------------------------------------------
2022-05-18T05:11:58.0549738Z Ran 1 test in 2.935s
2022-05-18T05:11:58.0550167Z 
2022-05-18T05:11:58.0550266Z OK
2022-05-18T05:11:58.0550401Z 
2022-05-18T05:11:58.0550519Z Generating XML reports...
2022-05-18T05:11:58.0591914Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051155.xml
2022-05-18T05:11:59.4618290Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:11:59.4632170Z 
2022-05-18T05:11:59.4632669Z Running tests...
2022-05-18T05:11:59.4633319Z ----------------------------------------------------------------------
2022-05-18T05:12:01.0311137Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:01.0714396Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84820
2022-05-18T05:12:01.0825079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84821
2022-05-18T05:12:02.2491318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:02.2491917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:02.2492693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:02.2493410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:02.2601111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:02.3502145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:02.3710343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:12:02.3711376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:12:02.3712723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:12:02.3713969Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:12:02.5877236Z ok (3.124s)
2022-05-18T05:12:02.5877440Z 
2022-05-18T05:12:02.5878437Z ----------------------------------------------------------------------
2022-05-18T05:12:02.5878964Z Ran 1 test in 3.124s
2022-05-18T05:12:02.5879131Z 
2022-05-18T05:12:02.5879234Z OK
2022-05-18T05:12:02.5879371Z 
2022-05-18T05:12:02.5879489Z Generating XML reports...
2022-05-18T05:12:02.5918762Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051159.xml
2022-05-18T05:12:04.0044244Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:04.0058449Z 
2022-05-18T05:12:04.0058904Z Running tests...
2022-05-18T05:12:04.0059393Z ----------------------------------------------------------------------
2022-05-18T05:12:05.5780431Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:05.6184759Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84935
2022-05-18T05:12:05.6295883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84936
2022-05-18T05:12:06.7674612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:06.7675163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:06.7675948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:06.7676643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:06.7682638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:06.7683357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:06.7890595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:12:06.7891123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:12:06.7891819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:12:06.7892495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:12:07.0347002Z ok (3.028s)
2022-05-18T05:12:07.0347252Z 
2022-05-18T05:12:07.0347658Z ----------------------------------------------------------------------
2022-05-18T05:12:07.0348005Z Ran 1 test in 3.029s
2022-05-18T05:12:07.0348173Z 
2022-05-18T05:12:07.0348290Z OK
2022-05-18T05:12:07.0348415Z 
2022-05-18T05:12:07.0348551Z Generating XML reports...
2022-05-18T05:12:07.0389898Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051204.xml
2022-05-18T05:12:08.4336111Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:08.4350480Z 
2022-05-18T05:12:08.4350918Z Running tests...
2022-05-18T05:12:08.4351400Z ----------------------------------------------------------------------
2022-05-18T05:12:10.0056001Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:10.0450269Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85050
2022-05-18T05:12:10.0559129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85051
2022-05-18T05:12:11.2374628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:11.2375243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:11.2376030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:11.2376931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:11.2381897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:11.2382396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:12.5155978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeahnnhc4
2022-05-18T05:12:12.5156623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeahnnhc4/_remote_module_non_scriptable.py
2022-05-18T05:12:12.5441276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5balxfe
2022-05-18T05:12:12.5443953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5balxfe/_remote_module_non_scriptable.py
2022-05-18T05:12:13.0637669Z ok (4.628s)
2022-05-18T05:12:13.0637903Z 
2022-05-18T05:12:13.0638684Z ----------------------------------------------------------------------
2022-05-18T05:12:13.0639055Z Ran 1 test in 4.629s
2022-05-18T05:12:13.0639238Z 
2022-05-18T05:12:13.0639383Z OK
2022-05-18T05:12:13.0639638Z 
2022-05-18T05:12:13.0639833Z Generating XML reports...
2022-05-18T05:12:13.0679649Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051208.xml
2022-05-18T05:12:14.4623402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:14.4636753Z 
2022-05-18T05:12:14.4637253Z Running tests...
2022-05-18T05:12:14.4637736Z ----------------------------------------------------------------------
2022-05-18T05:12:16.0002944Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:16.0396137Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85165
2022-05-18T05:12:16.0506839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85166
2022-05-18T05:12:17.2233671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:17.2234236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:17.2235024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:17.2235721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:17.2242360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:17.2243275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:17.4556316Z ok (2.992s)
2022-05-18T05:12:17.4556662Z 
2022-05-18T05:12:17.4557335Z ----------------------------------------------------------------------
2022-05-18T05:12:17.4557955Z Ran 1 test in 2.992s
2022-05-18T05:12:17.4558598Z 
2022-05-18T05:12:17.4558793Z OK
2022-05-18T05:12:17.4559054Z 
2022-05-18T05:12:17.4559283Z Generating XML reports...
2022-05-18T05:12:17.4600923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051214.xml
2022-05-18T05:12:18.8565424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:18.8579183Z 
2022-05-18T05:12:18.8579566Z Running tests...
2022-05-18T05:12:18.8580169Z ----------------------------------------------------------------------
2022-05-18T05:12:20.4265572Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:20.4670412Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85274
2022-05-18T05:12:20.4780588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85275
2022-05-18T05:12:21.6215113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:21.6215902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:21.6216751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:21.6217479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:21.6323200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:21.7227669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:21.8830129Z ok (3.025s)
2022-05-18T05:12:21.8830352Z 
2022-05-18T05:12:21.8830721Z ----------------------------------------------------------------------
2022-05-18T05:12:21.8831061Z Ran 1 test in 3.025s
2022-05-18T05:12:21.8831228Z 
2022-05-18T05:12:21.8831328Z OK
2022-05-18T05:12:21.8831464Z 
2022-05-18T05:12:21.8831618Z Generating XML reports...
2022-05-18T05:12:21.8872217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051218.xml
2022-05-18T05:12:23.2804112Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:23.2818234Z 
2022-05-18T05:12:23.2818653Z Running tests...
2022-05-18T05:12:23.2819373Z ----------------------------------------------------------------------
2022-05-18T05:12:24.8481295Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:24.8874439Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85383
2022-05-18T05:12:24.8984174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85384
2022-05-18T05:12:26.0927006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:26.0927591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:26.0928407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:26.0929108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:26.1037089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:26.1937344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:26.5038189Z ok (3.222s)
2022-05-18T05:12:26.5038727Z 
2022-05-18T05:12:26.5039194Z ----------------------------------------------------------------------
2022-05-18T05:12:26.5039560Z Ran 1 test in 3.222s
2022-05-18T05:12:26.5039727Z 
2022-05-18T05:12:26.5039827Z OK
2022-05-18T05:12:26.5039946Z 
2022-05-18T05:12:26.5040081Z Generating XML reports...
2022-05-18T05:12:26.5079589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051223.xml
2022-05-18T05:12:27.9320316Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:27.9334949Z 
2022-05-18T05:12:27.9335268Z Running tests...
2022-05-18T05:12:27.9335717Z ----------------------------------------------------------------------
2022-05-18T05:12:29.5104490Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:29.5500646Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85496
2022-05-18T05:12:29.5610140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85497
2022-05-18T05:12:30.7605375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:30.7605923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:30.7606935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:30.7607664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:30.7713715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:30.8617743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:31.1663258Z ok (3.232s)
2022-05-18T05:12:31.1663474Z 
2022-05-18T05:12:31.1663855Z ----------------------------------------------------------------------
2022-05-18T05:12:31.1664233Z Ran 1 test in 3.233s
2022-05-18T05:12:31.1664404Z 
2022-05-18T05:12:31.1664499Z OK
2022-05-18T05:12:31.1664636Z 
2022-05-18T05:12:31.1664770Z Generating XML reports...
2022-05-18T05:12:31.1705906Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051227.xml
2022-05-18T05:12:32.5879364Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:32.5893744Z 
2022-05-18T05:12:32.5894078Z Running tests...
2022-05-18T05:12:32.5894516Z ----------------------------------------------------------------------
2022-05-18T05:12:32.5914161Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T05:12:32.5914545Z 
2022-05-18T05:12:32.5914840Z ----------------------------------------------------------------------
2022-05-18T05:12:32.5915177Z Ran 1 test in 0.002s
2022-05-18T05:12:32.5915651Z 
2022-05-18T05:12:32.5915766Z OK (skipped=1)
2022-05-18T05:12:32.5915923Z 
2022-05-18T05:12:32.5916048Z Generating XML reports...
2022-05-18T05:12:32.5951187Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051232.xml
2022-05-18T05:12:33.8529880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:33.8548779Z 
2022-05-18T05:12:33.8549090Z Running tests...
2022-05-18T05:12:33.8549569Z ----------------------------------------------------------------------
2022-05-18T05:12:33.8568478Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-05-18T05:12:33.8568886Z 
2022-05-18T05:12:33.8569217Z ----------------------------------------------------------------------
2022-05-18T05:12:33.8569553Z Ran 1 test in 0.002s
2022-05-18T05:12:33.8569717Z 
2022-05-18T05:12:33.8569828Z OK (skipped=1)
2022-05-18T05:12:33.8569983Z 
2022-05-18T05:12:33.8570111Z Generating XML reports...
2022-05-18T05:12:33.8604908Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051233.xml
2022-05-18T05:12:35.1196518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:35.1211015Z 
2022-05-18T05:12:35.1211309Z Running tests...
2022-05-18T05:12:35.1211747Z ----------------------------------------------------------------------
2022-05-18T05:12:36.6970791Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:36.7374341Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85675
2022-05-18T05:12:36.7484261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85676
2022-05-18T05:12:37.9256782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:37.9257358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:37.9258176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:37.9259063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:37.9264874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:37.9265366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:38.1536137Z skip: Skipped due to small world size. (3.032s)
2022-05-18T05:12:38.1536401Z 
2022-05-18T05:12:38.1536953Z ----------------------------------------------------------------------
2022-05-18T05:12:38.1537457Z Ran 1 test in 3.032s
2022-05-18T05:12:38.1537626Z 
2022-05-18T05:12:38.1537761Z OK (skipped=1)
2022-05-18T05:12:38.1537925Z 
2022-05-18T05:12:38.1538053Z Generating XML reports...
2022-05-18T05:12:38.1579379Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051235.xml
2022-05-18T05:12:39.5666529Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:39.5681207Z 
2022-05-18T05:12:39.5681680Z Running tests...
2022-05-18T05:12:39.5682182Z ----------------------------------------------------------------------
2022-05-18T05:12:41.1442827Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:41.1849478Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85784
2022-05-18T05:12:41.1959416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85785
2022-05-18T05:12:42.3548918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:42.3549751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:42.3550556Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:42.3551266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:42.3656616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:42.4561125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:44.4722070Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T05:12:44.8051606Z ok (5.237s)
2022-05-18T05:12:44.8051923Z 
2022-05-18T05:12:44.8052327Z ----------------------------------------------------------------------
2022-05-18T05:12:44.8052673Z Ran 1 test in 5.237s
2022-05-18T05:12:44.8052837Z 
2022-05-18T05:12:44.8052935Z OK
2022-05-18T05:12:44.8053071Z 
2022-05-18T05:12:44.8055673Z Generating XML reports...
2022-05-18T05:12:44.8095756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051239.xml
2022-05-18T05:12:46.2344884Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:46.2360743Z 
2022-05-18T05:12:46.2360948Z Running tests...
2022-05-18T05:12:46.2361411Z ----------------------------------------------------------------------
2022-05-18T05:12:47.8072026Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:47.8470188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85893
2022-05-18T05:12:47.8580866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85894
2022-05-18T05:12:48.9925020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:48.9925586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:48.9926626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:48.9927352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:48.9932831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:48.9933300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:49.0141057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:12:49.0141566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:12:49.0142261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:12:49.0142934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:12:49.0144974Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-05-18T05:12:49.0145401Z No ranks successfully processed in monitoredBarrier.
2022-05-18T05:12:49.0173573Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-05-18T05:12:49.2632091Z ok (3.027s)
2022-05-18T05:12:49.2632332Z 
2022-05-18T05:12:49.2632728Z ----------------------------------------------------------------------
2022-05-18T05:12:49.2633071Z Ran 1 test in 3.027s
2022-05-18T05:12:49.2633239Z 
2022-05-18T05:12:49.2633335Z OK
2022-05-18T05:12:49.2633453Z 
2022-05-18T05:12:49.2633597Z Generating XML reports...
2022-05-18T05:12:49.2676185Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051246.xml
2022-05-18T05:12:50.6657543Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:50.6672785Z 
2022-05-18T05:12:50.6673206Z Running tests...
2022-05-18T05:12:50.6673709Z ----------------------------------------------------------------------
2022-05-18T05:12:52.2491177Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:52.2900146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86008
2022-05-18T05:12:52.3011513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86009
2022-05-18T05:12:53.4599990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:53.4600554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:53.4601357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:53.4602052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:53.4708214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:53.5612762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:53.5822597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:12:53.5823124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:12:53.5823818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:12:53.5824515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:12:53.6830569Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-05-18T05:12:54.0071954Z ok (3.340s)
2022-05-18T05:12:54.0072126Z 
2022-05-18T05:12:54.0073138Z ----------------------------------------------------------------------
2022-05-18T05:12:54.0073560Z Ran 1 test in 3.340s
2022-05-18T05:12:54.0073734Z 
2022-05-18T05:12:54.0073833Z OK
2022-05-18T05:12:54.0073976Z 
2022-05-18T05:12:54.0074111Z Generating XML reports...
2022-05-18T05:12:54.0115368Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051250.xml
2022-05-18T05:12:55.4335446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:55.4349583Z 
2022-05-18T05:12:55.4350000Z Running tests...
2022-05-18T05:12:55.4350468Z ----------------------------------------------------------------------
2022-05-18T05:12:57.0041101Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:12:57.0435648Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86123
2022-05-18T05:12:57.0543666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86124
2022-05-18T05:12:58.2386741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:12:58.2387319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:12:58.2388131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:58.2388816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:12:58.2493937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:12:58.3400138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:12:58.5595223Z skip: Skipped due to small world size. (3.124s)
2022-05-18T05:12:58.5595498Z 
2022-05-18T05:12:58.5595884Z ----------------------------------------------------------------------
2022-05-18T05:12:58.5596245Z Ran 1 test in 3.124s
2022-05-18T05:12:58.5596413Z 
2022-05-18T05:12:58.5596508Z OK (skipped=1)
2022-05-18T05:12:58.5596672Z 
2022-05-18T05:12:58.5596801Z Generating XML reports...
2022-05-18T05:12:58.5637931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051255.xml
2022-05-18T05:12:59.9674466Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:12:59.9689882Z 
2022-05-18T05:12:59.9690144Z Running tests...
2022-05-18T05:12:59.9690583Z ----------------------------------------------------------------------
2022-05-18T05:12:59.9723485Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T05:12:59.9723830Z 
2022-05-18T05:12:59.9724106Z ----------------------------------------------------------------------
2022-05-18T05:12:59.9724442Z Ran 1 test in 0.003s
2022-05-18T05:12:59.9724603Z 
2022-05-18T05:12:59.9724714Z OK (skipped=1)
2022-05-18T05:12:59.9724871Z 
2022-05-18T05:12:59.9725000Z Generating XML reports...
2022-05-18T05:12:59.9767909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051259.xml
2022-05-18T05:13:01.1942186Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:01.1957428Z 
2022-05-18T05:13:01.1957934Z Running tests...
2022-05-18T05:13:01.1958851Z ----------------------------------------------------------------------
2022-05-18T05:13:01.1992761Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T05:13:01.1993505Z 
2022-05-18T05:13:01.1994081Z ----------------------------------------------------------------------
2022-05-18T05:13:01.1994477Z Ran 1 test in 0.004s
2022-05-18T05:13:01.1994646Z 
2022-05-18T05:13:01.1994757Z OK (skipped=1)
2022-05-18T05:13:01.1995217Z 
2022-05-18T05:13:01.1995360Z Generating XML reports...
2022-05-18T05:13:01.2037593Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051301.xml
2022-05-18T05:13:02.4626048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:02.4640618Z 
2022-05-18T05:13:02.4641171Z Running tests...
2022-05-18T05:13:02.4642081Z ----------------------------------------------------------------------
2022-05-18T05:13:02.4672948Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T05:13:02.4673733Z 
2022-05-18T05:13:02.4674152Z ----------------------------------------------------------------------
2022-05-18T05:13:02.4674471Z Ran 1 test in 0.003s
2022-05-18T05:13:02.4674633Z 
2022-05-18T05:13:02.4674746Z OK (skipped=1)
2022-05-18T05:13:02.4674906Z 
2022-05-18T05:13:02.4675033Z Generating XML reports...
2022-05-18T05:13:02.4718468Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051302.xml
2022-05-18T05:13:03.7047941Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:03.7061555Z 
2022-05-18T05:13:03.7061885Z Running tests...
2022-05-18T05:13:03.7062838Z ----------------------------------------------------------------------
2022-05-18T05:13:03.7093285Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-05-18T05:13:03.7093984Z 
2022-05-18T05:13:03.7094902Z ----------------------------------------------------------------------
2022-05-18T05:13:03.7095246Z Ran 1 test in 0.003s
2022-05-18T05:13:03.7095410Z 
2022-05-18T05:13:03.7095523Z OK (skipped=1)
2022-05-18T05:13:03.7095683Z 
2022-05-18T05:13:03.7095791Z Generating XML reports...
2022-05-18T05:13:03.7129811Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051303.xml
2022-05-18T05:13:04.9646271Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:04.9660648Z 
2022-05-18T05:13:04.9660999Z Running tests...
2022-05-18T05:13:04.9661927Z ----------------------------------------------------------------------
2022-05-18T05:13:04.9692796Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.003s)
2022-05-18T05:13:04.9693569Z 
2022-05-18T05:13:04.9694131Z ----------------------------------------------------------------------
2022-05-18T05:13:04.9694502Z Ran 1 test in 0.003s
2022-05-18T05:13:04.9694666Z 
2022-05-18T05:13:04.9694759Z OK (skipped=1)
2022-05-18T05:13:04.9694915Z 
2022-05-18T05:13:04.9695040Z Generating XML reports...
2022-05-18T05:13:04.9738338Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051304.xml
2022-05-18T05:13:06.2379774Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:06.2394635Z 
2022-05-18T05:13:06.2394901Z Running tests...
2022-05-18T05:13:06.2395341Z ----------------------------------------------------------------------
2022-05-18T05:13:06.2423308Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-05-18T05:13:06.2423615Z 
2022-05-18T05:13:06.2423879Z ----------------------------------------------------------------------
2022-05-18T05:13:06.2424212Z Ran 1 test in 0.003s
2022-05-18T05:13:06.2424378Z 
2022-05-18T05:13:06.2424507Z OK (skipped=1)
2022-05-18T05:13:06.2424663Z 
2022-05-18T05:13:06.2424771Z Generating XML reports...
2022-05-18T05:13:06.2468515Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051306.xml
2022-05-18T05:13:07.4830695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:07.4845842Z 
2022-05-18T05:13:07.4846105Z Running tests...
2022-05-18T05:13:07.4846552Z ----------------------------------------------------------------------
2022-05-18T05:13:07.4879319Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-05-18T05:13:07.4879652Z 
2022-05-18T05:13:07.4879931Z ----------------------------------------------------------------------
2022-05-18T05:13:07.4880274Z Ran 1 test in 0.003s
2022-05-18T05:13:07.4880437Z 
2022-05-18T05:13:07.4880548Z OK (skipped=1)
2022-05-18T05:13:07.4880688Z 
2022-05-18T05:13:07.4880833Z Generating XML reports...
2022-05-18T05:13:07.4924437Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051307.xml
2022-05-18T05:13:08.7157526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:08.7173211Z 
2022-05-18T05:13:08.7173672Z Running tests...
2022-05-18T05:13:08.7174184Z ----------------------------------------------------------------------
2022-05-18T05:13:08.7202772Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-05-18T05:13:08.7203161Z 
2022-05-18T05:13:08.7203476Z ----------------------------------------------------------------------
2022-05-18T05:13:08.7203813Z Ran 1 test in 0.003s
2022-05-18T05:13:08.7203977Z 
2022-05-18T05:13:08.7204088Z OK (skipped=1)
2022-05-18T05:13:08.7204228Z 
2022-05-18T05:13:08.7204354Z Generating XML reports...
2022-05-18T05:13:08.7247855Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051308.xml
2022-05-18T05:13:09.9892721Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:09.9907791Z 
2022-05-18T05:13:09.9908209Z Running tests...
2022-05-18T05:13:09.9908722Z ----------------------------------------------------------------------
2022-05-18T05:13:11.5786457Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:11.6178881Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86496
2022-05-18T05:13:11.6287829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86497
2022-05-18T05:13:12.8269946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:12.8270517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:12.8271324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:12.8272006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:12.8277435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:12.8278412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:13.0338863Z ok (3.043s)
2022-05-18T05:13:13.0339087Z 
2022-05-18T05:13:13.0339490Z ----------------------------------------------------------------------
2022-05-18T05:13:13.0339818Z Ran 1 test in 3.043s
2022-05-18T05:13:13.0339995Z 
2022-05-18T05:13:13.0340092Z OK
2022-05-18T05:13:13.0340228Z 
2022-05-18T05:13:13.0340362Z Generating XML reports...
2022-05-18T05:13:13.0380894Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051309.xml
2022-05-18T05:13:14.4272133Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:14.4286383Z 
2022-05-18T05:13:14.4286577Z Running tests...
2022-05-18T05:13:14.4287292Z ----------------------------------------------------------------------
2022-05-18T05:13:15.9829341Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:16.0229175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86605
2022-05-18T05:13:16.0338361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86606
2022-05-18T05:13:17.2255553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:17.2256200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:17.2257002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:17.2257741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:17.2363563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:17.3265326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:17.5388525Z ok (3.110s)
2022-05-18T05:13:17.5388847Z 
2022-05-18T05:13:17.5389386Z ----------------------------------------------------------------------
2022-05-18T05:13:17.5389736Z Ran 1 test in 3.110s
2022-05-18T05:13:17.5389901Z 
2022-05-18T05:13:17.5389978Z OK
2022-05-18T05:13:17.5390114Z 
2022-05-18T05:13:17.5390245Z Generating XML reports...
2022-05-18T05:13:17.5430790Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051314.xml
2022-05-18T05:13:18.9422602Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:18.9436915Z 
2022-05-18T05:13:18.9437298Z Running tests...
2022-05-18T05:13:18.9437817Z ----------------------------------------------------------------------
2022-05-18T05:13:18.9459008Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T05:13:18.9459342Z 
2022-05-18T05:13:18.9459638Z ----------------------------------------------------------------------
2022-05-18T05:13:18.9459957Z Ran 1 test in 0.002s
2022-05-18T05:13:18.9460124Z 
2022-05-18T05:13:18.9460237Z OK (skipped=1)
2022-05-18T05:13:18.9460399Z 
2022-05-18T05:13:18.9460527Z Generating XML reports...
2022-05-18T05:13:18.9497045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051318.xml
2022-05-18T05:13:20.1853956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:20.1867286Z 
2022-05-18T05:13:20.1867567Z Running tests...
2022-05-18T05:13:20.1867996Z ----------------------------------------------------------------------
2022-05-18T05:13:20.1887339Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T05:13:20.1887686Z 
2022-05-18T05:13:20.1887966Z ----------------------------------------------------------------------
2022-05-18T05:13:20.1888296Z Ran 1 test in 0.002s
2022-05-18T05:13:20.1888461Z 
2022-05-18T05:13:20.1888553Z OK (skipped=1)
2022-05-18T05:13:20.1888707Z 
2022-05-18T05:13:20.1888833Z Generating XML reports...
2022-05-18T05:13:20.1922942Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051320.xml
2022-05-18T05:13:21.4371502Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:21.4385527Z 
2022-05-18T05:13:21.4385964Z Running tests...
2022-05-18T05:13:21.4386446Z ----------------------------------------------------------------------
2022-05-18T05:13:23.0182382Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:23.0575300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86780
2022-05-18T05:13:23.0684926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86781
2022-05-18T05:13:24.2364230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:24.2364801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:24.2365585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:24.2366262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:24.2471669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:24.3378272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:25.5247191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpccimddq_
2022-05-18T05:13:25.5247789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpccimddq_/_remote_module_non_scriptable.py
2022-05-18T05:13:25.5734574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmhd2osxx
2022-05-18T05:13:25.5736925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmhd2osxx/_remote_module_non_scriptable.py
2022-05-18T05:13:26.0765574Z ok (4.638s)
2022-05-18T05:13:26.0765810Z 
2022-05-18T05:13:26.0766203Z ----------------------------------------------------------------------
2022-05-18T05:13:26.0766830Z Ran 1 test in 4.638s
2022-05-18T05:13:26.0766996Z 
2022-05-18T05:13:26.0767097Z OK
2022-05-18T05:13:26.0767215Z 
2022-05-18T05:13:26.0767352Z Generating XML reports...
2022-05-18T05:13:26.0808972Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051321.xml
2022-05-18T05:13:27.4997770Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:27.5012652Z 
2022-05-18T05:13:27.5012920Z Running tests...
2022-05-18T05:13:27.5013348Z ----------------------------------------------------------------------
2022-05-18T05:13:29.0803689Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:29.1207117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86895
2022-05-18T05:13:29.1317239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86896
2022-05-18T05:13:30.3004297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:30.3004860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:30.3005664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:30.3006347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:30.3112079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:30.4018195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:31.5667053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7k5keael
2022-05-18T05:13:31.5667643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7k5keael/_remote_module_non_scriptable.py
2022-05-18T05:13:31.6551124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11vfgy8o
2022-05-18T05:13:31.6552474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11vfgy8o/_remote_module_non_scriptable.py
2022-05-18T05:13:32.1398508Z ok (4.638s)
2022-05-18T05:13:32.1399012Z 
2022-05-18T05:13:32.1399656Z ----------------------------------------------------------------------
2022-05-18T05:13:32.1400022Z Ran 1 test in 4.639s
2022-05-18T05:13:32.1400186Z 
2022-05-18T05:13:32.1400265Z OK
2022-05-18T05:13:32.1400400Z 
2022-05-18T05:13:32.1400540Z Generating XML reports...
2022-05-18T05:13:32.1439880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051327.xml
2022-05-18T05:13:33.5667377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:33.5681840Z 
2022-05-18T05:13:33.5682261Z Running tests...
2022-05-18T05:13:33.5682733Z ----------------------------------------------------------------------
2022-05-18T05:13:35.1501345Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:35.1904088Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87010
2022-05-18T05:13:35.2013387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87011
2022-05-18T05:13:36.4016528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:36.4017089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:36.4017873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:36.4018559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:36.4126257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:36.5030826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:39.0107168Z ok (5.442s)
2022-05-18T05:13:39.0107394Z 
2022-05-18T05:13:39.0107811Z ----------------------------------------------------------------------
2022-05-18T05:13:39.0108159Z Ran 1 test in 5.443s
2022-05-18T05:13:39.0108309Z 
2022-05-18T05:13:39.0108406Z OK
2022-05-18T05:13:39.0111017Z 
2022-05-18T05:13:39.0111477Z Generating XML reports...
2022-05-18T05:13:39.0149758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051333.xml
2022-05-18T05:13:40.4137597Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:40.4151728Z 
2022-05-18T05:13:40.4152053Z Running tests...
2022-05-18T05:13:40.4152481Z ----------------------------------------------------------------------
2022-05-18T05:13:41.9655320Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:42.0051597Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87122
2022-05-18T05:13:42.0158713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87123
2022-05-18T05:13:43.1885713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:43.1886577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:43.1887481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:43.1888351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:43.1995678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:43.2899891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:45.8254208Z ok (5.410s)
2022-05-18T05:13:45.8254518Z 
2022-05-18T05:13:45.8255063Z ----------------------------------------------------------------------
2022-05-18T05:13:45.8255641Z Ran 1 test in 5.410s
2022-05-18T05:13:45.8255914Z 
2022-05-18T05:13:45.8256102Z OK
2022-05-18T05:13:45.8256324Z 
2022-05-18T05:13:45.8256463Z Generating XML reports...
2022-05-18T05:13:45.8296229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051340.xml
2022-05-18T05:13:47.2585157Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:47.2599856Z 
2022-05-18T05:13:47.2600012Z Running tests...
2022-05-18T05:13:47.2600697Z ----------------------------------------------------------------------
2022-05-18T05:13:48.8416453Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:48.8573058Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.597s)
2022-05-18T05:13:48.8573635Z 
2022-05-18T05:13:48.8573935Z ----------------------------------------------------------------------
2022-05-18T05:13:48.8574275Z Ran 1 test in 1.597s
2022-05-18T05:13:48.8574441Z 
2022-05-18T05:13:48.8574533Z OK (skipped=1)
2022-05-18T05:13:48.8574692Z 
2022-05-18T05:13:48.8574819Z Generating XML reports...
2022-05-18T05:13:48.8608386Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051347.xml
2022-05-18T05:13:50.2358978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:50.2373912Z 
2022-05-18T05:13:50.2374963Z Running tests...
2022-05-18T05:13:50.2375622Z ----------------------------------------------------------------------
2022-05-18T05:13:51.8171861Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:51.8325832Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.595s)
2022-05-18T05:13:51.8326408Z 
2022-05-18T05:13:51.8326691Z ----------------------------------------------------------------------
2022-05-18T05:13:51.8327032Z Ran 1 test in 1.596s
2022-05-18T05:13:51.8327180Z 
2022-05-18T05:13:51.8327294Z OK (skipped=1)
2022-05-18T05:13:51.8327453Z 
2022-05-18T05:13:51.8327581Z Generating XML reports...
2022-05-18T05:13:51.8359864Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051350.xml
2022-05-18T05:13:53.2107603Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:53.2123101Z 
2022-05-18T05:13:53.2123546Z Running tests...
2022-05-18T05:13:53.2124060Z ----------------------------------------------------------------------
2022-05-18T05:13:54.7759506Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:54.8166553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87302
2022-05-18T05:13:54.8276191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87303
2022-05-18T05:13:56.0483844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:13:56.0484398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:13:56.0485234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:56.0486158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:13:56.0592560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:13:56.1497540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:13:56.3326320Z skip: Need at least 4 CUDA devices (3.120s)
2022-05-18T05:13:56.3326567Z 
2022-05-18T05:13:56.3326946Z ----------------------------------------------------------------------
2022-05-18T05:13:56.3327290Z Ran 1 test in 3.120s
2022-05-18T05:13:56.3327457Z 
2022-05-18T05:13:56.3327550Z OK (skipped=1)
2022-05-18T05:13:56.3327712Z 
2022-05-18T05:13:56.3327843Z Generating XML reports...
2022-05-18T05:13:56.3368213Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051353.xml
2022-05-18T05:13:57.7371252Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:13:57.7385258Z 
2022-05-18T05:13:57.7385797Z Running tests...
2022-05-18T05:13:57.7386327Z ----------------------------------------------------------------------
2022-05-18T05:13:59.3225739Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:13:59.3628614Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87411
2022-05-18T05:13:59.3740280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87412
2022-05-18T05:14:00.5520939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:00.5521510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:00.5522580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:00.5523271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:00.5629112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:00.6534832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:00.8790977Z skip: Need at least 4 CUDA devices (3.140s)
2022-05-18T05:14:00.8791236Z 
2022-05-18T05:14:00.8791838Z ----------------------------------------------------------------------
2022-05-18T05:14:00.8792440Z Ran 1 test in 3.141s
2022-05-18T05:14:00.8792612Z 
2022-05-18T05:14:00.8792724Z OK (skipped=1)
2022-05-18T05:14:00.8792893Z 
2022-05-18T05:14:00.8793021Z Generating XML reports...
2022-05-18T05:14:00.8833824Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051357.xml
2022-05-18T05:14:02.2957925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:02.2973414Z 
2022-05-18T05:14:02.2973889Z Running tests...
2022-05-18T05:14:02.2974394Z ----------------------------------------------------------------------
2022-05-18T05:14:03.8780792Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:03.9185806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87520
2022-05-18T05:14:03.9297517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87521
2022-05-18T05:14:05.0829797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:05.0830400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:05.0831223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:05.0832160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:05.0837909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:05.0838858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:05.0945537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:14:05.0946070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:14:05.0946805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:14:05.0947504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:14:05.3351999Z ok (3.037s)
2022-05-18T05:14:05.3352356Z 
2022-05-18T05:14:05.3353094Z ----------------------------------------------------------------------
2022-05-18T05:14:05.3353739Z Ran 1 test in 3.038s
2022-05-18T05:14:05.3353911Z 
2022-05-18T05:14:05.3354010Z OK
2022-05-18T05:14:05.3354157Z 
2022-05-18T05:14:05.3354296Z Generating XML reports...
2022-05-18T05:14:05.3396777Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051402.xml
2022-05-18T05:14:06.7585676Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:06.7599838Z 
2022-05-18T05:14:06.7600308Z Running tests...
2022-05-18T05:14:06.7601263Z ----------------------------------------------------------------------
2022-05-18T05:14:08.3446028Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:08.3851188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87639
2022-05-18T05:14:08.3963499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87640
2022-05-18T05:14:09.5734073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:09.5734621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:09.5735444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:09.5736140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:09.5842595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:09.6745490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:09.6954876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:14:09.6955425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:14:09.6956171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:14:09.6956879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:14:09.9016745Z ok (3.141s)
2022-05-18T05:14:09.9016959Z 
2022-05-18T05:14:09.9017375Z ----------------------------------------------------------------------
2022-05-18T05:14:09.9017725Z Ran 1 test in 3.142s
2022-05-18T05:14:09.9017891Z 
2022-05-18T05:14:09.9017970Z OK
2022-05-18T05:14:09.9018104Z 
2022-05-18T05:14:09.9018241Z Generating XML reports...
2022-05-18T05:14:09.9058894Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051406.xml
2022-05-18T05:14:11.3152991Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:11.3168150Z 
2022-05-18T05:14:11.3168447Z Running tests...
2022-05-18T05:14:11.3169151Z ----------------------------------------------------------------------
2022-05-18T05:14:12.8926942Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:12.9333886Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87758
2022-05-18T05:14:12.9443612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87759
2022-05-18T05:14:14.1209899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:14.1210459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:14.1211284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:14.1211995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:14.1218138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:14.1218659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:14.1425886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:14:14.1426410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:14:14.1427126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:14:14.1427832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:14:14.3493930Z ok (3.032s)
2022-05-18T05:14:14.3494156Z 
2022-05-18T05:14:14.3494558Z ----------------------------------------------------------------------
2022-05-18T05:14:14.3494884Z Ran 1 test in 3.033s
2022-05-18T05:14:14.3495055Z 
2022-05-18T05:14:14.3495176Z OK
2022-05-18T05:14:14.3495314Z 
2022-05-18T05:14:14.3495456Z Generating XML reports...
2022-05-18T05:14:14.3535892Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051411.xml
2022-05-18T05:14:15.7590253Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:15.7605565Z 
2022-05-18T05:14:15.7605982Z Running tests...
2022-05-18T05:14:15.7606475Z ----------------------------------------------------------------------
2022-05-18T05:14:17.3440575Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:17.3832960Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87877
2022-05-18T05:14:17.3941130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87878
2022-05-18T05:14:18.5536280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:18.5536835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:18.5537629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:18.5538322Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:18.5646798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:18.6547442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:18.6662708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:14:18.6663268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:14:18.6664177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:14:18.6664934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:14:18.8995880Z ok (3.139s)
2022-05-18T05:14:18.8996097Z 
2022-05-18T05:14:18.8996492Z ----------------------------------------------------------------------
2022-05-18T05:14:18.8996834Z Ran 1 test in 3.139s
2022-05-18T05:14:18.8996982Z 
2022-05-18T05:14:18.8997086Z OK
2022-05-18T05:14:18.8997223Z 
2022-05-18T05:14:18.8997357Z Generating XML reports...
2022-05-18T05:14:18.9038455Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051415.xml
2022-05-18T05:14:20.3017485Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:20.3031081Z 
2022-05-18T05:14:20.3031356Z Running tests...
2022-05-18T05:14:20.3031824Z ----------------------------------------------------------------------
2022-05-18T05:14:21.8435840Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:21.8827716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87996
2022-05-18T05:14:21.8938738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87997
2022-05-18T05:14:23.0945269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:23.0945861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:23.0946948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:23.0947648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:23.1055324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:23.1958783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:23.3990436Z skip: Skipped due to small world size. (3.096s)
2022-05-18T05:14:23.3990893Z 
2022-05-18T05:14:23.3991323Z ----------------------------------------------------------------------
2022-05-18T05:14:23.3991665Z Ran 1 test in 3.096s
2022-05-18T05:14:23.3991812Z 
2022-05-18T05:14:23.3991925Z OK (skipped=1)
2022-05-18T05:14:23.3992080Z 
2022-05-18T05:14:23.3992209Z Generating XML reports...
2022-05-18T05:14:23.4033007Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051420.xml
2022-05-18T05:14:24.8113682Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:24.8129221Z 
2022-05-18T05:14:24.8129677Z Running tests...
2022-05-18T05:14:24.8130202Z ----------------------------------------------------------------------
2022-05-18T05:14:26.3961372Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:26.4366910Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88105
2022-05-18T05:14:26.4477222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88106
2022-05-18T05:14:27.6397818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:27.6398756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:27.6399562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:27.6400304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:27.6506103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:27.7412117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:27.9530603Z skip: Skipped due to small world size. (3.140s)
2022-05-18T05:14:27.9530851Z 
2022-05-18T05:14:27.9531271Z ----------------------------------------------------------------------
2022-05-18T05:14:27.9531619Z Ran 1 test in 3.140s
2022-05-18T05:14:27.9531765Z 
2022-05-18T05:14:27.9531879Z OK (skipped=1)
2022-05-18T05:14:27.9532036Z 
2022-05-18T05:14:27.9532165Z Generating XML reports...
2022-05-18T05:14:27.9573254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051424.xml
2022-05-18T05:14:29.3362044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:29.3376661Z 
2022-05-18T05:14:29.3376997Z Running tests...
2022-05-18T05:14:29.3377442Z ----------------------------------------------------------------------
2022-05-18T05:14:30.8962186Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:30.9365189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88214
2022-05-18T05:14:30.9474188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88215
2022-05-18T05:14:32.1801705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:32.1802251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:32.1803061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:32.1804008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:32.1911301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:32.2815446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:32.4525430Z skip: Skipped due to small world size. (3.114s)
2022-05-18T05:14:32.4525878Z 
2022-05-18T05:14:32.4526566Z ----------------------------------------------------------------------
2022-05-18T05:14:32.4527213Z Ran 1 test in 3.115s
2022-05-18T05:14:32.4527517Z 
2022-05-18T05:14:32.4527701Z OK (skipped=1)
2022-05-18T05:14:32.4527997Z 
2022-05-18T05:14:32.4528225Z Generating XML reports...
2022-05-18T05:14:32.4569788Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051429.xml
2022-05-18T05:14:33.8509981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:33.8524415Z 
2022-05-18T05:14:33.8524842Z Running tests...
2022-05-18T05:14:33.8525339Z ----------------------------------------------------------------------
2022-05-18T05:14:35.4374941Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:35.4783966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88323
2022-05-18T05:14:35.4893824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88324
2022-05-18T05:14:36.6556156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:36.6556720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:36.6557516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:36.6558467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:36.6664029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:36.7569350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:36.8944095Z skip: Skipped due to small world size. (3.041s)
2022-05-18T05:14:36.8944355Z 
2022-05-18T05:14:36.8944708Z ----------------------------------------------------------------------
2022-05-18T05:14:36.8945051Z Ran 1 test in 3.042s
2022-05-18T05:14:36.8945222Z 
2022-05-18T05:14:36.8945334Z OK (skipped=1)
2022-05-18T05:14:36.8945495Z 
2022-05-18T05:14:36.8945629Z Generating XML reports...
2022-05-18T05:14:36.8987567Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051433.xml
2022-05-18T05:14:38.2607624Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:38.2621993Z 
2022-05-18T05:14:38.2622197Z Running tests...
2022-05-18T05:14:38.2622657Z ----------------------------------------------------------------------
2022-05-18T05:14:39.8288347Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:39.8688121Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88432
2022-05-18T05:14:39.8796525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88433
2022-05-18T05:14:41.0452544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:41.0453131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:41.0453943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:41.0454896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:41.0460401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:41.0460918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:41.2847951Z ok (3.022s)
2022-05-18T05:14:41.2848179Z 
2022-05-18T05:14:41.2848565Z ----------------------------------------------------------------------
2022-05-18T05:14:41.2848895Z Ran 1 test in 3.023s
2022-05-18T05:14:41.2849066Z 
2022-05-18T05:14:41.2849166Z OK
2022-05-18T05:14:41.2849305Z 
2022-05-18T05:14:41.2849441Z Generating XML reports...
2022-05-18T05:14:41.2890693Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051438.xml
2022-05-18T05:14:42.6959076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:42.6973678Z 
2022-05-18T05:14:42.6973928Z Running tests...
2022-05-18T05:14:42.6974390Z ----------------------------------------------------------------------
2022-05-18T05:14:44.2620084Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:44.3026514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88545
2022-05-18T05:14:44.3137740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88546
2022-05-18T05:14:45.4696354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:45.4696952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:45.4697734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:45.4698435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:45.4704189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:45.4704690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:45.7188740Z ok (3.021s)
2022-05-18T05:14:45.7188994Z 
2022-05-18T05:14:45.7189399Z ----------------------------------------------------------------------
2022-05-18T05:14:45.7189745Z Ran 1 test in 3.021s
2022-05-18T05:14:45.7189915Z 
2022-05-18T05:14:45.7189994Z OK
2022-05-18T05:14:45.7190131Z 
2022-05-18T05:14:45.7190265Z Generating XML reports...
2022-05-18T05:14:45.7231518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051442.xml
2022-05-18T05:14:47.1228567Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:47.1243529Z 
2022-05-18T05:14:47.1244054Z Running tests...
2022-05-18T05:14:47.1244583Z ----------------------------------------------------------------------
2022-05-18T05:14:47.1267110Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-05-18T05:14:47.1268036Z 
2022-05-18T05:14:47.1268793Z ----------------------------------------------------------------------
2022-05-18T05:14:47.1269167Z Ran 1 test in 0.002s
2022-05-18T05:14:47.1269337Z 
2022-05-18T05:14:47.1269460Z OK (skipped=1)
2022-05-18T05:14:47.1269598Z 
2022-05-18T05:14:47.1269727Z Generating XML reports...
2022-05-18T05:14:47.1305536Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051447.xml
2022-05-18T05:14:48.3818955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:48.3833125Z 
2022-05-18T05:14:48.3833612Z Running tests...
2022-05-18T05:14:48.3834099Z ----------------------------------------------------------------------
2022-05-18T05:14:49.9491201Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:49.9896967Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88691
2022-05-18T05:14:50.0007460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88692
2022-05-18T05:14:51.1714392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:51.1714957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:51.1715768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:51.1716468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:51.1823729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:51.2725843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:51.5059710Z ok (3.122s)
2022-05-18T05:14:51.5059924Z 
2022-05-18T05:14:51.5060327Z ----------------------------------------------------------------------
2022-05-18T05:14:51.5060691Z Ran 1 test in 3.123s
2022-05-18T05:14:51.5060866Z 
2022-05-18T05:14:51.5060964Z OK
2022-05-18T05:14:51.5061102Z 
2022-05-18T05:14:51.5061236Z Generating XML reports...
2022-05-18T05:14:51.5101668Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051448.xml
2022-05-18T05:14:52.8850113Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:52.8863739Z 
2022-05-18T05:14:52.8864171Z Running tests...
2022-05-18T05:14:52.8864662Z ----------------------------------------------------------------------
2022-05-18T05:14:54.4344892Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:14:54.4741946Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88804
2022-05-18T05:14:54.4849461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88805
2022-05-18T05:14:55.6570050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:14:55.6570684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:14:55.6571526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:55.6572252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:14:55.6577997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:14:55.6578970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:14:55.8898532Z ok (3.003s)
2022-05-18T05:14:55.8898753Z 
2022-05-18T05:14:55.8899127Z ----------------------------------------------------------------------
2022-05-18T05:14:55.8899471Z Ran 1 test in 3.003s
2022-05-18T05:14:55.8899636Z 
2022-05-18T05:14:55.8899736Z OK
2022-05-18T05:14:55.8899889Z 
2022-05-18T05:14:55.8900023Z Generating XML reports...
2022-05-18T05:14:55.8940223Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051452.xml
2022-05-18T05:14:57.2861013Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:57.2876724Z 
2022-05-18T05:14:57.2877161Z Running tests...
2022-05-18T05:14:57.2877649Z ----------------------------------------------------------------------
2022-05-18T05:14:57.2901624Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-05-18T05:14:57.2902424Z 
2022-05-18T05:14:57.2903089Z ----------------------------------------------------------------------
2022-05-18T05:14:57.2903782Z Ran 1 test in 0.003s
2022-05-18T05:14:57.2904124Z 
2022-05-18T05:14:57.2904314Z OK (skipped=1)
2022-05-18T05:14:57.2904617Z 
2022-05-18T05:14:57.2904855Z Generating XML reports...
2022-05-18T05:14:57.2942312Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051457.xml
2022-05-18T05:14:58.5463670Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:58.5478808Z 
2022-05-18T05:14:58.5479045Z Running tests...
2022-05-18T05:14:58.5479505Z ----------------------------------------------------------------------
2022-05-18T05:14:58.5503386Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-05-18T05:14:58.5503710Z 
2022-05-18T05:14:58.5504005Z ----------------------------------------------------------------------
2022-05-18T05:14:58.5504341Z Ran 1 test in 0.002s
2022-05-18T05:14:58.5504507Z 
2022-05-18T05:14:58.5504617Z OK (skipped=1)
2022-05-18T05:14:58.5504773Z 
2022-05-18T05:14:58.5506090Z Generating XML reports...
2022-05-18T05:14:58.5541337Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051458.xml
2022-05-18T05:14:59.8133268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:14:59.8147266Z 
2022-05-18T05:14:59.8147658Z Running tests...
2022-05-18T05:14:59.8148114Z ----------------------------------------------------------------------
2022-05-18T05:15:01.3957090Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:01.4377641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88983
2022-05-18T05:15:01.4491949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88984
2022-05-18T05:15:02.6086420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:02.6086981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:02.6088031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:02.6088784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:02.6095214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:02.6095710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:02.8545103Z ok (3.039s)
2022-05-18T05:15:02.8545505Z 
2022-05-18T05:15:02.8546178Z ----------------------------------------------------------------------
2022-05-18T05:15:02.8546832Z Ran 1 test in 3.040s
2022-05-18T05:15:02.8547134Z 
2022-05-18T05:15:02.8547304Z OK
2022-05-18T05:15:02.8547570Z 
2022-05-18T05:15:02.8547807Z Generating XML reports...
2022-05-18T05:15:02.8589879Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051459.xml
2022-05-18T05:15:04.2152442Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:04.2166704Z 
2022-05-18T05:15:04.2166859Z Running tests...
2022-05-18T05:15:04.2167859Z ----------------------------------------------------------------------
2022-05-18T05:15:05.7953573Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:05.8360085Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89096
2022-05-18T05:15:05.8469806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89097
2022-05-18T05:15:07.0058010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:07.0058813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:07.0059611Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:07.0060321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:07.0166402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:07.1069172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:07.3523390Z ok (3.135s)
2022-05-18T05:15:07.3523637Z 
2022-05-18T05:15:07.3524031Z ----------------------------------------------------------------------
2022-05-18T05:15:07.3524373Z Ran 1 test in 3.136s
2022-05-18T05:15:07.3524541Z 
2022-05-18T05:15:07.3524661Z OK
2022-05-18T05:15:07.3524801Z 
2022-05-18T05:15:07.3524919Z Generating XML reports...
2022-05-18T05:15:07.3566046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051504.xml
2022-05-18T05:15:08.7808867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:08.7822943Z 
2022-05-18T05:15:08.7823225Z Running tests...
2022-05-18T05:15:08.7823664Z ----------------------------------------------------------------------
2022-05-18T05:15:10.3681412Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:10.4086806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89209
2022-05-18T05:15:10.4195619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89210
2022-05-18T05:15:11.6344535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:11.6345148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:11.6345948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:11.6346847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:11.6452876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:11.7355813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:11.9247316Z ok (3.142s)
2022-05-18T05:15:11.9247508Z 
2022-05-18T05:15:11.9248052Z ----------------------------------------------------------------------
2022-05-18T05:15:11.9248491Z Ran 1 test in 3.142s
2022-05-18T05:15:11.9248666Z 
2022-05-18T05:15:11.9248762Z OK
2022-05-18T05:15:11.9248899Z 
2022-05-18T05:15:11.9249032Z Generating XML reports...
2022-05-18T05:15:11.9289964Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051508.xml
2022-05-18T05:15:13.3356899Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:13.3372136Z 
2022-05-18T05:15:13.3372734Z Running tests...
2022-05-18T05:15:13.3373192Z ----------------------------------------------------------------------
2022-05-18T05:15:14.9277866Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:14.9687523Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89318
2022-05-18T05:15:14.9800311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89319
2022-05-18T05:15:16.1905685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:16.1906379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:16.1907677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:16.1908586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:16.2014317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:16.2916172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:16.4853514Z ok (3.148s)
2022-05-18T05:15:16.4853733Z 
2022-05-18T05:15:16.4854146Z ----------------------------------------------------------------------
2022-05-18T05:15:16.4854484Z Ran 1 test in 3.148s
2022-05-18T05:15:16.4854654Z 
2022-05-18T05:15:16.4854731Z OK
2022-05-18T05:15:16.4854870Z 
2022-05-18T05:15:16.4855002Z Generating XML reports...
2022-05-18T05:15:16.4895600Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051513.xml
2022-05-18T05:15:17.8936168Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:17.8950479Z 
2022-05-18T05:15:17.8950907Z Running tests...
2022-05-18T05:15:17.8951417Z ----------------------------------------------------------------------
2022-05-18T05:15:17.8971532Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T05:15:17.8971851Z 
2022-05-18T05:15:17.8972269Z ----------------------------------------------------------------------
2022-05-18T05:15:17.8972744Z Ran 1 test in 0.002s
2022-05-18T05:15:17.8972894Z 
2022-05-18T05:15:17.8973005Z OK (skipped=1)
2022-05-18T05:15:17.8973160Z 
2022-05-18T05:15:17.8973284Z Generating XML reports...
2022-05-18T05:15:17.9009128Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051517.xml
2022-05-18T05:15:19.1320572Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:19.1335836Z 
2022-05-18T05:15:19.1336168Z Running tests...
2022-05-18T05:15:19.1336603Z ----------------------------------------------------------------------
2022-05-18T05:15:19.1356988Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T05:15:19.1357375Z 
2022-05-18T05:15:19.1357723Z ----------------------------------------------------------------------
2022-05-18T05:15:19.1358448Z Ran 1 test in 0.002s
2022-05-18T05:15:19.1358632Z 
2022-05-18T05:15:19.1358730Z OK (skipped=1)
2022-05-18T05:15:19.1358889Z 
2022-05-18T05:15:19.1359018Z Generating XML reports...
2022-05-18T05:15:19.1393529Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051519.xml
2022-05-18T05:15:20.3616405Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:20.3630865Z 
2022-05-18T05:15:20.3631219Z Running tests...
2022-05-18T05:15:20.3631655Z ----------------------------------------------------------------------
2022-05-18T05:15:21.9391189Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:21.9795876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89497
2022-05-18T05:15:21.9905808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89498
2022-05-18T05:15:23.1414832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:23.1415386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:23.1416185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:23.1416891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:23.1523319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:23.2425210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:23.2633063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:15:23.2634102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:15:23.2635452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:15:23.2636939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:15:23.4958780Z ok (3.132s)
2022-05-18T05:15:23.4959495Z 
2022-05-18T05:15:23.4960137Z ----------------------------------------------------------------------
2022-05-18T05:15:23.4960793Z Ran 1 test in 3.133s
2022-05-18T05:15:23.4961036Z 
2022-05-18T05:15:23.4961134Z OK
2022-05-18T05:15:23.4961273Z 
2022-05-18T05:15:23.4961394Z Generating XML reports...
2022-05-18T05:15:23.5001196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051520.xml
2022-05-18T05:15:24.9248617Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:24.9263040Z 
2022-05-18T05:15:24.9263312Z Running tests...
2022-05-18T05:15:24.9263764Z ----------------------------------------------------------------------
2022-05-18T05:15:26.5055739Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:26.5459726Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89616
2022-05-18T05:15:26.5569377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89617
2022-05-18T05:15:27.7942777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:27.7943344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:27.7944328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:27.7945057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:27.8050927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:27.8956234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:28.0623810Z skip: Skipped due to small world size. (3.136s)
2022-05-18T05:15:28.0624067Z 
2022-05-18T05:15:28.0624474Z ----------------------------------------------------------------------
2022-05-18T05:15:28.0624844Z Ran 1 test in 3.136s
2022-05-18T05:15:28.0625009Z 
2022-05-18T05:15:28.0625104Z OK (skipped=1)
2022-05-18T05:15:28.0625262Z 
2022-05-18T05:15:28.0625393Z Generating XML reports...
2022-05-18T05:15:28.0666653Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051524.xml
2022-05-18T05:15:29.4667763Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:29.4681860Z 
2022-05-18T05:15:29.4682272Z Running tests...
2022-05-18T05:15:29.4682917Z ----------------------------------------------------------------------
2022-05-18T05:15:31.0289256Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:31.0693385Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89725
2022-05-18T05:15:31.0803505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89726
2022-05-18T05:15:32.2310255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:32.2310827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:32.2311679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:32.2312369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:32.2318318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:32.2319622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:32.3851050Z ok (2.917s)
2022-05-18T05:15:32.3851287Z 
2022-05-18T05:15:32.3851666Z ----------------------------------------------------------------------
2022-05-18T05:15:32.3852013Z Ran 1 test in 2.917s
2022-05-18T05:15:32.3852205Z 
2022-05-18T05:15:32.3852300Z OK
2022-05-18T05:15:32.3852437Z 
2022-05-18T05:15:32.3852582Z Generating XML reports...
2022-05-18T05:15:32.3894245Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051529.xml
2022-05-18T05:15:33.7875439Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:33.7889757Z 
2022-05-18T05:15:33.7901529Z Running tests...
2022-05-18T05:15:33.7902072Z ----------------------------------------------------------------------
2022-05-18T05:15:35.3772118Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:35.4176013Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89834
2022-05-18T05:15:35.4286957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89835
2022-05-18T05:15:36.6098716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:36.6099290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:36.6100089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:36.6101002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:36.6206739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:36.7108602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:36.9339477Z ok (3.145s)
2022-05-18T05:15:36.9339662Z 
2022-05-18T05:15:36.9340124Z ----------------------------------------------------------------------
2022-05-18T05:15:36.9340448Z Ran 1 test in 3.145s
2022-05-18T05:15:36.9340621Z 
2022-05-18T05:15:36.9340726Z OK
2022-05-18T05:15:36.9340861Z 
2022-05-18T05:15:36.9341004Z Generating XML reports...
2022-05-18T05:15:36.9381359Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051533.xml
2022-05-18T05:15:38.3371148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:38.3385465Z 
2022-05-18T05:15:38.3385762Z Running tests...
2022-05-18T05:15:38.3386210Z ----------------------------------------------------------------------
2022-05-18T05:15:39.9228276Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:39.9634507Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89943
2022-05-18T05:15:39.9745443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89944
2022-05-18T05:15:41.1858601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:41.1859178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:41.1860245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:41.1860959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:41.1966590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:41.2868987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:41.4798076Z ok (3.141s)
2022-05-18T05:15:41.4798553Z 
2022-05-18T05:15:41.4798947Z ----------------------------------------------------------------------
2022-05-18T05:15:41.4799295Z Ran 1 test in 3.141s
2022-05-18T05:15:41.4799467Z 
2022-05-18T05:15:41.4799548Z OK
2022-05-18T05:15:41.4799692Z 
2022-05-18T05:15:41.4801191Z Generating XML reports...
2022-05-18T05:15:41.4840838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051538.xml
2022-05-18T05:15:42.8802231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:42.8816484Z 
2022-05-18T05:15:42.8816739Z Running tests...
2022-05-18T05:15:42.8817200Z ----------------------------------------------------------------------
2022-05-18T05:15:44.4730302Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:44.5124805Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90052
2022-05-18T05:15:44.5234045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90053
2022-05-18T05:15:45.7244669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:45.7245248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:45.7246053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:45.7246764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:45.7352390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:45.8257320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:46.0287457Z ok (3.147s)
2022-05-18T05:15:46.0287677Z 
2022-05-18T05:15:46.0288077Z ----------------------------------------------------------------------
2022-05-18T05:15:46.0288423Z Ran 1 test in 3.147s
2022-05-18T05:15:46.0288593Z 
2022-05-18T05:15:46.0288672Z OK
2022-05-18T05:15:46.0288815Z 
2022-05-18T05:15:46.0288953Z Generating XML reports...
2022-05-18T05:15:46.0328548Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051542.xml
2022-05-18T05:15:47.4292338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:47.4306882Z 
2022-05-18T05:15:47.4307094Z Running tests...
2022-05-18T05:15:47.4307550Z ----------------------------------------------------------------------
2022-05-18T05:15:49.0104068Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:49.0500210Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90165
2022-05-18T05:15:49.0609921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90166
2022-05-18T05:15:50.2274351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:50.2274926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:50.2275976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:50.2276689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:50.2382204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:50.3286806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:50.5661508Z ok (3.135s)
2022-05-18T05:15:50.5661736Z 
2022-05-18T05:15:50.5662133Z ----------------------------------------------------------------------
2022-05-18T05:15:50.5662457Z Ran 1 test in 3.135s
2022-05-18T05:15:50.5662631Z 
2022-05-18T05:15:50.5662965Z OK
2022-05-18T05:15:50.5663170Z 
2022-05-18T05:15:50.5663316Z Generating XML reports...
2022-05-18T05:15:50.5703563Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051547.xml
2022-05-18T05:15:51.9637714Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:51.9651795Z 
2022-05-18T05:15:51.9652253Z Running tests...
2022-05-18T05:15:51.9652736Z ----------------------------------------------------------------------
2022-05-18T05:15:53.5081544Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:15:53.5477178Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90278
2022-05-18T05:15:53.5584527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90279
2022-05-18T05:15:54.7362810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:15:54.7363375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:15:54.7364188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:54.7364902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:15:54.7371482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:15:54.7372221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:15:54.9633031Z ok (2.998s)
2022-05-18T05:15:54.9633225Z 
2022-05-18T05:15:54.9633617Z ----------------------------------------------------------------------
2022-05-18T05:15:54.9633967Z Ran 1 test in 2.998s
2022-05-18T05:15:54.9634136Z 
2022-05-18T05:15:54.9634235Z OK
2022-05-18T05:15:54.9634355Z 
2022-05-18T05:15:54.9634492Z Generating XML reports...
2022-05-18T05:15:54.9675346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051551.xml
2022-05-18T05:15:56.3398895Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:56.3414530Z 
2022-05-18T05:15:56.3414797Z Running tests...
2022-05-18T05:15:56.3415239Z ----------------------------------------------------------------------
2022-05-18T05:15:56.3434575Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T05:15:56.3434868Z 
2022-05-18T05:15:56.3435901Z ----------------------------------------------------------------------
2022-05-18T05:15:56.3436611Z Ran 1 test in 0.002s
2022-05-18T05:15:56.3436902Z 
2022-05-18T05:15:56.3437125Z OK (skipped=1)
2022-05-18T05:15:56.3437442Z 
2022-05-18T05:15:56.3437694Z Generating XML reports...
2022-05-18T05:15:56.3475134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051556.xml
2022-05-18T05:15:57.5727911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:57.5742926Z 
2022-05-18T05:15:57.5744009Z Running tests...
2022-05-18T05:15:57.5744967Z ----------------------------------------------------------------------
2022-05-18T05:15:57.5765914Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T05:15:57.5766561Z 
2022-05-18T05:15:57.5767152Z ----------------------------------------------------------------------
2022-05-18T05:15:57.5767792Z Ran 1 test in 0.002s
2022-05-18T05:15:57.5768109Z 
2022-05-18T05:15:57.5768323Z OK (skipped=1)
2022-05-18T05:15:57.5768621Z 
2022-05-18T05:15:57.5768865Z Generating XML reports...
2022-05-18T05:15:57.5806317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051557.xml
2022-05-18T05:15:58.8310952Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:15:58.8326516Z 
2022-05-18T05:15:58.8326791Z Running tests...
2022-05-18T05:15:58.8327241Z ----------------------------------------------------------------------
2022-05-18T05:15:58.8349567Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T05:15:58.8350369Z 
2022-05-18T05:15:58.8350874Z ----------------------------------------------------------------------
2022-05-18T05:15:58.8351491Z Ran 1 test in 0.002s
2022-05-18T05:15:58.8351858Z 
2022-05-18T05:15:58.8352097Z OK (skipped=1)
2022-05-18T05:15:58.8352409Z 
2022-05-18T05:15:58.8352650Z Generating XML reports...
2022-05-18T05:15:58.8389400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051558.xml
2022-05-18T05:16:00.0896658Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:00.0910632Z 
2022-05-18T05:16:00.0911055Z Running tests...
2022-05-18T05:16:00.0911565Z ----------------------------------------------------------------------
2022-05-18T05:16:01.6859331Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:01.7261619Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90490
2022-05-18T05:16:01.7372846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90491
2022-05-18T05:16:02.9274707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:02.9275300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:02.9276101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:02.9276805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:02.9282607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:02.9283360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:03.1424290Z ok (3.051s)
2022-05-18T05:16:03.1424512Z 
2022-05-18T05:16:03.1425056Z ----------------------------------------------------------------------
2022-05-18T05:16:03.1425514Z Ran 1 test in 3.051s
2022-05-18T05:16:03.1425694Z 
2022-05-18T05:16:03.1425810Z OK
2022-05-18T05:16:03.1427036Z 
2022-05-18T05:16:03.1427372Z Generating XML reports...
2022-05-18T05:16:03.1468802Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051600.xml
2022-05-18T05:16:04.5310019Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:04.5323885Z 
2022-05-18T05:16:04.5324167Z Running tests...
2022-05-18T05:16:04.5324631Z ----------------------------------------------------------------------
2022-05-18T05:16:06.0702730Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:06.1098008Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90603
2022-05-18T05:16:06.1209089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90604
2022-05-18T05:16:07.2838903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:07.2839450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:07.2840258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:07.2840969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:07.2948520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:07.3850053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:07.6260211Z ok (3.093s)
2022-05-18T05:16:07.6260438Z 
2022-05-18T05:16:07.6260829Z ----------------------------------------------------------------------
2022-05-18T05:16:07.6261180Z Ran 1 test in 3.094s
2022-05-18T05:16:07.6261328Z 
2022-05-18T05:16:07.6261438Z OK
2022-05-18T05:16:07.6261576Z 
2022-05-18T05:16:07.6261731Z Generating XML reports...
2022-05-18T05:16:07.6301904Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051604.xml
2022-05-18T05:16:09.0437377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:09.0452294Z 
2022-05-18T05:16:09.0452583Z Running tests...
2022-05-18T05:16:09.0453011Z ----------------------------------------------------------------------
2022-05-18T05:16:10.6317011Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:10.6711885Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90712
2022-05-18T05:16:10.6823302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90713
2022-05-18T05:16:11.9031049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:11.9031832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:11.9032670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:11.9033373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:11.9039388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:11.9039868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:12.0873166Z ok (3.042s)
2022-05-18T05:16:12.0873408Z 
2022-05-18T05:16:12.0873852Z ----------------------------------------------------------------------
2022-05-18T05:16:12.0874205Z Ran 1 test in 3.042s
2022-05-18T05:16:12.0874373Z 
2022-05-18T05:16:12.0874471Z OK
2022-05-18T05:16:12.0874615Z 
2022-05-18T05:16:12.0874732Z Generating XML reports...
2022-05-18T05:16:12.0915341Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051609.xml
2022-05-18T05:16:13.4813104Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:13.4826624Z 
2022-05-18T05:16:13.4826894Z Running tests...
2022-05-18T05:16:13.4827324Z ----------------------------------------------------------------------
2022-05-18T05:16:15.0210877Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:15.0603175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90825
2022-05-18T05:16:15.0714916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90826
2022-05-18T05:16:16.2805574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:16.2806142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:16.2806935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:16.2807640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:16.2915590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:16.3817855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:16.5767711Z ok (3.094s)
2022-05-18T05:16:16.5767942Z 
2022-05-18T05:16:16.5768371Z ----------------------------------------------------------------------
2022-05-18T05:16:16.5768722Z Ran 1 test in 3.094s
2022-05-18T05:16:16.5768872Z 
2022-05-18T05:16:16.5768975Z OK
2022-05-18T05:16:16.5769114Z 
2022-05-18T05:16:16.5769248Z Generating XML reports...
2022-05-18T05:16:16.5817444Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051613.xml
2022-05-18T05:16:17.9826619Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:17.9840623Z 
2022-05-18T05:16:17.9841009Z Running tests...
2022-05-18T05:16:17.9841507Z ----------------------------------------------------------------------
2022-05-18T05:16:19.5693372Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:19.6097147Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90938
2022-05-18T05:16:19.6207266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90939
2022-05-18T05:16:20.8119341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:20.8119941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:20.8120961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:20.8121698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:20.8128387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:20.8128907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:21.0258177Z ok (3.041s)
2022-05-18T05:16:21.0258404Z 
2022-05-18T05:16:21.0258820Z ----------------------------------------------------------------------
2022-05-18T05:16:21.0259176Z Ran 1 test in 3.042s
2022-05-18T05:16:21.0259347Z 
2022-05-18T05:16:21.0259446Z OK
2022-05-18T05:16:21.0259594Z 
2022-05-18T05:16:21.0259733Z Generating XML reports...
2022-05-18T05:16:21.0300711Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051617.xml
2022-05-18T05:16:22.4300339Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:22.4314509Z 
2022-05-18T05:16:22.4314944Z Running tests...
2022-05-18T05:16:22.4315461Z ----------------------------------------------------------------------
2022-05-18T05:16:24.0015071Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:24.0409467Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91137
2022-05-18T05:16:24.0518824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91138
2022-05-18T05:16:25.2512109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:25.2512716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:25.2513529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:25.2514238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:25.2622062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:25.3526194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:26.9597511Z ok (4.528s)
2022-05-18T05:16:26.9597744Z 
2022-05-18T05:16:26.9598456Z ----------------------------------------------------------------------
2022-05-18T05:16:26.9598817Z Ran 1 test in 4.528s
2022-05-18T05:16:26.9598986Z 
2022-05-18T05:16:26.9599090Z OK
2022-05-18T05:16:26.9599232Z 
2022-05-18T05:16:26.9599374Z Generating XML reports...
2022-05-18T05:16:26.9637393Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051622.xml
2022-05-18T05:16:28.3841753Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:28.3856459Z 
2022-05-18T05:16:28.3856813Z Running tests...
2022-05-18T05:16:28.3857262Z ----------------------------------------------------------------------
2022-05-18T05:16:29.9657356Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:30.0052058Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91338
2022-05-18T05:16:30.0161307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91339
2022-05-18T05:16:31.2386585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:31.2387190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:31.2387983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:31.2388899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:31.2494659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:31.3400264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:33.1244473Z ok (4.738s)
2022-05-18T05:16:33.1244699Z 
2022-05-18T05:16:33.1245354Z ----------------------------------------------------------------------
2022-05-18T05:16:33.1245740Z Ran 1 test in 4.739s
2022-05-18T05:16:33.1245912Z 
2022-05-18T05:16:33.1246021Z OK
2022-05-18T05:16:33.1246167Z 
2022-05-18T05:16:33.1246308Z Generating XML reports...
2022-05-18T05:16:33.1286994Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051628.xml
2022-05-18T05:16:34.5283283Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:34.5296916Z 
2022-05-18T05:16:34.5297400Z Running tests...
2022-05-18T05:16:34.5297902Z ----------------------------------------------------------------------
2022-05-18T05:16:36.0698291Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:36.1099153Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91453
2022-05-18T05:16:36.1210030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91454
2022-05-18T05:16:37.2780916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:37.2781760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:37.2782537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:37.2783256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:37.2887931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:37.2982173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47b55t8s
2022-05-18T05:16:37.2984448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47b55t8s/_remote_module_non_scriptable.py
2022-05-18T05:16:37.3792615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:37.3891987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0lefsqdv
2022-05-18T05:16:37.3895064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0lefsqdv/_remote_module_non_scriptable.py
2022-05-18T05:16:37.6261801Z ok (3.096s)
2022-05-18T05:16:37.6262005Z 
2022-05-18T05:16:37.6262398Z ----------------------------------------------------------------------
2022-05-18T05:16:37.6262721Z Ran 1 test in 3.096s
2022-05-18T05:16:37.6262904Z 
2022-05-18T05:16:37.6263000Z OK
2022-05-18T05:16:37.6263139Z 
2022-05-18T05:16:37.6263276Z Generating XML reports...
2022-05-18T05:16:37.6304364Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051634.xml
2022-05-18T05:16:39.0542341Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:39.0557165Z 
2022-05-18T05:16:39.0557672Z Running tests...
2022-05-18T05:16:39.0558585Z ----------------------------------------------------------------------
2022-05-18T05:16:40.6215265Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:40.6612959Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91566
2022-05-18T05:16:40.6723528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91567
2022-05-18T05:16:41.8941985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:41.8942605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:41.8943390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:41.8944095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:41.9049461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:41.9954878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:43.1604865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb9y4892n
2022-05-18T05:16:43.1605791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb9y4892n/_remote_module_non_scriptable.py
2022-05-18T05:16:43.2583464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaqiiktvr
2022-05-18T05:16:43.2584362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaqiiktvr/_remote_module_non_scriptable.py
2022-05-18T05:16:43.5799345Z ok (4.524s)
2022-05-18T05:16:43.5799697Z 
2022-05-18T05:16:43.5800259Z ----------------------------------------------------------------------
2022-05-18T05:16:43.5800609Z Ran 1 test in 4.524s
2022-05-18T05:16:43.5800759Z 
2022-05-18T05:16:43.5800856Z OK
2022-05-18T05:16:43.5801004Z 
2022-05-18T05:16:43.5801141Z Generating XML reports...
2022-05-18T05:16:43.5841269Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051639.xml
2022-05-18T05:16:45.0321558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:45.0336171Z 
2022-05-18T05:16:45.0336328Z Running tests...
2022-05-18T05:16:45.0337164Z ----------------------------------------------------------------------
2022-05-18T05:16:46.6235326Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:46.6640158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91677
2022-05-18T05:16:46.6750292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91678
2022-05-18T05:16:47.8361367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:47.8361948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:47.8362750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:47.8363479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:47.8469117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:47.9375473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:49.0930724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22ovubki
2022-05-18T05:16:49.0931355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22ovubki/_remote_module_non_scriptable.py
2022-05-18T05:16:49.2019303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1aw1p12
2022-05-18T05:16:49.2020180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1aw1p12/_remote_module_non_scriptable.py
2022-05-18T05:16:49.3859670Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:16:49.3865169Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T05:16:49.6827961Z ok (4.649s)
2022-05-18T05:16:49.6828300Z 
2022-05-18T05:16:49.6829084Z ----------------------------------------------------------------------
2022-05-18T05:16:49.6829676Z Ran 1 test in 4.649s
2022-05-18T05:16:49.6829846Z 
2022-05-18T05:16:49.6829924Z OK
2022-05-18T05:16:49.6830079Z 
2022-05-18T05:16:49.6830215Z Generating XML reports...
2022-05-18T05:16:49.6871663Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051645.xml
2022-05-18T05:16:51.1027719Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:16:51.1041969Z 
2022-05-18T05:16:51.1042474Z Running tests...
2022-05-18T05:16:51.1042953Z ----------------------------------------------------------------------
2022-05-18T05:16:52.6822857Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:16:52.7225397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91792
2022-05-18T05:16:52.7335481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91793
2022-05-18T05:16:53.8904367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:16:53.8904929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:16:53.8905721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:53.8906421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:16:53.9013978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:16:53.9915134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:16:54.0123688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:16:54.0124210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:16:54.0124929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:16:54.0125609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:16:54.0333209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:16:54.0333724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:16:54.0334407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:16:54.0335081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:16:55.3032894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpknljlhtt
2022-05-18T05:16:55.3033731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpknljlhtt/_remote_module_non_scriptable.py
2022-05-18T05:16:55.3364036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphecgnnyr
2022-05-18T05:16:55.3366623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphecgnnyr/_remote_module_non_scriptable.py
2022-05-18T05:17:00.6500852Z ok (9.545s)
2022-05-18T05:17:00.6501077Z 
2022-05-18T05:17:00.6501460Z ----------------------------------------------------------------------
2022-05-18T05:17:00.6501806Z Ran 1 test in 9.546s
2022-05-18T05:17:00.6501972Z 
2022-05-18T05:17:00.6502071Z OK
2022-05-18T05:17:00.6502206Z 
2022-05-18T05:17:00.6502341Z Generating XML reports...
2022-05-18T05:17:00.6543032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051651.xml
2022-05-18T05:17:02.0916040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T05:17:02.0931117Z 
2022-05-18T05:17:02.0931399Z Running tests...
2022-05-18T05:17:02.0931891Z ----------------------------------------------------------------------
2022-05-18T05:17:03.6873660Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:03.7281973Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91915
2022-05-18T05:17:03.7392688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91916
2022-05-18T05:17:04.8969702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:04.8970284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:04.8971368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:04.8972069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:04.9077598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:04.9979916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:05.0090736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T05:17:05.0091245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T05:17:05.0091949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:17:05.0092651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T05:17:05.0301422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T05:17:05.0301930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T05:17:05.0302620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:17:05.0303308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T05:17:06.2896208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97kqgqyj
2022-05-18T05:17:06.2897384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97kqgqyj/_remote_module_non_scriptable.py
2022-05-18T05:17:06.3395304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1arakrq
2022-05-18T05:17:06.3396545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1arakrq/_remote_module_non_scriptable.py
2022-05-18T05:17:11.6558388Z ok (9.562s)
2022-05-18T05:17:11.6558636Z 
2022-05-18T05:17:11.6559448Z ----------------------------------------------------------------------
2022-05-18T05:17:11.6560070Z Ran 1 test in 9.563s
2022-05-18T05:17:11.6560270Z 
2022-05-18T05:17:11.6560370Z OK
2022-05-18T05:17:11.6560513Z 
2022-05-18T05:17:11.6560649Z Generating XML reports...
2022-05-18T05:17:11.6599508Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051702.xml
2022-05-18T05:17:12.0629061Z Running distributed/fsdp/test_fsdp_state_dict ... [2022-05-18 05:17:12.062319]
2022-05-18T05:17:12.0629793Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:17:12.062418]
2022-05-18T05:17:13.0000988Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict
2022-05-18T05:17:13.0025733Z 
2022-05-18T05:17:13.0026194Z Running tests...
2022-05-18T05:17:13.0026705Z ----------------------------------------------------------------------
2022-05-18T05:17:13.0048015Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:17:14.5948749Z Tests that we can save a state_dict and load it into a blank model ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:17:14.6357014Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92038
2022-05-18T05:17:14.6472549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92039
2022-05-18T05:17:15.5830798Z dist init r=1, world=2
2022-05-18T05:17:15.5832777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:15.5984690Z dist init r=0, world=2
2022-05-18T05:17:15.5988981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:15.5989882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:15.6037651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:16.9568440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:16.9569479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:17.0050847Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:17:17.0052895Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:17:17.3538994Z ok (4.351s)
2022-05-18T05:17:17.3559768Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:17:17.3696113Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92117
2022-05-18T05:17:17.3811344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92118
2022-05-18T05:17:18.3332583Z dist init r=0, world=2
2022-05-18T05:17:18.3335557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:18.3605743Z dist init r=1, world=2
2022-05-18T05:17:18.3609971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:18.3611035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:18.3641403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:19.7025705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:19.7026256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:19.9870777Z ok (2.633s)
2022-05-18T05:17:19.9892584Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:17:20.0021593Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92196
2022-05-18T05:17:20.0136601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92197
2022-05-18T05:17:20.9409924Z dist init r=0, world=2
2022-05-18T05:17:20.9413735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:20.9460003Z dist init r=1, world=2
2022-05-18T05:17:20.9464661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:20.9465847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:20.9515855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:22.2784209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:22.2784764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:22.6196511Z ok (2.632s)
2022-05-18T05:17:22.6217574Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:17:22.6346467Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92275
2022-05-18T05:17:22.6457030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92276
2022-05-18T05:17:23.5396596Z dist init r=1, world=2
2022-05-18T05:17:23.5399937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:23.5748704Z dist init r=0, world=2
2022-05-18T05:17:23.5753433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:23.5754243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:23.5806440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:24.8818684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:24.8819227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:25.1515221Z ok (2.532s)
2022-05-18T05:17:25.1536212Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:17:25.1665268Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92354
2022-05-18T05:17:25.1777672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92355
2022-05-18T05:17:26.1314505Z dist init r=1, world=2
2022-05-18T05:17:26.1318860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:26.1741353Z dist init r=0, world=2
2022-05-18T05:17:26.1745664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:26.1746457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:26.1826215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:27.5413984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:27.5414543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:27.5886061Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:17:27.5887453Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:17:27.8836789Z ok (2.732s)
2022-05-18T05:17:27.8858317Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:17:27.8991726Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92433
2022-05-18T05:17:27.9106533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92434
2022-05-18T05:17:28.8725807Z dist init r=1, world=2
2022-05-18T05:17:28.8729479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:28.8733675Z dist init r=0, world=2
2022-05-18T05:17:28.8738017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:28.8739269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:28.8832251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:30.2273191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:30.2273735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:30.5164479Z ok (2.633s)
2022-05-18T05:17:30.5184942Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:17:30.5314767Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92512
2022-05-18T05:17:30.5426093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92513
2022-05-18T05:17:31.4752260Z dist init r=0, world=2
2022-05-18T05:17:31.4755529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:31.4805643Z dist init r=1, world=2
2022-05-18T05:17:31.4810384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:31.4811230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:31.4858819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:32.8339844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:32.8340381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:33.1487071Z ok (2.632s)
2022-05-18T05:17:33.1507449Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:17:33.1637107Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92591
2022-05-18T05:17:33.1750211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92592
2022-05-18T05:17:34.1203756Z dist init r=0, world=2
2022-05-18T05:17:34.1207011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:34.1426699Z dist init r=1, world=2
2022-05-18T05:17:34.1431584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:34.1432410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:34.1512407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:35.4921977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:35.4922502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:35.7817374Z ok (2.633s)
2022-05-18T05:17:35.7837892Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:17:35.7967030Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92670
2022-05-18T05:17:35.8076273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92671
2022-05-18T05:17:36.7245523Z dist init r=1, world=2
2022-05-18T05:17:36.7248328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:36.7584509Z dist init r=0, world=2
2022-05-18T05:17:36.7588904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:36.7589710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:36.7655029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:38.1050661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:38.1051196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:38.1554693Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:17:38.1556651Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:17:38.4137221Z ok (2.632s)
2022-05-18T05:17:38.4159092Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:17:38.4286405Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92749
2022-05-18T05:17:38.4395744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92750
2022-05-18T05:17:39.3767861Z dist init r=0, world=2
2022-05-18T05:17:39.3771018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:39.4017733Z dist init r=1, world=2
2022-05-18T05:17:39.4022471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:39.4023538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:39.4076597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:40.7394477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:40.7395027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:41.0455388Z ok (2.632s)
2022-05-18T05:17:41.0476119Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:17:41.0605776Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92828
2022-05-18T05:17:41.0714856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92829
2022-05-18T05:17:42.0259770Z dist init r=0, world=2
2022-05-18T05:17:42.0262624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:42.0367634Z dist init r=1, world=2
2022-05-18T05:17:42.0372340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:42.0373360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:42.0466730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:43.3747612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:43.3748156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:43.6773461Z ok (2.632s)
2022-05-18T05:17:43.6793701Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:17:43.6922010Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92907
2022-05-18T05:17:43.7031831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92908
2022-05-18T05:17:44.6651934Z dist init r=0, world=2
2022-05-18T05:17:44.6654225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:44.6711258Z dist init r=1, world=2
2022-05-18T05:17:44.6715628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:44.6716774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:44.6757029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:45.9832789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:45.9833673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:46.3100929Z ok (2.633s)
2022-05-18T05:17:46.3121620Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:17:46.3249626Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92986
2022-05-18T05:17:46.3360606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92987
2022-05-18T05:17:47.2524513Z dist init r=1, world=2
2022-05-18T05:17:47.2527711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:47.2787019Z dist init r=0, world=2
2022-05-18T05:17:47.2791357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:47.2792212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:47.2833095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:48.6185645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:48.6186232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:48.6687831Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:17:48.6689816Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:17:48.9419561Z ok (2.632s)
2022-05-18T05:17:48.9440282Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:17:48.9568440Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93065
2022-05-18T05:17:48.9678937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93066
2022-05-18T05:17:49.9031056Z dist init r=0, world=2
2022-05-18T05:17:49.9033878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:49.9147974Z dist init r=1, world=2
2022-05-18T05:17:49.9152365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:49.9153407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:49.9238092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:51.2555978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:51.2556497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:51.5747922Z ok (2.633s)
2022-05-18T05:17:51.5769024Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:17:51.5896500Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93144
2022-05-18T05:17:51.6005252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93145
2022-05-18T05:17:52.5163503Z dist init r=0, world=2
2022-05-18T05:17:52.5166662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:52.5425522Z dist init r=1, world=2
2022-05-18T05:17:52.5430274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:52.5431097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:52.5471921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:53.8652494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:53.8653037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:54.2064521Z ok (2.631s)
2022-05-18T05:17:54.2085261Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:17:54.2213254Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93223
2022-05-18T05:17:54.2324093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93224
2022-05-18T05:17:55.1326989Z dist init r=1, world=2
2022-05-18T05:17:55.1330492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:55.1450328Z dist init r=0, world=2
2022-05-18T05:17:55.1454958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:55.1455796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:55.1534265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:56.4940517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:56.4941028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:56.7380587Z ok (2.531s)
2022-05-18T05:17:56.7401118Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:17:56.7531687Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93302
2022-05-18T05:17:56.7640342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93303
2022-05-18T05:17:57.6854155Z dist init r=1, world=2
2022-05-18T05:17:57.6857393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:17:57.7329690Z dist init r=0, world=2
2022-05-18T05:17:57.7334413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:17:57.7335237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:57.7365086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:17:59.0717471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:17:59.0718003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:17:59.4699198Z ok (2.732s)
2022-05-18T05:17:59.4720018Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:17:59.4847839Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93381
2022-05-18T05:17:59.4956298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93382
2022-05-18T05:18:00.4449914Z dist init r=0, world=2
2022-05-18T05:18:00.4453675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:00.4586102Z dist init r=1, world=2
2022-05-18T05:18:00.4590404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:00.4592880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:00.4657248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:01.8116228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:01.8116798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:02.1014742Z ok (2.631s)
2022-05-18T05:18:02.1035338Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:18:02.1166492Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93460
2022-05-18T05:18:02.1278083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93461
2022-05-18T05:18:03.0488728Z dist init r=0, world=2
2022-05-18T05:18:03.0491801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:03.0947463Z dist init r=1, world=2
2022-05-18T05:18:03.0951851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:03.0952915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:03.0999617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:04.4452113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:04.4452652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:04.8337994Z ok (2.732s)
2022-05-18T05:18:04.8358118Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:18:04.8489220Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93539
2022-05-18T05:18:04.8604437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93540
2022-05-18T05:18:05.7727273Z dist init r=1, world=2
2022-05-18T05:18:05.7730448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:05.7764209Z dist init r=0, world=2
2022-05-18T05:18:05.7768606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:05.7769730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:05.7833137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:07.1057307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:07.1057867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:07.1583284Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:18:07.4661403Z ok (2.632s)
2022-05-18T05:18:07.4682641Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:18:07.4811254Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93618
2022-05-18T05:18:07.4920335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93619
2022-05-18T05:18:08.4126868Z dist init r=0, world=2
2022-05-18T05:18:08.4129841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:08.4183613Z dist init r=1, world=2
2022-05-18T05:18:08.4188055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:08.4189004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:08.4232643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:09.7373414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:09.7373941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:10.0978128Z ok (2.631s)
2022-05-18T05:18:10.0999095Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:18:10.1128861Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93697
2022-05-18T05:18:10.1238684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93698
2022-05-18T05:18:11.0640161Z dist init r=0, world=2
2022-05-18T05:18:11.0643286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:11.0696381Z dist init r=1, world=2
2022-05-18T05:18:11.0700860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:11.0701658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:11.0746137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:12.3985261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:12.3985794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:12.4528565Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:18:12.7298468Z ok (2.632s)
2022-05-18T05:18:12.7318778Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:18:12.7448150Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93776
2022-05-18T05:18:12.7556848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93777
2022-05-18T05:18:13.6748079Z dist init r=1, world=2
2022-05-18T05:18:13.6751241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:13.6831011Z dist init r=0, world=2
2022-05-18T05:18:13.6835713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:13.6836642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:13.6853576Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:15.0562756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:15.0563293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:15.3614649Z ok (2.631s)
2022-05-18T05:18:15.3635255Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:18:15.3762434Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93855
2022-05-18T05:18:15.3874152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93856
2022-05-18T05:18:16.3076288Z dist init r=0, world=2
2022-05-18T05:18:16.3079645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:16.3300229Z dist init r=1, world=2
2022-05-18T05:18:16.3304444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:16.3305278Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:16.3384776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:17.6975565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:17.6976090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:17.7492222Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:18:17.9932454Z ok (2.632s)
2022-05-18T05:18:18.0071070Z   test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93934
2022-05-18T05:18:18.0181571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93935
2022-05-18T05:18:18.9432206Z dist init r=0, world=2
2022-05-18T05:18:18.9435143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:18.9467021Z dist init r=1, world=2
2022-05-18T05:18:18.9471624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:18.9472625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:18.9537832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:20.3016227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:20.3017100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:20.3223883Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:18:20.3224487Z   warnings.warn(
2022-05-18T05:18:20.3225271Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:18:20.3225825Z   warnings.warn(
2022-05-18T05:18:20.6241665Z ok (2.631s)
2022-05-18T05:18:20.6382143Z   test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94013
2022-05-18T05:18:20.6492336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94014
2022-05-18T05:18:21.6054266Z dist init r=0, world=2
2022-05-18T05:18:21.6057695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:21.6344781Z dist init r=1, world=2
2022-05-18T05:18:21.6349057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:21.6349893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:21.6362587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:22.9645927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:22.9861280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:22.9862324Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:18:22.9862903Z   warnings.warn(
2022-05-18T05:18:22.9863672Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:18:22.9864216Z   warnings.warn(
2022-05-18T05:18:23.2551480Z ok (2.631s)
2022-05-18T05:18:23.2692481Z   test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94092
2022-05-18T05:18:23.2801875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94093
2022-05-18T05:18:24.1948984Z dist init r=0, world=2
2022-05-18T05:18:24.1952006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:24.2019511Z dist init r=1, world=2
2022-05-18T05:18:24.2023811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:24.2024877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:24.2054157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:25.5455186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:25.5455752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:25.5663222Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:18:25.5663916Z   warnings.warn(
2022-05-18T05:18:25.5664696Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:18:25.5665227Z   warnings.warn(
2022-05-18T05:18:25.8860230Z ok (2.631s)
2022-05-18T05:18:25.9001327Z   test_fsdp_state_dict_with_activation_checkpoint_checkpoint_wrap_both (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94171
2022-05-18T05:18:25.9113776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94172
2022-05-18T05:18:26.8265992Z dist init r=0, world=2
2022-05-18T05:18:26.8268922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:26.8680140Z dist init r=1, world=2
2022-05-18T05:18:26.8684746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:26.8685544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:26.8776880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:28.2237912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:28.2239005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:28.5170778Z ok (2.631s)
2022-05-18T05:18:28.5309458Z   test_fsdp_state_dict_with_activation_checkpoint_checkpoint_wrap_first (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94250
2022-05-18T05:18:28.5420347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94251
2022-05-18T05:18:29.4617274Z dist init r=1, world=2
2022-05-18T05:18:29.4620337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:29.4663531Z dist init r=0, world=2
2022-05-18T05:18:29.4668673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:29.4669485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:29.4722758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:30.8215578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:30.8216202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:31.1478885Z ok (2.631s)
2022-05-18T05:18:31.1618434Z   test_fsdp_state_dict_with_activation_checkpoint_checkpoint_wrap_second (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94329
2022-05-18T05:18:31.1727960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94330
2022-05-18T05:18:32.0926755Z dist init r=1, world=2
2022-05-18T05:18:32.0929613Z dist init r=0, world=2
2022-05-18T05:18:32.0930022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:32.0933501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:32.0934632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:32.1032669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:33.4115902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:33.4116753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:33.7787588Z ok (2.631s)
2022-05-18T05:18:33.7929723Z   test_load_activation_checkpointed_module (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94408
2022-05-18T05:18:33.8039241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94409
2022-05-18T05:18:34.7806552Z dist init r=0, world=2
2022-05-18T05:18:34.7809916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:34.8016680Z dist init r=1, world=2
2022-05-18T05:18:34.8021140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:34.8022196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:34.8114955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:36.1394315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:36.1394870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:36.1645161Z <class 'torch.distributed.algorithms._checkpoint.checkpoint_wrapper.CheckpointWrapper'>
2022-05-18T05:18:36.1676591Z <class 'torch.distributed.algorithms._checkpoint.checkpoint_wrapper.CheckpointWrapper'>
2022-05-18T05:18:36.4096415Z ok (2.631s)
2022-05-18T05:18:36.4123708Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:18:36.4251896Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94487
2022-05-18T05:18:36.4363575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94488
2022-05-18T05:18:37.3885882Z dist init r=1, world=2
2022-05-18T05:18:37.3888434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:37.4000173Z dist init r=0, world=2
2022-05-18T05:18:37.4004894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:37.4005972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:37.4092313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:38.7626378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:38.7626906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:39.2424924Z ok (2.833s)
2022-05-18T05:18:39.2452128Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:18:39.2585346Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94570
2022-05-18T05:18:39.2695221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94571
2022-05-18T05:18:40.1815479Z dist init r=0, world=2
2022-05-18T05:18:40.1818507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:40.2031920Z dist init r=1, world=2
2022-05-18T05:18:40.2036121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:40.2036930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:40.2124537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:41.5498980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:41.5499519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:41.8754441Z ok (2.633s)
2022-05-18T05:18:41.8781200Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:18:41.8909074Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94649
2022-05-18T05:18:41.9019334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94650
2022-05-18T05:18:42.8186256Z dist init r=1, world=2
2022-05-18T05:18:42.8189207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:42.8776374Z dist init r=0, world=2
2022-05-18T05:18:42.8781157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:42.8781961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:42.8797555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:44.2257391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:44.2258329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:44.8084615Z ok (2.933s)
2022-05-18T05:18:44.8110759Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:18:44.8238756Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94732
2022-05-18T05:18:44.8348226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94733
2022-05-18T05:18:45.7292799Z dist init r=0, world=2
2022-05-18T05:18:45.7295708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:45.7410808Z dist init r=1, world=2
2022-05-18T05:18:45.7415513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:45.7416607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:45.7500242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:47.0816847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:47.0817401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:47.3405760Z ok (2.532s)
2022-05-18T05:18:47.3432882Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:18:47.3563052Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94811
2022-05-18T05:18:47.3671743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94812
2022-05-18T05:18:48.2698501Z dist init r=0, world=2
2022-05-18T05:18:48.2701484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:48.2776077Z dist init r=1, world=2
2022-05-18T05:18:48.2780502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:48.2781342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:48.2803987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:49.6092492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:49.6094589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:50.1733571Z ok (2.833s)
2022-05-18T05:18:50.1760228Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:18:50.1889726Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94894
2022-05-18T05:18:50.2001371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94895
2022-05-18T05:18:51.0973385Z dist init r=0, world=2
2022-05-18T05:18:51.0976043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:51.1122553Z dist init r=1, world=2
2022-05-18T05:18:51.1126972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:51.1128213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:51.1180381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:52.4553313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:52.4554237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:52.7061346Z ok (2.533s)
2022-05-18T05:18:52.7087928Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:18:52.7215468Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94973
2022-05-18T05:18:52.7324678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94974
2022-05-18T05:18:53.6607039Z dist init r=0, world=2
2022-05-18T05:18:53.6610019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:53.6713040Z dist init r=1, world=2
2022-05-18T05:18:53.6717300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:53.6718809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:53.6814739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:55.0056823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:55.0057354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:55.5388556Z ok (2.833s)
2022-05-18T05:18:55.5415606Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:18:55.5543384Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95056
2022-05-18T05:18:55.5653009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95057
2022-05-18T05:18:56.5379707Z dist init r=0, world=2
2022-05-18T05:18:56.5382952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:56.5597471Z dist init r=1, world=2
2022-05-18T05:18:56.5601807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:56.5602639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:56.5688076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:57.9017290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:18:57.9017838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:18:58.1713292Z ok (2.632s)
2022-05-18T05:18:58.1739612Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:18:58.1865872Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95135
2022-05-18T05:18:58.1975706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95136
2022-05-18T05:18:59.1279503Z dist init r=0, world=2
2022-05-18T05:18:59.1282920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:18:59.1706348Z dist init r=1, world=2
2022-05-18T05:18:59.1710775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:18:59.1712092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:18:59.1790989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:00.4955088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:00.4955639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:01.0037149Z ok (2.832s)
2022-05-18T05:19:01.0063890Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:19:01.0191371Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95218
2022-05-18T05:19:01.0301085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95219
2022-05-18T05:19:01.9531422Z dist init r=0, world=2
2022-05-18T05:19:01.9534742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:01.9628141Z dist init r=1, world=2
2022-05-18T05:19:01.9632296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:01.9633364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:01.9637402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:03.3453840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:03.3454385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:03.8364844Z ok (2.833s)
2022-05-18T05:19:03.8390720Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:19:03.8518114Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95301
2022-05-18T05:19:03.8628531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95302
2022-05-18T05:19:04.7909422Z dist init r=0, world=2
2022-05-18T05:19:04.7912456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:04.8412942Z dist init r=1, world=2
2022-05-18T05:19:04.8417355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:04.8418197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:04.8419425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:06.1741785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:06.1742324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:06.6690637Z ok (2.832s)
2022-05-18T05:19:06.6716612Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:19:06.6847165Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95384
2022-05-18T05:19:06.6955383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95385
2022-05-18T05:19:07.6119694Z dist init r=0, world=2
2022-05-18T05:19:07.6122711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:07.6148379Z dist init r=1, world=2
2022-05-18T05:19:07.6152952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:07.6154171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:07.6225523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:08.9515454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:08.9515995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:09.5018594Z ok (2.833s)
2022-05-18T05:19:09.5045230Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:19:09.5181203Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95467
2022-05-18T05:19:09.5289943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95468
2022-05-18T05:19:10.4636461Z dist init r=0, world=2
2022-05-18T05:19:10.4639849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:10.4669784Z dist init r=1, world=2
2022-05-18T05:19:10.4674214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:10.4675429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:10.4742623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:11.8007053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:11.8007608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:11.8222616Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:11.8223488Z   warnings.warn(
2022-05-18T05:19:11.8256947Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:11.8257488Z   warnings.warn(
2022-05-18T05:19:12.3354005Z ok (2.833s)
2022-05-18T05:19:12.3375725Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:19:12.3515376Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95550
2022-05-18T05:19:12.3624248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95551
2022-05-18T05:19:13.3026840Z dist init r=0, world=2
2022-05-18T05:19:13.3029958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:13.3256257Z dist init r=1, world=2
2022-05-18T05:19:13.3260764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:13.3261575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:13.3335137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:14.6517563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:14.6518438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:14.9683885Z ok (2.633s)
2022-05-18T05:19:14.9710330Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T05:19:14.9847033Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95629
2022-05-18T05:19:14.9956752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95630
2022-05-18T05:19:15.9152298Z dist init r=0, world=2
2022-05-18T05:19:15.9155189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:15.9228083Z dist init r=1, world=2
2022-05-18T05:19:15.9232533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:15.9233500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:15.9257841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:17.2590731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:17.2591342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:17.2782437Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:17.2783032Z   warnings.warn(
2022-05-18T05:19:17.2783794Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:17.2784966Z   warnings.warn(
2022-05-18T05:19:17.8019913Z ok (2.833s)
2022-05-18T05:19:17.8040444Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T05:19:17.8169522Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95712
2022-05-18T05:19:17.8278294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95713
2022-05-18T05:19:18.7775300Z dist init r=0, world=2
2022-05-18T05:19:18.7778457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:18.7989595Z dist init r=1, world=2
2022-05-18T05:19:18.7994131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:18.7994938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:18.8083759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:20.1449893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:20.1450474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:20.1702538Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:20.1703268Z   warnings.warn(
2022-05-18T05:19:20.1704040Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:20.1704923Z   warnings.warn(
2022-05-18T05:19:20.6341271Z ok (2.832s)
2022-05-18T05:19:20.6489928Z   test_state_dict_rank0_offload_save_load_flow (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95795
2022-05-18T05:19:20.6599250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95796
2022-05-18T05:19:21.5739732Z dist init r=0, world=2
2022-05-18T05:19:21.5743286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:21.6091447Z dist init r=1, world=2
2022-05-18T05:19:21.6096152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:21.6097437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:21.6150412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:22.9595892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:22.9596421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:22.9924008Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:22.9924589Z   warnings.warn(
2022-05-18T05:19:22.9925355Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:22.9925885Z   warnings.warn(
2022-05-18T05:19:23.4660999Z ok (2.832s)
2022-05-18T05:19:23.4797725Z   test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95874
2022-05-18T05:19:23.4908500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95875
2022-05-18T05:19:24.4011368Z dist init r=1, world=2
2022-05-18T05:19:24.4014811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:24.4105922Z dist init r=0, world=2
2022-05-18T05:19:24.4110667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:24.4111488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:24.4116985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:25.7638521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:25.7639074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:25.7863907Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:25.7864491Z   warnings.warn(
2022-05-18T05:19:25.7865257Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:25.7865784Z   warnings.warn(
2022-05-18T05:19:25.9962764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:25.9963495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:25.9990748Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:19:25.9992393Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:19:26.2975161Z ok (2.831s)
2022-05-18T05:19:26.3110843Z   test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95957
2022-05-18T05:19:26.3224281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95958
2022-05-18T05:19:27.2411334Z dist init r=0, world=2
2022-05-18T05:19:27.2414347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:27.2918263Z dist init r=1, world=2
2022-05-18T05:19:27.2923003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:27.2923907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:27.3023180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:28.6421181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:28.6421720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:28.6620467Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:28.6621053Z   warnings.warn(
2022-05-18T05:19:28.6622103Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:28.6622669Z   warnings.warn(
2022-05-18T05:19:28.8723528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:28.8726156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:29.1286633Z ok (2.831s)
2022-05-18T05:19:29.1420916Z   test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96040
2022-05-18T05:19:29.1530959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96041
2022-05-18T05:19:30.0597458Z dist init r=0, world=2
2022-05-18T05:19:30.0600825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:30.1068176Z dist init r=1, world=2
2022-05-18T05:19:30.1072677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:30.1074046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:30.1108391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:31.4396315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:31.4396859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:31.4620568Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:31.4621491Z   warnings.warn(
2022-05-18T05:19:31.4656367Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:31.4656917Z   warnings.warn(
2022-05-18T05:19:31.6685210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:31.6685744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:19:31.9595952Z ok (2.831s)
2022-05-18T05:19:31.9754836Z   test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96123
2022-05-18T05:19:31.9865366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96124
2022-05-18T05:19:32.8965976Z dist init r=0, world=2
2022-05-18T05:19:32.8969117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:32.9048554Z dist init r=1, world=2
2022-05-18T05:19:32.9052823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:32.9053925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:32.9071631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:34.2495474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:34.2496484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:34.7934295Z ok (2.834s)
2022-05-18T05:19:34.8093162Z   test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96206
2022-05-18T05:19:34.8203616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96207
2022-05-18T05:19:35.7335985Z dist init r=1, world=2
2022-05-18T05:19:35.7339117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:35.7993886Z dist init r=0, world=2
2022-05-18T05:19:35.7998654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:35.7999521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:35.8049267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:37.1552515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:37.1553110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:37.6268526Z ok (2.833s)
2022-05-18T05:19:37.6428072Z   test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96289
2022-05-18T05:19:37.6536902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96290
2022-05-18T05:19:38.5748916Z dist init r=1, world=2
2022-05-18T05:19:38.5751912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:38.5780919Z dist init r=0, world=2
2022-05-18T05:19:38.5785368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:38.5786692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:38.5854968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:39.9124185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:39.9125071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:40.4602268Z ok (2.833s)
2022-05-18T05:19:40.4738694Z   test_state_dict_type (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96372
2022-05-18T05:19:40.4848444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96373
2022-05-18T05:19:41.3996960Z dist init r=0, world=2
2022-05-18T05:19:41.4000422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:41.4119574Z dist init r=1, world=2
2022-05-18T05:19:41.4124249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:41.4125255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:41.4205253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:42.7645951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:42.7646496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:43.0905912Z ok (2.630s)
2022-05-18T05:19:43.1056558Z   test_state_dict_with_ignored_modules (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96451
2022-05-18T05:19:43.1166036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96452
2022-05-18T05:19:44.0421957Z dist init r=0, world=2
2022-05-18T05:19:44.0424867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:44.0539518Z dist init r=1, world=2
2022-05-18T05:19:44.0543864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:44.0544739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:44.0629266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:45.3719774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:45.3720288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:45.3943423Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:45.3944039Z   warnings.warn(
2022-05-18T05:19:45.3944811Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:45.3945363Z   warnings.warn(
2022-05-18T05:19:45.7225640Z ok (2.632s)
2022-05-18T05:19:45.7361335Z   test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96530
2022-05-18T05:19:45.7472764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96531
2022-05-18T05:19:46.6560588Z dist init r=1, world=2
2022-05-18T05:19:46.6563972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:46.6671209Z dist init r=0, world=2
2022-05-18T05:19:46.6676066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:46.6676896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:46.6768071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:48.0221467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:48.0221983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:48.0420273Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:48.0420875Z   warnings.warn(
2022-05-18T05:19:48.0421646Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:19:48.0422188Z   warnings.warn(
2022-05-18T05:19:48.3530856Z ok (2.630s)
2022-05-18T05:19:48.3531383Z 
2022-05-18T05:19:48.3531818Z ----------------------------------------------------------------------
2022-05-18T05:19:48.3532170Z Ran 57 tests in 155.351s
2022-05-18T05:19:48.3532349Z 
2022-05-18T05:19:48.3532430Z OK
2022-05-18T05:19:48.3532568Z 
2022-05-18T05:19:48.3532702Z Generating XML reports...
2022-05-18T05:19:48.3634441Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20220518051712.xml
2022-05-18T05:19:48.6333106Z Running distributed/fsdp/test_fsdp_optim_state ... [2022-05-18 05:19:48.632819]
2022-05-18T05:19:48.6333901Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:19:48.632927]
2022-05-18T05:19:49.5431513Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state
2022-05-18T05:19:49.5451858Z 
2022-05-18T05:19:49.5452183Z Running tests...
2022-05-18T05:19:49.5452664Z ----------------------------------------------------------------------
2022-05-18T05:19:49.5468571Z   test_full_optim_state_dict_nested_use_multiple_param_groups_False_rank0_only_False (__main__.TestFSDPOptimState)
2022-05-18T05:19:51.1291575Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:19:51.1689733Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96644
2022-05-18T05:19:51.1802844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96645
2022-05-18T05:19:52.0903079Z dist init r=0, world=2
2022-05-18T05:19:52.0906733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:52.1059914Z dist init r=1, world=2
2022-05-18T05:19:52.1063907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:52.1064985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:52.1111125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:53.4675152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:53.4675783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:54.0871779Z ok (4.542s)
2022-05-18T05:19:54.0889017Z   test_full_optim_state_dict_nested_use_multiple_param_groups_False_rank0_only_True (__main__.TestFSDPOptimState)
2022-05-18T05:19:54.1018251Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96727
2022-05-18T05:19:54.1128506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96728
2022-05-18T05:19:55.0265613Z dist init r=1, world=2
2022-05-18T05:19:55.0269166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:55.0570880Z dist init r=0, world=2
2022-05-18T05:19:55.0575244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:55.0576072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:55.0675776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:56.3820212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:56.3820733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:56.9200503Z ok (2.833s)
2022-05-18T05:19:56.9216854Z   test_full_optim_state_dict_nested_use_multiple_param_groups_True_rank0_only_False (__main__.TestFSDPOptimState)
2022-05-18T05:19:56.9346588Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96810
2022-05-18T05:19:56.9455889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96811
2022-05-18T05:19:57.8703116Z dist init r=1, world=2
2022-05-18T05:19:57.8706040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:19:57.9077897Z dist init r=0, world=2
2022-05-18T05:19:57.9082654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:19:57.9083480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:57.9112535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:19:59.2451250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:19:59.2451789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:19:59.8519887Z ok (2.932s)
2022-05-18T05:19:59.8536884Z   test_full_optim_state_dict_nested_use_multiple_param_groups_True_rank0_only_True (__main__.TestFSDPOptimState)
2022-05-18T05:19:59.8663882Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96893
2022-05-18T05:19:59.8773843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96894
2022-05-18T05:20:00.8361566Z dist init r=0, world=2
2022-05-18T05:20:00.8364950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:00.8416794Z dist init r=1, world=2
2022-05-18T05:20:00.8420829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:00.8422021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:00.8467937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:02.2081549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:02.2082103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:02.7838799Z ok (2.932s)
2022-05-18T05:20:02.7852830Z   test_rekey_optim_state_dict_to_ids_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2022-05-18T05:20:02.7984772Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96976
2022-05-18T05:20:02.8096441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96977
2022-05-18T05:20:03.7206877Z dist init r=0, world=2
2022-05-18T05:20:03.7209903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:03.7548880Z dist init r=1, world=2
2022-05-18T05:20:03.7553117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:03.7553934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:03.7616633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:05.1147074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:05.1147673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:05.7159763Z ok (2.932s)
2022-05-18T05:20:05.7174353Z   test_rekey_optim_state_dict_to_ids_use_multiple_param_groups_True (__main__.TestFSDPOptimState)
2022-05-18T05:20:05.7301538Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97059
2022-05-18T05:20:05.7413218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97060
2022-05-18T05:20:06.6695374Z dist init r=0, world=2
2022-05-18T05:20:06.6698965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:06.6996621Z dist init r=1, world=2
2022-05-18T05:20:06.7001298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:06.7002143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:06.7004616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:08.0413391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:08.0413917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:08.6477410Z ok (2.932s)
2022-05-18T05:20:08.6492653Z   test_rekey_optim_state_dict_to_names_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2022-05-18T05:20:08.6620381Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97142
2022-05-18T05:20:08.6729188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97143
2022-05-18T05:20:09.5887264Z dist init r=0, world=2
2022-05-18T05:20:09.5890364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:09.6114441Z dist init r=1, world=2
2022-05-18T05:20:09.6118672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:09.6119499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:09.6195850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:10.9526506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:10.9527184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:11.5791833Z ok (2.931s)
2022-05-18T05:20:11.5801132Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:20:11.5947362Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97225
2022-05-18T05:20:11.6058278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97226
2022-05-18T05:20:12.5248597Z dist init r=0, world=2
2022-05-18T05:20:12.5251419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:12.5308755Z dist init r=1, world=2
2022-05-18T05:20:12.5312959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:12.5314082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:12.5354396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:13.8914217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:13.8914768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:14.2117715Z skip: Need at least 4 CUDA devices (2.632s)
2022-05-18T05:20:14.2126942Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:20:14.2257390Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97304
2022-05-18T05:20:14.2368596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97305
2022-05-18T05:20:15.1527229Z dist init r=0, world=2
2022-05-18T05:20:15.1529693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:15.1531508Z dist init r=1, world=2
2022-05-18T05:20:15.1535727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:15.1536930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:15.1633093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:16.4793293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:16.4793833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:16.7428153Z skip: Need at least 4 CUDA devices (2.531s)
2022-05-18T05:20:16.7437217Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:20:16.7567833Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97383
2022-05-18T05:20:16.7680622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97384
2022-05-18T05:20:17.6903302Z dist init r=0, world=2
2022-05-18T05:20:17.6906349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:17.7413056Z dist init r=1, world=2
2022-05-18T05:20:17.7418009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:17.7418833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:17.7515063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:19.0670853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:19.0671568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:19.3740034Z skip: Need at least 4 CUDA devices (2.631s)
2022-05-18T05:20:19.3749054Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:20:19.3877946Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97462
2022-05-18T05:20:19.3987987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97463
2022-05-18T05:20:20.3286581Z dist init r=1, world=2
2022-05-18T05:20:20.3289579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:20.3314893Z dist init r=0, world=2
2022-05-18T05:20:20.3319502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:20.3320378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:20.3392170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:21.6899295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:21.6899812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:22.0046835Z skip: Need at least 4 CUDA devices (2.630s)
2022-05-18T05:20:22.0055744Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:20:22.0184178Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97541
2022-05-18T05:20:22.0293239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97542
2022-05-18T05:20:22.9454960Z dist init r=0, world=2
2022-05-18T05:20:22.9458198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:22.9925926Z dist init r=1, world=2
2022-05-18T05:20:22.9930657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:22.9931506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:22.9966078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:24.3344095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:24.3344610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:24.6352782Z skip: Need at least 4 CUDA devices (2.630s)
2022-05-18T05:20:24.6362088Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:20:24.6490701Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97620
2022-05-18T05:20:24.6599343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97621
2022-05-18T05:20:25.6177046Z dist init r=0, world=2
2022-05-18T05:20:25.6180300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:25.6423205Z dist init r=1, world=2
2022-05-18T05:20:25.6427390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:25.6428610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:25.6484971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:26.9937441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:26.9937986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:27.2656757Z skip: Need at least 4 CUDA devices (2.630s)
2022-05-18T05:20:27.2665470Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:20:27.2796349Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97699
2022-05-18T05:20:27.2905971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97700
2022-05-18T05:20:28.2068141Z dist init r=1, world=2
2022-05-18T05:20:28.2071203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:28.2189169Z dist init r=0, world=2
2022-05-18T05:20:28.2193522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:28.2194782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:28.2275027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:29.5705941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:29.5706837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:29.8967592Z skip: Need at least 4 CUDA devices (2.631s)
2022-05-18T05:20:29.8976452Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:20:29.9105681Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97778
2022-05-18T05:20:29.9214151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97779
2022-05-18T05:20:30.8759749Z dist init r=0, world=2
2022-05-18T05:20:30.8763218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:30.8971780Z dist init r=1, world=2
2022-05-18T05:20:30.8976424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:30.8977240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:30.9068086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:32.2276683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:32.2277237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:32.5273337Z skip: Need at least 4 CUDA devices (2.630s)
2022-05-18T05:20:32.5280066Z   test_scatter_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-05-18T05:20:32.5409552Z Tests :meth:`scatter_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97857
2022-05-18T05:20:32.5519066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97858
2022-05-18T05:20:33.4717402Z dist init r=0, world=2
2022-05-18T05:20:33.4720630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:33.5249736Z dist init r=1, world=2
2022-05-18T05:20:33.5255752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:33.5256546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:33.5329450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:34.8616556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:34.8617092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:35.1577392Z skip: Need at least 4 CUDA devices (2.630s)
2022-05-18T05:20:35.1586577Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:20:35.1716037Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97936
2022-05-18T05:20:35.1827077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97937
2022-05-18T05:20:36.1050538Z dist init r=0, world=2
2022-05-18T05:20:36.1053826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:36.1123362Z dist init r=1, world=2
2022-05-18T05:20:36.1128037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:36.1130082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:36.1156163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:37.4507686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:37.4508212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:37.6885749Z skip: Need at least 4 CUDA devices (2.531s)
2022-05-18T05:20:37.6894740Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:20:37.7024055Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98015
2022-05-18T05:20:37.7132391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98016
2022-05-18T05:20:38.6783432Z dist init r=0, world=2
2022-05-18T05:20:38.6786062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:38.6791933Z dist init r=1, world=2
2022-05-18T05:20:38.6796926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:38.6798041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:38.6889052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:40.0211803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:40.0212373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:40.3190846Z skip: Need at least 4 CUDA devices (2.630s)
2022-05-18T05:20:40.3199710Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:20:40.3328395Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98094
2022-05-18T05:20:40.3437415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98095
2022-05-18T05:20:41.3065070Z dist init r=0, world=2
2022-05-18T05:20:41.3068215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:41.3094571Z dist init r=1, world=2
2022-05-18T05:20:41.3099165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:41.3099993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:41.3170998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:42.6553843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:42.6554389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:42.9498073Z skip: Need at least 4 CUDA devices (2.631s)
2022-05-18T05:20:42.9507069Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:20:42.9635865Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98173
2022-05-18T05:20:42.9745480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98174
2022-05-18T05:20:43.9030048Z dist init r=0, world=2
2022-05-18T05:20:43.9033283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:43.9385675Z dist init r=1, world=2
2022-05-18T05:20:43.9390316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:43.9391134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:43.9439972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:45.2987063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:45.2987614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:45.5806268Z skip: Need at least 4 CUDA devices (2.631s)
2022-05-18T05:20:45.5815469Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:20:45.5946339Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98252
2022-05-18T05:20:45.6057310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98253
2022-05-18T05:20:46.5405495Z dist init r=1, world=2
2022-05-18T05:20:46.5408814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:46.5628313Z dist init r=0, world=2
2022-05-18T05:20:46.5633080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:46.5633981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:46.5714373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:47.8966105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:47.8966642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:48.2116973Z skip: Need at least 4 CUDA devices (2.631s)
2022-05-18T05:20:48.2126460Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:20:48.2265011Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98331
2022-05-18T05:20:48.2378491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98332
2022-05-18T05:20:49.1555923Z dist init r=1, world=2
2022-05-18T05:20:49.1559214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:49.1890819Z dist init r=0, world=2
2022-05-18T05:20:49.1895577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:49.1896612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:49.1966120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:50.5366850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:50.5367389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:50.8437617Z skip: Need at least 4 CUDA devices (2.632s)
2022-05-18T05:20:50.8447299Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T05:20:50.8577245Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98410
2022-05-18T05:20:50.8687541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98411
2022-05-18T05:20:51.7933700Z dist init r=0, world=2
2022-05-18T05:20:51.7937094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:51.8048947Z dist init r=1, world=2
2022-05-18T05:20:51.8053579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:51.8054392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:51.8141369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:53.1380106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:53.1380639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:53.4745115Z skip: Need at least 4 CUDA devices (2.631s)
2022-05-18T05:20:53.4753961Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T05:20:53.4883078Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98489
2022-05-18T05:20:53.4991539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98490
2022-05-18T05:20:54.4106901Z dist init r=0, world=2
2022-05-18T05:20:54.4110254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:54.4123377Z dist init r=1, world=2
2022-05-18T05:20:54.4127667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:54.4128715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:54.4213527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:55.7706558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:55.7707092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:56.1048868Z skip: Need at least 4 CUDA devices (2.630s)
2022-05-18T05:20:56.1055266Z   test_shard_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-05-18T05:20:56.1186811Z Tests :meth:`shard_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98568
2022-05-18T05:20:56.1296441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98569
2022-05-18T05:20:57.0555407Z dist init r=0, world=2
2022-05-18T05:20:57.0558674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:57.0762756Z dist init r=1, world=2
2022-05-18T05:20:57.0766854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:57.0767678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:57.0864019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:58.3973902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:20:58.3974458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:20:58.6355392Z skip: Need at least 4 CUDA devices (2.530s)
2022-05-18T05:20:58.6371669Z   test_shard_full_optim_state_dict_unmanaged_params_add_to_fsdp_module_False (__main__.TestFSDPOptimState)
2022-05-18T05:20:58.6502553Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98647
2022-05-18T05:20:58.6610777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98648
2022-05-18T05:20:59.5939241Z dist init r=1, world=2
2022-05-18T05:20:59.5942185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:20:59.6381200Z dist init r=0, world=2
2022-05-18T05:20:59.6385651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:20:59.6386466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:20:59.6450379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:00.9866454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:00.9867339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:01.5673299Z ok (2.932s)
2022-05-18T05:21:01.5691208Z   test_shard_full_optim_state_dict_unmanaged_params_add_to_fsdp_module_True (__main__.TestFSDPOptimState)
2022-05-18T05:21:01.5824596Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98730
2022-05-18T05:21:01.5937486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98731
2022-05-18T05:21:02.5229111Z dist init r=0, world=2
2022-05-18T05:21:02.5232318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:21:02.5579724Z dist init r=1, world=2
2022-05-18T05:21:02.5584331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:21:02.5585441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:02.5639669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:21:03.9287947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:21:03.9288456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:21:04.5002050Z ok (2.933s)
2022-05-18T05:21:04.5002266Z 
2022-05-18T05:21:04.5002647Z ----------------------------------------------------------------------
2022-05-18T05:21:04.5002991Z Ran 27 tests in 74.955s
2022-05-18T05:21:04.5005540Z 
2022-05-18T05:21:04.5005793Z OK (skipped=18)
2022-05-18T05:21:04.5005978Z 
2022-05-18T05:21:04.5006110Z Generating XML reports...
2022-05-18T05:21:04.5081758Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20220518051949.xml
2022-05-18T05:21:04.7781381Z Running distributed/test_store ... [2022-05-18 05:21:04.777558]
2022-05-18T05:21:04.7782116Z Executing ['/opt/conda/bin/python', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:21:04.777670]
2022-05-18T05:21:05.6632473Z test_compare_set (__main__.FileStoreTest)
2022-05-18T05:21:05.6633110Z test_set_get (__main__.FileStoreTest)
2022-05-18T05:21:05.6633683Z test_compare_set (__main__.HashStoreTest)
2022-05-18T05:21:05.6634098Z test_set_get (__main__.HashStoreTest)
2022-05-18T05:21:05.6634419Z test_compare_set (__main__.PrefixFileStoreTest)
2022-05-18T05:21:05.6634768Z test_set_get (__main__.PrefixFileStoreTest)
2022-05-18T05:21:05.6635135Z test_compare_set (__main__.PrefixTCPStoreTest)
2022-05-18T05:21:05.6635471Z test_set_get (__main__.PrefixTCPStoreTest)
2022-05-18T05:21:05.6635772Z test_set_get (__main__.PythonStoreTest)
2022-05-18T05:21:05.6636090Z test_nominal (__main__.RendezvousEnvTest)
2022-05-18T05:21:05.6636432Z test_common_errors (__main__.RendezvousFileTest)
2022-05-18T05:21:05.6637044Z test_nominal (__main__.RendezvousFileTest)
2022-05-18T05:21:05.6637562Z test_common_errors (__main__.RendezvousTCPTest)
2022-05-18T05:21:05.6638835Z test_dns_timeout (__main__.RendezvousTCPTest)
2022-05-18T05:21:05.6639487Z test_nominal (__main__.RendezvousTCPTest)
2022-05-18T05:21:05.6640269Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest)
2022-05-18T05:21:05.6640749Z test_unknown_handler (__main__.RendezvousTest)
2022-05-18T05:21:05.6641074Z test_address_already_in_use (__main__.TCPStoreTest)
2022-05-18T05:21:05.6641407Z test_compare_set (__main__.TCPStoreTest)
2022-05-18T05:21:05.6641771Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest)
2022-05-18T05:21:05.6642167Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest)
2022-05-18T05:21:05.6642534Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest)
2022-05-18T05:21:05.6642888Z test_multitenancy (__main__.TCPStoreTest)
2022-05-18T05:21:05.6643463Z test_numkeys_delkeys (__main__.TCPStoreTest)
2022-05-18T05:21:05.6643776Z test_set_get (__main__.TCPStoreTest)
2022-05-18T05:21:06.5503017Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:06.5517402Z 
2022-05-18T05:21:06.5517698Z Running tests...
2022-05-18T05:21:06.5518479Z ----------------------------------------------------------------------
2022-05-18T05:21:08.1493855Z   test_compare_set (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:08.1681129Z ok (1.616s)
2022-05-18T05:21:08.1682976Z 
2022-05-18T05:21:08.1683588Z ----------------------------------------------------------------------
2022-05-18T05:21:08.1684085Z Ran 1 test in 1.617s
2022-05-18T05:21:08.1684261Z 
2022-05-18T05:21:08.1684358Z OK
2022-05-18T05:21:08.1684494Z 
2022-05-18T05:21:08.1684622Z Generating XML reports...
2022-05-18T05:21:08.1716212Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518052106.xml
2022-05-18T05:21:09.2898869Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:09.2912834Z 
2022-05-18T05:21:09.2913072Z Running tests...
2022-05-18T05:21:09.2913490Z ----------------------------------------------------------------------
2022-05-18T05:21:10.8601849Z   test_set_get (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:10.8772804Z ok (1.586s)
2022-05-18T05:21:10.8774852Z 
2022-05-18T05:21:10.8775552Z ----------------------------------------------------------------------
2022-05-18T05:21:10.8775928Z Ran 1 test in 1.586s
2022-05-18T05:21:10.8776097Z 
2022-05-18T05:21:10.8776429Z OK
2022-05-18T05:21:10.8776568Z 
2022-05-18T05:21:10.8776689Z Generating XML reports...
2022-05-18T05:21:10.8806756Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518052109.xml
2022-05-18T05:21:11.9889425Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:11.9903596Z 
2022-05-18T05:21:11.9903984Z Running tests...
2022-05-18T05:21:11.9904456Z ----------------------------------------------------------------------
2022-05-18T05:21:13.5738382Z   test_compare_set (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:13.5918646Z ok (1.601s)
2022-05-18T05:21:13.5920543Z 
2022-05-18T05:21:13.5921503Z ----------------------------------------------------------------------
2022-05-18T05:21:13.5921868Z Ran 1 test in 1.602s
2022-05-18T05:21:13.5922044Z 
2022-05-18T05:21:13.5922152Z OK
2022-05-18T05:21:13.5922294Z 
2022-05-18T05:21:13.5922429Z Generating XML reports...
2022-05-18T05:21:13.5954627Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518052111.xml
2022-05-18T05:21:14.7120532Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:14.7134468Z 
2022-05-18T05:21:14.7134621Z Running tests...
2022-05-18T05:21:14.7135399Z ----------------------------------------------------------------------
2022-05-18T05:21:16.2835398Z   test_set_get (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:16.2995164Z ok (1.586s)
2022-05-18T05:21:16.2995641Z 
2022-05-18T05:21:16.2996039Z ----------------------------------------------------------------------
2022-05-18T05:21:16.2996376Z Ran 1 test in 1.586s
2022-05-18T05:21:16.2996544Z 
2022-05-18T05:21:16.2996643Z OK
2022-05-18T05:21:16.2996782Z 
2022-05-18T05:21:16.2996921Z Generating XML reports...
2022-05-18T05:21:16.3028291Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518052114.xml
2022-05-18T05:21:17.4266375Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:17.4280442Z 
2022-05-18T05:21:17.4280903Z Running tests...
2022-05-18T05:21:17.4281371Z ----------------------------------------------------------------------
2022-05-18T05:21:19.0247779Z   test_compare_set (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:19.0432663Z ok (1.615s)
2022-05-18T05:21:19.0434877Z 
2022-05-18T05:21:19.0435411Z ----------------------------------------------------------------------
2022-05-18T05:21:19.0435745Z Ran 1 test in 1.615s
2022-05-18T05:21:19.0435923Z 
2022-05-18T05:21:19.0436024Z OK
2022-05-18T05:21:19.0436165Z 
2022-05-18T05:21:19.0436298Z Generating XML reports...
2022-05-18T05:21:19.0467853Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518052117.xml
2022-05-18T05:21:20.1712736Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:20.1726815Z 
2022-05-18T05:21:20.1727036Z Running tests...
2022-05-18T05:21:20.1727488Z ----------------------------------------------------------------------
2022-05-18T05:21:21.7507404Z   test_set_get (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:21.7676902Z ok (1.595s)
2022-05-18T05:21:21.7679042Z 
2022-05-18T05:21:21.7679557Z ----------------------------------------------------------------------
2022-05-18T05:21:21.7679906Z Ran 1 test in 1.595s
2022-05-18T05:21:21.7680061Z 
2022-05-18T05:21:21.7680161Z OK
2022-05-18T05:21:21.7680299Z 
2022-05-18T05:21:21.7680433Z Generating XML reports...
2022-05-18T05:21:21.7710869Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518052120.xml
2022-05-18T05:21:22.8520333Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:22.8535130Z 
2022-05-18T05:21:22.8535761Z Running tests...
2022-05-18T05:21:22.8536526Z ----------------------------------------------------------------------
2022-05-18T05:21:24.4303364Z   test_compare_set (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:24.4493910Z ok (1.596s)
2022-05-18T05:21:24.4496979Z 
2022-05-18T05:21:24.4497550Z ----------------------------------------------------------------------
2022-05-18T05:21:24.4497888Z Ran 1 test in 1.596s
2022-05-18T05:21:24.4498063Z 
2022-05-18T05:21:24.4498162Z OK
2022-05-18T05:21:24.4498300Z 
2022-05-18T05:21:24.4498433Z Generating XML reports...
2022-05-18T05:21:24.4530133Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518052122.xml
2022-05-18T05:21:25.5284443Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:25.5300233Z 
2022-05-18T05:21:25.5300843Z Running tests...
2022-05-18T05:21:25.5301726Z ----------------------------------------------------------------------
2022-05-18T05:21:27.1109155Z   test_set_get (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:27.1305421Z ok (1.600s)
2022-05-18T05:21:27.1308316Z 
2022-05-18T05:21:27.1309107Z ----------------------------------------------------------------------
2022-05-18T05:21:27.1309790Z Ran 1 test in 1.601s
2022-05-18T05:21:27.1310085Z 
2022-05-18T05:21:27.1310316Z OK
2022-05-18T05:21:27.1310576Z 
2022-05-18T05:21:27.1310817Z Generating XML reports...
2022-05-18T05:21:27.1345635Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518052125.xml
2022-05-18T05:21:28.2582901Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:28.2597541Z 
2022-05-18T05:21:28.2598034Z Running tests...
2022-05-18T05:21:28.2598817Z ----------------------------------------------------------------------
2022-05-18T05:21:29.8294761Z   test_set_get (__main__.PythonStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:29.8451146Z ok (1.585s)
2022-05-18T05:21:29.8451881Z 
2022-05-18T05:21:29.8452669Z ----------------------------------------------------------------------
2022-05-18T05:21:29.8453369Z Ran 1 test in 1.585s
2022-05-18T05:21:29.8453704Z 
2022-05-18T05:21:29.8453890Z OK
2022-05-18T05:21:29.8454147Z 
2022-05-18T05:21:29.8454375Z Generating XML reports...
2022-05-18T05:21:29.8488833Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220518052128.xml
2022-05-18T05:21:30.9635942Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:30.9653943Z 
2022-05-18T05:21:30.9654408Z Running tests...
2022-05-18T05:21:30.9655279Z ----------------------------------------------------------------------
2022-05-18T05:21:32.5461513Z   test_nominal (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:32.5629678Z ok (1.597s)
2022-05-18T05:21:32.5631158Z 
2022-05-18T05:21:32.5631942Z ----------------------------------------------------------------------
2022-05-18T05:21:32.5632353Z Ran 1 test in 1.598s
2022-05-18T05:21:32.5632529Z 
2022-05-18T05:21:32.5632638Z OK
2022-05-18T05:21:32.5632779Z 
2022-05-18T05:21:32.5632898Z Generating XML reports...
2022-05-18T05:21:32.5663468Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220518052130.xml
2022-05-18T05:21:33.6811570Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:33.6825210Z 
2022-05-18T05:21:33.6825362Z Running tests...
2022-05-18T05:21:33.6825819Z ----------------------------------------------------------------------
2022-05-18T05:21:35.2609711Z   test_common_errors (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:35.2769842Z ok (1.594s)
2022-05-18T05:21:35.2770750Z 
2022-05-18T05:21:35.2771151Z ----------------------------------------------------------------------
2022-05-18T05:21:35.2771521Z Ran 1 test in 1.595s
2022-05-18T05:21:35.2772017Z 
2022-05-18T05:21:35.2772117Z OK
2022-05-18T05:21:35.2772259Z 
2022-05-18T05:21:35.2772399Z Generating XML reports...
2022-05-18T05:21:35.2804341Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518052133.xml
2022-05-18T05:21:36.3636935Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:36.3652125Z 
2022-05-18T05:21:36.3652446Z Running tests...
2022-05-18T05:21:36.3652896Z ----------------------------------------------------------------------
2022-05-18T05:21:37.9439924Z   test_nominal (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:37.9613176Z ok (1.596s)
2022-05-18T05:21:37.9613516Z 
2022-05-18T05:21:37.9614188Z ----------------------------------------------------------------------
2022-05-18T05:21:37.9614852Z Ran 1 test in 1.596s
2022-05-18T05:21:37.9615150Z 
2022-05-18T05:21:37.9615350Z OK
2022-05-18T05:21:37.9615607Z 
2022-05-18T05:21:37.9615865Z Generating XML reports...
2022-05-18T05:21:37.9649926Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518052136.xml
2022-05-18T05:21:39.0806196Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:39.0822226Z 
2022-05-18T05:21:39.0822704Z Running tests...
2022-05-18T05:21:39.0823233Z ----------------------------------------------------------------------
2022-05-18T05:21:40.6551614Z   test_common_errors (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:40.6707304Z ok (1.588s)
2022-05-18T05:21:40.6708725Z 
2022-05-18T05:21:40.6709646Z ----------------------------------------------------------------------
2022-05-18T05:21:40.6710308Z Ran 1 test in 1.589s
2022-05-18T05:21:40.6710649Z 
2022-05-18T05:21:40.6710832Z OK
2022-05-18T05:21:40.6711092Z 
2022-05-18T05:21:40.6711342Z Generating XML reports...
2022-05-18T05:21:40.6743966Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518052139.xml
2022-05-18T05:21:41.7776782Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:41.7790576Z 
2022-05-18T05:21:41.7790870Z Running tests...
2022-05-18T05:21:41.7791302Z ----------------------------------------------------------------------
2022-05-18T05:21:43.3511852Z   test_dns_timeout (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:43.3845000Z [W socket.cpp:558] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known).
2022-05-18T05:21:43.3845545Z [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456).
2022-05-18T05:21:43.3848678Z ok (1.606s)
2022-05-18T05:21:43.3849729Z 
2022-05-18T05:21:43.3850048Z ----------------------------------------------------------------------
2022-05-18T05:21:43.3850402Z Ran 1 test in 1.606s
2022-05-18T05:21:43.3850597Z 
2022-05-18T05:21:43.3850675Z OK
2022-05-18T05:21:43.3850814Z 
2022-05-18T05:21:43.3850946Z Generating XML reports...
2022-05-18T05:21:43.3881858Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518052141.xml
2022-05-18T05:21:44.4957670Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:44.4971978Z 
2022-05-18T05:21:44.4972295Z Running tests...
2022-05-18T05:21:44.4972752Z ----------------------------------------------------------------------
2022-05-18T05:21:46.0685269Z   test_nominal (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:46.0854368Z ok (1.588s)
2022-05-18T05:21:46.0854699Z 
2022-05-18T05:21:46.0855204Z ----------------------------------------------------------------------
2022-05-18T05:21:46.0855561Z Ran 1 test in 1.588s
2022-05-18T05:21:46.0855737Z 
2022-05-18T05:21:46.0855834Z OK
2022-05-18T05:21:46.0855971Z 
2022-05-18T05:21:46.0856107Z Generating XML reports...
2022-05-18T05:21:46.0888166Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518052144.xml
2022-05-18T05:21:47.2094448Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:47.2108838Z 
2022-05-18T05:21:47.2109147Z Running tests...
2022-05-18T05:21:47.2109627Z ----------------------------------------------------------------------
2022-05-18T05:21:48.7794455Z   test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:21:58.8322426Z ok (11.621s)
2022-05-18T05:21:58.8322645Z 
2022-05-18T05:21:58.8323062Z ----------------------------------------------------------------------
2022-05-18T05:21:58.8323411Z Ran 1 test in 11.621s
2022-05-18T05:21:58.8323560Z 
2022-05-18T05:21:58.8323661Z OK
2022-05-18T05:21:58.8323796Z 
2022-05-18T05:21:58.8323931Z Generating XML reports...
2022-05-18T05:21:58.8358712Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518052147.xml
2022-05-18T05:21:59.9932826Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:21:59.9947716Z 
2022-05-18T05:21:59.9948184Z Running tests...
2022-05-18T05:21:59.9948722Z ----------------------------------------------------------------------
2022-05-18T05:22:01.5811140Z   test_unknown_handler (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:01.5967090Z ok (1.602s)
2022-05-18T05:22:01.5967433Z 
2022-05-18T05:22:01.5968134Z ----------------------------------------------------------------------
2022-05-18T05:22:01.5968783Z Ran 1 test in 1.602s
2022-05-18T05:22:01.5969084Z 
2022-05-18T05:22:01.5969803Z OK
2022-05-18T05:22:01.5969993Z 
2022-05-18T05:22:01.5970137Z Generating XML reports...
2022-05-18T05:22:01.6003379Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220518052159.xml
2022-05-18T05:22:02.7339111Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:22:02.7353280Z 
2022-05-18T05:22:02.7353684Z Running tests...
2022-05-18T05:22:02.7354207Z ----------------------------------------------------------------------
2022-05-18T05:22:04.3169770Z   test_address_already_in_use (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:04.3330726Z [W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:37571 (errno: 98 - Address already in use).
2022-05-18T05:22:04.3350276Z [W socket.cpp:401] [c10d] The server socket has failed to bind to 0.0.0.0:37571 (errno: 98 - Address already in use).
2022-05-18T05:22:04.3350854Z [E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address.
2022-05-18T05:22:04.3355192Z ok (1.600s)
2022-05-18T05:22:04.3356127Z 
2022-05-18T05:22:04.3356658Z ----------------------------------------------------------------------
2022-05-18T05:22:04.3357386Z Ran 1 test in 1.600s
2022-05-18T05:22:04.3357733Z 
2022-05-18T05:22:04.3357890Z OK
2022-05-18T05:22:04.3358031Z 
2022-05-18T05:22:04.3358436Z Generating XML reports...
2022-05-18T05:22:04.3389939Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052202.xml
2022-05-18T05:22:05.4564912Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:22:05.4578924Z 
2022-05-18T05:22:05.4579226Z Running tests...
2022-05-18T05:22:05.4579682Z ----------------------------------------------------------------------
2022-05-18T05:22:07.0300044Z   test_compare_set (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:07.0495911Z ok (1.591s)
2022-05-18T05:22:07.0496270Z 
2022-05-18T05:22:07.0496706Z ----------------------------------------------------------------------
2022-05-18T05:22:07.0497089Z Ran 1 test in 1.592s
2022-05-18T05:22:07.0497258Z 
2022-05-18T05:22:07.0497362Z OK
2022-05-18T05:22:07.0497501Z 
2022-05-18T05:22:07.0497632Z Generating XML reports...
2022-05-18T05:22:07.0529121Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052205.xml
2022-05-18T05:22:08.1676040Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:22:08.1690883Z 
2022-05-18T05:22:08.1691016Z Running tests...
2022-05-18T05:22:08.1691491Z ----------------------------------------------------------------------
2022-05-18T05:22:09.7411399Z   test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:09.7585884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:09.7587652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:22:09.8228624Z ok (1.654s)
2022-05-18T05:22:09.8229028Z 
2022-05-18T05:22:09.8229369Z ----------------------------------------------------------------------
2022-05-18T05:22:09.8229722Z Ran 1 test in 1.654s
2022-05-18T05:22:09.8229887Z 
2022-05-18T05:22:09.8229986Z OK
2022-05-18T05:22:09.8230124Z 
2022-05-18T05:22:09.8230255Z Generating XML reports...
2022-05-18T05:22:09.8264867Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052208.xml
2022-05-18T05:22:10.9599719Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:22:10.9614244Z 
2022-05-18T05:22:10.9614563Z Running tests...
2022-05-18T05:22:10.9615015Z ----------------------------------------------------------------------
2022-05-18T05:22:12.5284993Z   test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:12.5515045Z ok (1.590s)
2022-05-18T05:22:12.5515654Z 
2022-05-18T05:22:12.5516459Z ----------------------------------------------------------------------
2022-05-18T05:22:12.5517174Z Ran 1 test in 1.590s
2022-05-18T05:22:12.5517522Z 
2022-05-18T05:22:12.5517721Z OK
2022-05-18T05:22:12.5517965Z 
2022-05-18T05:22:12.5518621Z Generating XML reports...
2022-05-18T05:22:12.5552246Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052210.xml
2022-05-18T05:22:13.6852000Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:22:13.6865568Z 
2022-05-18T05:22:13.6866116Z Running tests...
2022-05-18T05:22:13.6866626Z ----------------------------------------------------------------------
2022-05-18T05:22:15.2616360Z   test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:15.2795736Z ok (1.593s)
2022-05-18T05:22:15.2796352Z 
2022-05-18T05:22:15.2796712Z ----------------------------------------------------------------------
2022-05-18T05:22:15.2797074Z Ran 1 test in 1.593s
2022-05-18T05:22:15.2797242Z 
2022-05-18T05:22:15.2797342Z OK
2022-05-18T05:22:15.2797482Z 
2022-05-18T05:22:15.2797614Z Generating XML reports...
2022-05-18T05:22:15.2830607Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052213.xml
2022-05-18T05:22:16.3991153Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:22:16.4005025Z 
2022-05-18T05:22:16.4005417Z Running tests...
2022-05-18T05:22:16.4005922Z ----------------------------------------------------------------------
2022-05-18T05:22:17.9857888Z   test_multitenancy (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:18.0032731Z ok (1.603s)
2022-05-18T05:22:18.0033513Z 
2022-05-18T05:22:18.0033918Z ----------------------------------------------------------------------
2022-05-18T05:22:18.0034277Z Ran 1 test in 1.603s
2022-05-18T05:22:18.0034566Z 
2022-05-18T05:22:18.0034715Z OK
2022-05-18T05:22:18.0034925Z 
2022-05-18T05:22:18.0035060Z Generating XML reports...
2022-05-18T05:22:18.0067586Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052216.xml
2022-05-18T05:22:19.1116277Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:22:19.1129888Z 
2022-05-18T05:22:19.1130159Z Running tests...
2022-05-18T05:22:19.1130613Z ----------------------------------------------------------------------
2022-05-18T05:22:20.6577826Z   test_numkeys_delkeys (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:22.7051498Z ok (3.592s)
2022-05-18T05:22:22.7051695Z 
2022-05-18T05:22:22.7052084Z ----------------------------------------------------------------------
2022-05-18T05:22:22.7052445Z Ran 1 test in 3.592s
2022-05-18T05:22:22.7052614Z 
2022-05-18T05:22:22.7052694Z OK
2022-05-18T05:22:22.7052833Z 
2022-05-18T05:22:22.7052969Z Generating XML reports...
2022-05-18T05:22:22.7085409Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052219.xml
2022-05-18T05:22:23.8389853Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T05:22:23.8404031Z 
2022-05-18T05:22:23.8404306Z Running tests...
2022-05-18T05:22:23.8404852Z ----------------------------------------------------------------------
2022-05-18T05:22:25.4237006Z   test_set_get (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:25.4425507Z ok (1.602s)
2022-05-18T05:22:25.4425778Z 
2022-05-18T05:22:25.4426261Z ----------------------------------------------------------------------
2022-05-18T05:22:25.4426641Z Ran 1 test in 1.602s
2022-05-18T05:22:25.4426791Z 
2022-05-18T05:22:25.4426891Z OK
2022-05-18T05:22:25.4427028Z 
2022-05-18T05:22:25.4427160Z Generating XML reports...
2022-05-18T05:22:25.4458764Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052223.xml
2022-05-18T05:22:25.8070530Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2022-05-18 05:22:25.806510]
2022-05-18T05:22:25.8071306Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:22:25.806622]
2022-05-18T05:22:26.7230097Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm
2022-05-18T05:22:26.7248300Z 
2022-05-18T05:22:26.7248610Z Running tests...
2022-05-18T05:22:26.7249580Z ----------------------------------------------------------------------
2022-05-18T05:22:26.7258116Z   test_fsdp_calc_grad_norm_error_norm_type_1_3 (__main__.TestCalcuGradNorm)
2022-05-18T05:22:28.3069351Z Test the abnormal cases of grad norm cal API. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:22:28.3462883Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99811
2022-05-18T05:22:28.3576563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99812
2022-05-18T05:22:29.2641690Z dist init r=1, world=2
2022-05-18T05:22:29.2644924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:29.2822953Z dist init r=0, world=2
2022-05-18T05:22:29.2827311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:29.2828123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:29.2848668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:30.6184227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:30.6184772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:31.0642305Z ok (4.339s)
2022-05-18T05:22:31.0651590Z   test_fsdp_calc_grad_norm_error_norm_type_2_5 (__main__.TestCalcuGradNorm)
2022-05-18T05:22:31.0787689Z Test the abnormal cases of grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99894
2022-05-18T05:22:31.0900596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99895
2022-05-18T05:22:32.0338603Z dist init r=1, world=2
2022-05-18T05:22:32.0341583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:32.0589237Z dist init r=0, world=2
2022-05-18T05:22:32.0594287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:32.0595116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:32.0647045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:33.4044436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:33.4044980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:33.8964134Z ok (2.832s)
2022-05-18T05:22:33.8973258Z   test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-05-18T05:22:33.9106140Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99977
2022-05-18T05:22:33.9215937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99978
2022-05-18T05:22:34.8701726Z dist init r=1, world=2
2022-05-18T05:22:34.8704908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:34.8884379Z dist init r=0, world=2
2022-05-18T05:22:34.8889041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:34.8890170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:34.8908536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:36.2504690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:36.2505233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:36.7280522Z ok (2.831s)
2022-05-18T05:22:36.7289769Z   test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-05-18T05:22:36.7418995Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100060
2022-05-18T05:22:36.7528727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100061
2022-05-18T05:22:37.7108121Z dist init r=0, world=2
2022-05-18T05:22:37.7112495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:37.7189789Z dist init r=1, world=2
2022-05-18T05:22:37.7193186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:37.7194018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:37.7214167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:39.0297318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:39.0297873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:39.5589686Z ok (2.831s)
2022-05-18T05:22:39.5599173Z   test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-05-18T05:22:39.5728062Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100143
2022-05-18T05:22:39.5836548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100144
2022-05-18T05:22:40.5168171Z dist init r=0, world=2
2022-05-18T05:22:40.5171556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:40.5596704Z dist init r=1, world=2
2022-05-18T05:22:40.5600698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:40.5602958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:40.5679233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:41.9011087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:41.9011604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:42.0913802Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:42.0914605Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:22:42.0945845Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:42.0946594Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:22:42.3898747Z ok (2.831s)
2022-05-18T05:22:42.3909674Z   test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-05-18T05:22:42.4040367Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100226
2022-05-18T05:22:42.4148935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100227
2022-05-18T05:22:43.3596625Z dist init r=1, world=2
2022-05-18T05:22:43.3599164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:43.3714885Z dist init r=0, world=2
2022-05-18T05:22:43.3719421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:43.3720238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:43.3802698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:44.7083644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:44.7084175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:44.9119793Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:44.9120596Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:22:44.9124217Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:44.9124936Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:22:45.2211233Z ok (2.831s)
2022-05-18T05:22:45.2220437Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-05-18T05:22:45.2352760Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100309
2022-05-18T05:22:45.2460889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100310
2022-05-18T05:22:46.1722961Z dist init r=0, world=2
2022-05-18T05:22:46.1726000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:46.1914305Z dist init r=1, world=2
2022-05-18T05:22:46.1918930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:46.1919751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:46.1929981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:47.5323630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:47.5324173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:47.7218444Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:47.7219169Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:22:47.7270111Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:47.7270766Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:22:47.7317778Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:47.7319145Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:22:47.7320150Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:47.7320867Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:22:48.4531818Z ok (3.232s)
2022-05-18T05:22:48.4539626Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-05-18T05:22:48.4668818Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100392
2022-05-18T05:22:48.4777711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100393
2022-05-18T05:22:49.3843525Z dist init r=0, world=2
2022-05-18T05:22:49.3846609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:49.3955734Z dist init r=1, world=2
2022-05-18T05:22:49.3960075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:49.3961251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:49.4050962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:50.7060138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:50.7061012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:50.8959684Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:50.8960453Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:22:50.8999969Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:50.9000614Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:22:50.9056945Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:50.9057643Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:22:50.9058549Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:50.9059206Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:22:51.5859661Z ok (3.133s)
2022-05-18T05:22:51.5867160Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-05-18T05:22:51.5995371Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100475
2022-05-18T05:22:51.6105310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100476
2022-05-18T05:22:52.5299905Z dist init r=0, world=2
2022-05-18T05:22:52.5302751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:52.5729685Z dist init r=1, world=2
2022-05-18T05:22:52.5734416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:52.5735251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:52.5810668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:53.9043720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:53.9044263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:54.0922902Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:54.0923736Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:22:54.0953875Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:54.0954537Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:22:54.1020976Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:54.1021946Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:22:54.1022835Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:54.1023489Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:22:54.8174857Z ok (3.231s)
2022-05-18T05:22:54.8182088Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-05-18T05:22:54.8311331Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100558
2022-05-18T05:22:54.8420219Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100559
2022-05-18T05:22:55.7996446Z dist init r=1, world=2
2022-05-18T05:22:55.7999395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:55.8309858Z dist init r=0, world=2
2022-05-18T05:22:55.8314613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:55.8315420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:55.8406223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:57.1866967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:22:57.1867506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:22:57.3673593Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:57.3674310Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:22:57.3781995Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:57.3782679Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:22:57.3862815Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:57.3863489Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:22:57.3864378Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:22:57.3865035Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:22:58.1492563Z ok (3.332s)
2022-05-18T05:22:58.1500184Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-05-18T05:22:58.1631399Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100641
2022-05-18T05:22:58.1739510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100642
2022-05-18T05:22:59.0944863Z dist init r=1, world=2
2022-05-18T05:22:59.0948659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:22:59.1308681Z dist init r=0, world=2
2022-05-18T05:22:59.1313043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:22:59.1314111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:22:59.1355488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:00.4832499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:00.4833015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:00.6708977Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:00.6709701Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:23:00.6715138Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:00.6715794Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:23:00.6761956Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:00.6762622Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:23:00.6763504Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:00.6764173Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:23:00.6872796Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:00.6873605Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:23:00.6876773Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:00.6877507Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:23:00.9801266Z ok (2.831s)
2022-05-18T05:23:00.9809229Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-05-18T05:23:00.9938042Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100724
2022-05-18T05:23:01.0049586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100725
2022-05-18T05:23:01.9243502Z dist init r=1, world=2
2022-05-18T05:23:01.9246800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:01.9781164Z dist init r=0, world=2
2022-05-18T05:23:01.9785676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:01.9786782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:01.9855854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:03.3173079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:03.3173662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:03.5417617Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:03.5418928Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:23:03.5437327Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:03.5439046Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:23:03.5491981Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:03.5492885Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:23:03.5494607Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:03.5495947Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:23:03.5602136Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:03.5603657Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:23:03.5610333Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:03.5611763Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:23:03.8111999Z ok (2.831s)
2022-05-18T05:23:03.8119734Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-05-18T05:23:03.8248168Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100807
2022-05-18T05:23:03.8356603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100808
2022-05-18T05:23:04.7430008Z dist init r=1, world=2
2022-05-18T05:23:04.7432700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:04.7473788Z dist init r=0, world=2
2022-05-18T05:23:04.7478834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:04.7480358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:04.7535804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:06.0967039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:06.0967605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:06.2840612Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:06.2841337Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:23:06.2898692Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:06.2899343Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:23:06.2967005Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:06.2967695Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:23:06.2968602Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:06.2969258Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:23:06.3079203Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:06.3080787Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:23:06.3082048Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:06.3082839Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:23:06.6419775Z ok (2.831s)
2022-05-18T05:23:06.6428507Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-05-18T05:23:06.6560482Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100890
2022-05-18T05:23:06.6673422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100891
2022-05-18T05:23:07.6259007Z dist init r=1, world=2
2022-05-18T05:23:07.6261868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:07.6309154Z dist init r=0, world=2
2022-05-18T05:23:07.6313502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:07.6314312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:07.6364629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:08.9399392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:08.9399916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:09.1317365Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:09.1318349Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:23:09.1319050Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:09.1319696Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-05-18T05:23:09.1395647Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:09.1396328Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:23:09.1397234Z /opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py:711: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:09.1397889Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-05-18T05:23:09.1505600Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:09.1506330Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:23:09.1508892Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:3852: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-05-18T05:23:09.1509871Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-05-18T05:23:09.4736058Z ok (2.831s)
2022-05-18T05:23:09.4736439Z 
2022-05-18T05:23:09.4737110Z ----------------------------------------------------------------------
2022-05-18T05:23:09.4737727Z Ran 14 tests in 42.749s
2022-05-18T05:23:09.4738024Z 
2022-05-18T05:23:09.4738201Z OK
2022-05-18T05:23:09.4738459Z 
2022-05-18T05:23:09.4738696Z Generating XML reports...
2022-05-18T05:23:09.4801455Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220518052226.xml
2022-05-18T05:23:09.4814779Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220518052226.xml
2022-05-18T05:23:09.7487621Z Running distributed/fsdp/test_wrap ... [2022-05-18 05:23:09.748199]
2022-05-18T05:23:09.7488368Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:23:09.748309]
2022-05-18T05:23:10.6627169Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap
2022-05-18T05:23:10.6648465Z 
2022-05-18T05:23:10.6648594Z Running tests...
2022-05-18T05:23:10.6649290Z ----------------------------------------------------------------------
2022-05-18T05:23:10.6657620Z   test_always_wrap (__main__.TestAutoWrap)
2022-05-18T05:23:12.2137752Z Test to ensure that if `always_wrap_policy` is ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:12.2344616Z ok (1.569s)
2022-05-18T05:23:12.2379071Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:12.2380171Z   warnings.warn(
2022-05-18T05:23:12.2416134Z ok (0.007s)
2022-05-18T05:23:12.2477058Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.006s)
2022-05-18T05:23:12.2484153Z   test_auto_wrap_api (__main__.TestAutoWrap)
2022-05-18T05:23:12.2504107Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.2505522Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.2506806Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.2518960Z ok (0.004s)
2022-05-18T05:23:12.2528342Z   test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap)
2022-05-18T05:23:12.2544917Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... ok (0.002s)
2022-05-18T05:23:12.2552475Z   test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap)
2022-05-18T05:23:12.2569033Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... ok (0.002s)
2022-05-18T05:23:12.2578378Z   test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap)
2022-05-18T05:23:12.2611831Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... ok (0.004s)
2022-05-18T05:23:12.2621750Z   test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap)
2022-05-18T05:23:12.2639320Z Test to ensure force-leaf modules are not wrapped. ... ok (0.003s)
2022-05-18T05:23:12.2671487Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:12.2672372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:23:12.2676822Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.2678455Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.2679759Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.2681158Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.2682423Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.4382381Z ok (0.174s)
2022-05-18T05:23:12.4412881Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:12.4413913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:23:12.4468407Z ok (0.009s)
2022-05-18T05:23:12.4492206Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... ok (0.002s)
2022-05-18T05:23:12.4515164Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... ok (0.002s)
2022-05-18T05:23:12.4543306Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:12.4544193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:23:12.4560620Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.4561957Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.4563240Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.4564515Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.4602826Z ok (0.009s)
2022-05-18T05:23:12.4630524Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:12.4631410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:23:12.4682538Z ok (0.008s)
2022-05-18T05:23:12.4710186Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:12.4711089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:23:12.4795616Z ok (0.011s)
2022-05-18T05:23:12.4824091Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:12.4824968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:23:12.4834317Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.4835628Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.4836889Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.4838497Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.4902518Z ok (0.011s)
2022-05-18T05:23:12.4942266Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... ok (0.004s)
2022-05-18T05:23:12.4981722Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.004s)
2022-05-18T05:23:12.5010776Z   test_transformer_auto_wrap_policy (__main__.TestAutoWrap) ... [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.5012127Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:12.5205210Z ok (0.022s)
2022-05-18T05:23:12.5227889Z   test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... ok (0.002s)
2022-05-18T05:23:12.5252949Z   test_wrap_override_defaults (__main__.TestAutoWrap) ... ok (0.002s)
2022-05-18T05:23:12.5276833Z   test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... ok (0.002s)
2022-05-18T05:23:12.5301071Z   test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.002s)
2022-05-18T05:23:12.5315872Z   test_bn_always_wrapped_individually (__main__.TestFSDPWrap)
2022-05-18T05:23:12.5592064Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101016
2022-05-18T05:23:12.5716708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101017
2022-05-18T05:23:13.4778848Z dist init r=0, world=2
2022-05-18T05:23:13.4782448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:13.5226383Z dist init r=1, world=2
2022-05-18T05:23:13.5230827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:13.5232702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:13.5291407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:14.8901536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:14.8902077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:14.9153349Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:14.9153948Z   warnings.warn(
2022-05-18T05:23:14.9154699Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:14.9155234Z   warnings.warn(
2022-05-18T05:23:15.1777048Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:15.1778734Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:15.1780173Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:15.1781701Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:15.1782558Z ok (2.648s)
2022-05-18T05:23:15.1792270Z   test_error_already_wrapped_nested_False_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-05-18T05:23:15.1931199Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101095
2022-05-18T05:23:15.2051372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101096
2022-05-18T05:23:16.1274507Z dist init r=1, world=2
2022-05-18T05:23:16.1277861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:16.1298352Z dist init r=0, world=2
2022-05-18T05:23:16.1303276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:16.1305626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:16.1381721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:17.4554583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:17.4555161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:17.4780970Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:17.4781558Z   warnings.warn(
2022-05-18T05:23:17.4782336Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:17.4782892Z   warnings.warn(
2022-05-18T05:23:17.8112121Z ok (2.633s)
2022-05-18T05:23:17.8122314Z   test_error_already_wrapped_nested_False_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-05-18T05:23:17.8262837Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101174
2022-05-18T05:23:17.8386378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101175
2022-05-18T05:23:18.7518518Z dist init r=0, world=2
2022-05-18T05:23:18.7522171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:18.7850595Z dist init r=1, world=2
2022-05-18T05:23:18.7854696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:18.7856888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:18.7929462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:20.1264340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:20.1265166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:20.4451812Z ok (2.634s)
2022-05-18T05:23:20.4461259Z   test_error_already_wrapped_nested_True_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-05-18T05:23:20.4600254Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101253
2022-05-18T05:23:20.4721649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101254
2022-05-18T05:23:21.3881572Z dist init r=1, world=2
2022-05-18T05:23:21.3885606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:21.4267731Z dist init r=0, world=2
2022-05-18T05:23:21.4271916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:21.4274195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:21.4292877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:22.7716034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:22.7716606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:22.7943941Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:22.7944856Z   warnings.warn(
2022-05-18T05:23:22.7945633Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:22.7946153Z   warnings.warn(
2022-05-18T05:23:23.0782195Z ok (2.633s)
2022-05-18T05:23:23.0791536Z   test_error_already_wrapped_nested_True_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-05-18T05:23:23.0931442Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101332
2022-05-18T05:23:23.1052942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101333
2022-05-18T05:23:24.0584436Z dist init r=1, world=2
2022-05-18T05:23:24.0587368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:24.0798996Z dist init r=0, world=2
2022-05-18T05:23:24.0804149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:24.0806680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:24.0893487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:25.4126057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:25.4126579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:25.7111165Z ok (2.633s)
2022-05-18T05:23:25.7281375Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101411
2022-05-18T05:23:25.7401919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101412
2022-05-18T05:23:26.6783514Z dist init r=0, world=2
2022-05-18T05:23:26.6786998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:26.6841681Z dist init r=1, world=2
2022-05-18T05:23:26.6846694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:26.6848654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:26.6890746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:28.0197960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:28.0198850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:28.0426015Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:28.0426583Z   warnings.warn(
2022-05-18T05:23:28.0427362Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:28.0427904Z   warnings.warn(
2022-05-18T05:23:28.5465495Z ok (2.835s)
2022-05-18T05:23:28.5638517Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101494
2022-05-18T05:23:28.5757815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101495
2022-05-18T05:23:29.4783653Z dist init r=1, world=2
2022-05-18T05:23:29.4787013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:29.5048503Z dist init r=0, world=2
2022-05-18T05:23:29.5053149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:29.5055690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:29.5093026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:30.8361721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:30.8362247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:31.3823820Z ok (2.836s)
2022-05-18T05:23:31.3995896Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101577
2022-05-18T05:23:31.4118087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101578
2022-05-18T05:23:32.3292890Z dist init r=1, world=2
2022-05-18T05:23:32.3295834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:32.3319213Z dist init r=0, world=2
2022-05-18T05:23:32.3323819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:32.3326569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:32.3399549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:33.6672707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:33.6673228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:33.6905343Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:33.6905951Z   warnings.warn(
2022-05-18T05:23:33.6906723Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:33.6907266Z   warnings.warn(
2022-05-18T05:23:34.2181475Z ok (2.836s)
2022-05-18T05:23:34.2354492Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101660
2022-05-18T05:23:34.2476811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101661
2022-05-18T05:23:35.1934483Z dist init r=0, world=2
2022-05-18T05:23:35.1937526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:35.2030975Z dist init r=1, world=2
2022-05-18T05:23:35.2035885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:35.2038734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:35.2042292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:36.5457037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:36.5457581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:37.0541509Z ok (2.836s)
2022-05-18T05:23:37.0713421Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101743
2022-05-18T05:23:37.0839134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101744
2022-05-18T05:23:37.9935553Z dist init r=1, world=2
2022-05-18T05:23:37.9938589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:38.0039287Z dist init r=0, world=2
2022-05-18T05:23:38.0044100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:38.0045892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:38.0143657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:39.3301033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:39.3301575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:39.5897510Z ok (2.535s)
2022-05-18T05:23:39.6069345Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101822
2022-05-18T05:23:39.6191017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101823
2022-05-18T05:23:40.5382770Z dist init r=1, world=2
2022-05-18T05:23:40.5383108Z dist init r=0, world=2
2022-05-18T05:23:40.5386882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:40.5387616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:40.5389113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:40.5389871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:41.8817363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:41.8817923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:42.4254778Z ok (2.836s)
2022-05-18T05:23:42.4427633Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101905
2022-05-18T05:23:42.4549864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101906
2022-05-18T05:23:43.4025452Z dist init r=0, world=2
2022-05-18T05:23:43.4028525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:43.4127884Z dist init r=1, world=2
2022-05-18T05:23:43.4132331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:43.4134398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:43.4233814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:44.7307108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:44.7308003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:45.0611268Z ok (2.635s)
2022-05-18T05:23:45.0785077Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101984
2022-05-18T05:23:45.0906754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101985
2022-05-18T05:23:46.0112465Z dist init r=1, world=2
2022-05-18T05:23:46.0116245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:46.0421328Z dist init r=0, world=2
2022-05-18T05:23:46.0425345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:46.0427564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:46.0523990Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:47.3841032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:47.3841579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:47.8971995Z ok (2.836s)
2022-05-18T05:23:47.9123280Z   test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102067
2022-05-18T05:23:47.9246378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102068
2022-05-18T05:23:48.8723067Z dist init r=1, world=2
2022-05-18T05:23:48.8726193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:48.8924599Z dist init r=0, world=2
2022-05-18T05:23:48.8929364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:48.8930226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:48.8932528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:50.2307043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:50.2307575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:50.2511856Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:50.2512447Z   warnings.warn(
2022-05-18T05:23:50.2513228Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:50.2513777Z   warnings.warn(
2022-05-18T05:23:50.5306923Z ok (2.633s)
2022-05-18T05:23:50.5462987Z   test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102146
2022-05-18T05:23:50.5587063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102147
2022-05-18T05:23:51.4788423Z dist init r=1, world=2
2022-05-18T05:23:51.4792316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:51.5125562Z dist init r=0, world=2
2022-05-18T05:23:51.5130103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:51.5131808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:51.5199579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:52.8454087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:52.8454619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:52.8672530Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:52.8673114Z   warnings.warn(
2022-05-18T05:23:52.8673873Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:23:52.8674404Z   warnings.warn(
2022-05-18T05:23:53.1649090Z ok (2.634s)
2022-05-18T05:23:53.1649289Z 
2022-05-18T05:23:53.1649975Z ----------------------------------------------------------------------
2022-05-18T05:23:53.1650742Z Ran 38 tests in 42.500s
2022-05-18T05:23:53.1651001Z 
2022-05-18T05:23:53.1651099Z OK
2022-05-18T05:23:53.1651218Z 
2022-05-18T05:23:53.1651349Z Generating XML reports...
2022-05-18T05:23:53.1712116Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1713546Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1715052Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1716350Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1717680Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1719193Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1720473Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1721759Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1723168Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1724439Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1725704Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1726976Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1728248Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1729498Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1730845Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1732126Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1733580Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1734850Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:23:53.1747102Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20220518052310.xml
2022-05-18T05:23:53.1764139Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20220518052310.xml
2022-05-18T05:23:53.4563034Z Running distributed/fsdp/test_fsdp_freezing_weights ... [2022-05-18 05:23:53.455764]
2022-05-18T05:23:53.4563835Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_freezing_weights.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:23:53.455899]
2022-05-18T05:23:54.3458463Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights
2022-05-18T05:23:54.3474208Z 
2022-05-18T05:23:54.3474609Z Running tests...
2022-05-18T05:23:54.3475125Z ----------------------------------------------------------------------
2022-05-18T05:23:55.8833824Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:23:55.9231778Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102260
2022-05-18T05:23:55.9345058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102261
2022-05-18T05:23:56.8516386Z dist init r=1, world=2
2022-05-18T05:23:56.8519497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:23:56.8569452Z dist init r=0, world=2
2022-05-18T05:23:56.8574429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:23:56.8575826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:56.8622601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:23:58.2154425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:23:58.2154962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:23:58.9768573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:23:58.9769175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:23:59.3425055Z ok (4.995s)
2022-05-18T05:23:59.3565881Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102343
2022-05-18T05:23:59.3676156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102344
2022-05-18T05:24:00.2699927Z dist init r=1, world=2
2022-05-18T05:24:00.2703014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:00.3129788Z dist init r=0, world=2
2022-05-18T05:24:00.3134391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:00.3135869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:00.3211263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:01.6356820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:01.6357379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:02.3999304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:02.4003428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:02.4114261Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:24:02.4114925Z   warnings.warn(
2022-05-18T05:24:02.4116021Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:24:02.4116561Z   warnings.warn(
2022-05-18T05:24:02.7750495Z ok (3.432s)
2022-05-18T05:24:02.7891646Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102426
2022-05-18T05:24:02.8000736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102427
2022-05-18T05:24:03.7131448Z dist init r=1, world=2
2022-05-18T05:24:03.7134318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:03.7164094Z dist init r=0, world=2
2022-05-18T05:24:03.7168678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:03.7169794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:03.7237100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:05.0493788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:05.0494438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:05.7970082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:05.7970619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:06.1072272Z ok (3.332s)
2022-05-18T05:24:06.1211873Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102509
2022-05-18T05:24:06.1323412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102510
2022-05-18T05:24:07.0524542Z dist init r=1, world=2
2022-05-18T05:24:07.0527954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:07.0873219Z dist init r=0, world=2
2022-05-18T05:24:07.0877769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:07.0878901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:07.0934667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:08.4289880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:08.4290456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:09.1846567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:09.1848717Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:09.1923169Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:24:09.1923755Z   warnings.warn(
2022-05-18T05:24:09.1924516Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:24:09.1925061Z   warnings.warn(
2022-05-18T05:24:09.5397178Z ok (3.432s)
2022-05-18T05:24:09.5537540Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102592
2022-05-18T05:24:09.5646779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102593
2022-05-18T05:24:10.4839225Z dist init r=0, world=2
2022-05-18T05:24:10.4842711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:10.5242492Z dist init r=1, world=2
2022-05-18T05:24:10.5246997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:10.5247770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:10.5248761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:11.8406221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:11.8406884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:12.5978935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:12.5979495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:13.0720587Z ok (3.532s)
2022-05-18T05:24:13.0861594Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102675
2022-05-18T05:24:13.0972763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102676
2022-05-18T05:24:14.0214994Z dist init r=1, world=2
2022-05-18T05:24:14.0217848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:14.0242902Z dist init r=0, world=2
2022-05-18T05:24:14.0247712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:14.0249092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:14.0321022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:15.3775995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:15.3776522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:16.1435163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:16.1435721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:16.1728972Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:24:16.1729557Z   warnings.warn(
2022-05-18T05:24:16.1730339Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:24:16.1730885Z   warnings.warn(
2022-05-18T05:24:16.6048432Z ok (3.533s)
2022-05-18T05:24:16.6188871Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102758
2022-05-18T05:24:16.6298135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102759
2022-05-18T05:24:17.5409321Z dist init r=1, world=2
2022-05-18T05:24:17.5412951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:17.5436301Z dist init r=0, world=2
2022-05-18T05:24:17.5441075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:17.5441918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:17.5515822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:18.8839014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:18.8839570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:19.6401239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:19.6405575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:20.0371543Z ok (3.432s)
2022-05-18T05:24:20.0512812Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102841
2022-05-18T05:24:20.0625689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102842
2022-05-18T05:24:21.0149210Z dist init r=0, world=2
2022-05-18T05:24:21.0152349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:21.0349650Z dist init r=1, world=2
2022-05-18T05:24:21.0354413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:21.0355257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:21.0355970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:22.3705735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:22.3706675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:23.1246501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:23.1247063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:24:23.1361142Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:24:23.1361734Z   warnings.warn(
2022-05-18T05:24:23.1362519Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:24:23.1363064Z   warnings.warn(
2022-05-18T05:24:23.4700794Z ok (3.433s)
2022-05-18T05:24:23.4701128Z 
2022-05-18T05:24:23.4701838Z ----------------------------------------------------------------------
2022-05-18T05:24:23.4702463Z Ran 8 tests in 29.123s
2022-05-18T05:24:23.4702775Z 
2022-05-18T05:24:23.4702955Z OK
2022-05-18T05:24:23.4705812Z 
2022-05-18T05:24:23.4706152Z Generating XML reports...
2022-05-18T05:24:23.4766087Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20220518052354.xml
2022-05-18T05:24:23.7429701Z Running distributed/test_c10d_spawn_nccl ... [2022-05-18 05:24:23.742449]
2022-05-18T05:24:23.7430440Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:24:23.742556]
2022-05-18T05:24:24.6423963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxkzi2w1
2022-05-18T05:24:24.6425304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxkzi2w1/_remote_module_non_scriptable.py
2022-05-18T05:24:26.2417269Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:26.2455519Z <unittest.suite.TestSuite tests=[]>
2022-05-18T05:24:26.2456479Z <unittest.suite.TestSuite tests=[]>
2022-05-18T05:24:26.2458572Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>]>
2022-05-18T05:24:26.2459831Z test_all_gather (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T05:24:26.2460238Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T05:24:26.2460826Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T05:24:26.2461583Z test_allreduce (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T05:24:26.2462308Z test_broadcast (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T05:24:26.2463033Z test_reduce (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T05:24:26.2463779Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T05:24:27.1154901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6xd3to5
2022-05-18T05:24:27.1155876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6xd3to5/_remote_module_non_scriptable.py
2022-05-18T05:24:28.6966927Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:28.7025833Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T05:24:28.7041859Z 
2022-05-18T05:24:28.7042084Z Running tests...
2022-05-18T05:24:28.7042519Z ----------------------------------------------------------------------
2022-05-18T05:24:28.7407877Z   test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102993
2022-05-18T05:24:28.7508138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102994
2022-05-18T05:24:29.6529159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx2l2xivr
2022-05-18T05:24:29.6530391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx2l2xivr/_remote_module_non_scriptable.py
2022-05-18T05:24:29.6668809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3pnm2_m
2022-05-18T05:24:29.6671562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3pnm2_m/_remote_module_non_scriptable.py
2022-05-18T05:24:31.2864686Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:31.2877405Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:31.2904235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:31.2908260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:31.2917627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:31.2921485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:31.2922704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:31.3011025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:32.6594890Z ok (3.955s)
2022-05-18T05:24:32.6595258Z 
2022-05-18T05:24:32.6595910Z ----------------------------------------------------------------------
2022-05-18T05:24:32.6596537Z Ran 1 test in 3.955s
2022-05-18T05:24:32.6596846Z 
2022-05-18T05:24:32.6597012Z OK
2022-05-18T05:24:32.6597250Z 
2022-05-18T05:24:32.6597459Z Generating XML reports...
2022-05-18T05:24:32.6641083Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052428.xml
2022-05-18T05:24:33.8342284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1mi46_r
2022-05-18T05:24:33.8343574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1mi46_r/_remote_module_non_scriptable.py
2022-05-18T05:24:35.4011999Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:35.4068793Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T05:24:35.4084147Z 
2022-05-18T05:24:35.4084296Z Running tests...
2022-05-18T05:24:35.4085011Z ----------------------------------------------------------------------
2022-05-18T05:24:35.4441302Z   test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103112
2022-05-18T05:24:35.4540811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103113
2022-05-18T05:24:36.3249361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8vnn2np
2022-05-18T05:24:36.3250470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8vnn2np/_remote_module_non_scriptable.py
2022-05-18T05:24:36.3273657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftx6gv45
2022-05-18T05:24:36.3276794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftx6gv45/_remote_module_non_scriptable.py
2022-05-18T05:24:37.9400919Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:37.9445550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:37.9448379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:37.9560284Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:37.9599799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:37.9603353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:37.9604154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:37.9652478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:39.3627209Z ok (3.954s)
2022-05-18T05:24:39.3627463Z 
2022-05-18T05:24:39.3627845Z ----------------------------------------------------------------------
2022-05-18T05:24:39.3628186Z Ran 1 test in 3.954s
2022-05-18T05:24:39.3628374Z 
2022-05-18T05:24:39.3628470Z OK
2022-05-18T05:24:39.3628612Z 
2022-05-18T05:24:39.3628729Z Generating XML reports...
2022-05-18T05:24:39.3671148Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052435.xml
2022-05-18T05:24:40.5359708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hmkxdhg
2022-05-18T05:24:40.5361802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hmkxdhg/_remote_module_non_scriptable.py
2022-05-18T05:24:42.1158845Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:42.1219126Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T05:24:42.1236157Z 
2022-05-18T05:24:42.1236568Z Running tests...
2022-05-18T05:24:42.1237036Z ----------------------------------------------------------------------
2022-05-18T05:24:42.1603340Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103231
2022-05-18T05:24:42.1704420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103232
2022-05-18T05:24:43.0461387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5v4jw8ee
2022-05-18T05:24:43.0462776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5v4jw8ee/_remote_module_non_scriptable.py
2022-05-18T05:24:43.0472867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12axap3y
2022-05-18T05:24:43.0475533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12axap3y/_remote_module_non_scriptable.py
2022-05-18T05:24:44.6816688Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:44.6855467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:44.6859753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:44.7146970Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:44.7188377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:44.7192076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:44.7193519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:44.7266322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:46.1791873Z ok (4.055s)
2022-05-18T05:24:46.1792823Z 
2022-05-18T05:24:46.1793240Z ----------------------------------------------------------------------
2022-05-18T05:24:46.1793583Z Ran 1 test in 4.056s
2022-05-18T05:24:46.1793753Z 
2022-05-18T05:24:46.1793850Z OK
2022-05-18T05:24:46.1793986Z 
2022-05-18T05:24:46.1794125Z Generating XML reports...
2022-05-18T05:24:46.1837756Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052442.xml
2022-05-18T05:24:47.3118874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxczy3m2f
2022-05-18T05:24:47.3120304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxczy3m2f/_remote_module_non_scriptable.py
2022-05-18T05:24:48.9010523Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:48.9070569Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T05:24:48.9087833Z 
2022-05-18T05:24:48.9088118Z Running tests...
2022-05-18T05:24:48.9088568Z ----------------------------------------------------------------------
2022-05-18T05:24:48.9455511Z   test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103350
2022-05-18T05:24:48.9555926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103351
2022-05-18T05:24:49.8570158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2f28ava8
2022-05-18T05:24:49.8571598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2f28ava8/_remote_module_non_scriptable.py
2022-05-18T05:24:49.8755796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbggqyu2
2022-05-18T05:24:49.8758342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbggqyu2/_remote_module_non_scriptable.py
2022-05-18T05:24:51.4703394Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:51.4742378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:51.4745338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:51.4909538Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:51.4950171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:51.4953754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:51.4954568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:51.5050550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:52.8643008Z ok (3.955s)
2022-05-18T05:24:52.8643227Z 
2022-05-18T05:24:52.8643606Z ----------------------------------------------------------------------
2022-05-18T05:24:52.8643933Z Ran 1 test in 3.955s
2022-05-18T05:24:52.8644106Z 
2022-05-18T05:24:52.8644202Z OK
2022-05-18T05:24:52.8644341Z 
2022-05-18T05:24:52.8644473Z Generating XML reports...
2022-05-18T05:24:52.8686333Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052448.xml
2022-05-18T05:24:54.0201839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnqgzk783
2022-05-18T05:24:54.0203282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnqgzk783/_remote_module_non_scriptable.py
2022-05-18T05:24:55.5741514Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:55.5799676Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T05:24:55.5815214Z 
2022-05-18T05:24:55.5815610Z Running tests...
2022-05-18T05:24:55.5816113Z ----------------------------------------------------------------------
2022-05-18T05:24:55.6180369Z   test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103469
2022-05-18T05:24:55.6278863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103470
2022-05-18T05:24:56.4896251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5k3xxdns
2022-05-18T05:24:56.4897234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5k3xxdns/_remote_module_non_scriptable.py
2022-05-18T05:24:56.4968745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngycuxw8
2022-05-18T05:24:56.4971561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngycuxw8/_remote_module_non_scriptable.py
2022-05-18T05:24:58.1189290Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:58.1229023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:24:58.1231887Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:24:58.1232306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:24:58.1271528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:24:58.1274998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:24:58.1275974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:58.1335662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:24:59.6369758Z ok (4.055s)
2022-05-18T05:24:59.6369974Z 
2022-05-18T05:24:59.6370394Z ----------------------------------------------------------------------
2022-05-18T05:24:59.6370736Z Ran 1 test in 4.055s
2022-05-18T05:24:59.6370883Z 
2022-05-18T05:24:59.6370979Z OK
2022-05-18T05:24:59.6371114Z 
2022-05-18T05:24:59.6371247Z Generating XML reports...
2022-05-18T05:24:59.6414483Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052455.xml
2022-05-18T05:25:00.8196607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwnsbempu
2022-05-18T05:25:00.8197638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwnsbempu/_remote_module_non_scriptable.py
2022-05-18T05:25:02.4033616Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:02.4094303Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T05:25:02.4109509Z 
2022-05-18T05:25:02.4109706Z Running tests...
2022-05-18T05:25:02.4110150Z ----------------------------------------------------------------------
2022-05-18T05:25:02.4481253Z   test_reduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103588
2022-05-18T05:25:02.4582573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103589
2022-05-18T05:25:03.3292757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptsarc4fd
2022-05-18T05:25:03.3293689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptsarc4fd/_remote_module_non_scriptable.py
2022-05-18T05:25:03.3335675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkncr2ex
2022-05-18T05:25:03.3338555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkncr2ex/_remote_module_non_scriptable.py
2022-05-18T05:25:04.9453265Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:04.9478801Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:04.9492427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:04.9496255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:04.9517567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:04.9521278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:04.9522086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:04.9599335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:06.3670354Z ok (3.956s)
2022-05-18T05:25:06.3670580Z 
2022-05-18T05:25:06.3670999Z ----------------------------------------------------------------------
2022-05-18T05:25:06.3671342Z Ran 1 test in 3.956s
2022-05-18T05:25:06.3671492Z 
2022-05-18T05:25:06.3671588Z OK
2022-05-18T05:25:06.3671725Z 
2022-05-18T05:25:06.3672160Z Generating XML reports...
2022-05-18T05:25:06.3715208Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052502.xml
2022-05-18T05:25:07.5564175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1pj7m2b
2022-05-18T05:25:07.5565354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1pj7m2b/_remote_module_non_scriptable.py
2022-05-18T05:25:09.1312403Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:09.1370792Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T05:25:09.1386578Z 
2022-05-18T05:25:09.1386927Z Running tests...
2022-05-18T05:25:09.1387353Z ----------------------------------------------------------------------
2022-05-18T05:25:09.1759427Z   test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103707
2022-05-18T05:25:09.1859383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103708
2022-05-18T05:25:10.0875646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp91snuib6
2022-05-18T05:25:10.0876638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp91snuib6/_remote_module_non_scriptable.py
2022-05-18T05:25:10.1082071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizai8h_6
2022-05-18T05:25:10.1084980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizai8h_6/_remote_module_non_scriptable.py
2022-05-18T05:25:11.7145569Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:11.7183770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:11.7187181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:11.7300936Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:11.7342101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:11.7345997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:11.7346808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:11.7391540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:13.1947591Z ok (4.056s)
2022-05-18T05:25:13.1947815Z 
2022-05-18T05:25:13.1948216Z ----------------------------------------------------------------------
2022-05-18T05:25:13.1948595Z Ran 1 test in 4.056s
2022-05-18T05:25:13.1948761Z 
2022-05-18T05:25:13.1948859Z OK
2022-05-18T05:25:13.1948998Z 
2022-05-18T05:25:13.1949133Z Generating XML reports...
2022-05-18T05:25:13.1992447Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052509.xml
2022-05-18T05:25:13.7287646Z Running distributed/test_c10d_common ... [2022-05-18 05:25:13.728225]
2022-05-18T05:25:13.7288397Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:25:13.728335]
2022-05-18T05:25:14.6328960Z test_debug_level (__main__.CommTest)
2022-05-18T05:25:14.6329425Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-05-18T05:25:14.6329868Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-05-18T05:25:14.6330287Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-05-18T05:25:14.6330959Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-05-18T05:25:14.6331765Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest)
2022-05-18T05:25:14.6332220Z test_collectives (__main__.PythonProcessGroupExtensionTest)
2022-05-18T05:25:14.6332885Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest)
2022-05-18T05:25:14.6333361Z test_send_recv (__main__.PythonProcessGroupExtensionTest)
2022-05-18T05:25:15.5323714Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T05:25:15.5337473Z 
2022-05-18T05:25:15.5337735Z Running tests...
2022-05-18T05:25:15.5338159Z ----------------------------------------------------------------------
2022-05-18T05:25:17.1046968Z   test_debug_level (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:17.1450580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103859
2022-05-18T05:25:17.1561788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103860
2022-05-18T05:25:18.0425497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:18.0459804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:18.2601968Z ok (2.726s)
2022-05-18T05:25:18.2602197Z 
2022-05-18T05:25:18.2602617Z ----------------------------------------------------------------------
2022-05-18T05:25:18.2602945Z Ran 1 test in 2.726s
2022-05-18T05:25:18.2603123Z 
2022-05-18T05:25:18.2603220Z OK
2022-05-18T05:25:18.2603356Z 
2022-05-18T05:25:18.2603490Z Generating XML reports...
2022-05-18T05:25:18.2645456Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220518052515.xml
2022-05-18T05:25:19.3714061Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T05:25:19.3729003Z 
2022-05-18T05:25:19.3729197Z Running tests...
2022-05-18T05:25:19.3729636Z ----------------------------------------------------------------------
2022-05-18T05:25:20.9601283Z   test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:20.9763902Z ok (1.603s)
2022-05-18T05:25:20.9764304Z 
2022-05-18T05:25:20.9764730Z ----------------------------------------------------------------------
2022-05-18T05:25:20.9765067Z Ran 1 test in 1.604s
2022-05-18T05:25:20.9765235Z 
2022-05-18T05:25:20.9765330Z OK
2022-05-18T05:25:20.9765466Z 
2022-05-18T05:25:20.9765600Z Generating XML reports...
2022-05-18T05:25:20.9797723Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518052519.xml
2022-05-18T05:25:22.1126707Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T05:25:22.1141282Z 
2022-05-18T05:25:22.1141684Z Running tests...
2022-05-18T05:25:22.1142190Z ----------------------------------------------------------------------
2022-05-18T05:25:23.6913563Z   test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:23.7073937Z ok (1.593s)
2022-05-18T05:25:23.7074426Z 
2022-05-18T05:25:23.7074816Z ----------------------------------------------------------------------
2022-05-18T05:25:23.7075195Z Ran 1 test in 1.593s
2022-05-18T05:25:23.7075362Z 
2022-05-18T05:25:23.7075459Z OK
2022-05-18T05:25:23.7075578Z 
2022-05-18T05:25:23.7075709Z Generating XML reports...
2022-05-18T05:25:23.7108839Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518052522.xml
2022-05-18T05:25:24.8367714Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T05:25:24.8381970Z 
2022-05-18T05:25:24.8382399Z Running tests...
2022-05-18T05:25:24.8382893Z ----------------------------------------------------------------------
2022-05-18T05:25:26.3999306Z   test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:26.4153446Z ok (1.577s)
2022-05-18T05:25:26.4153971Z 
2022-05-18T05:25:26.4154375Z ----------------------------------------------------------------------
2022-05-18T05:25:26.4154714Z Ran 1 test in 1.577s
2022-05-18T05:25:26.4154886Z 
2022-05-18T05:25:26.4155258Z OK
2022-05-18T05:25:26.4155420Z 
2022-05-18T05:25:26.4155558Z Generating XML reports...
2022-05-18T05:25:26.4186659Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518052524.xml
2022-05-18T05:25:27.5343211Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T05:25:27.5357553Z 
2022-05-18T05:25:27.5357852Z Running tests...
2022-05-18T05:25:27.5358556Z ----------------------------------------------------------------------
2022-05-18T05:25:29.1170859Z   test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:29.1331846Z ok (1.597s)
2022-05-18T05:25:29.1332373Z 
2022-05-18T05:25:29.1332859Z ----------------------------------------------------------------------
2022-05-18T05:25:29.1333232Z Ran 1 test in 1.598s
2022-05-18T05:25:29.1333406Z 
2022-05-18T05:25:29.1333504Z OK
2022-05-18T05:25:29.1333641Z 
2022-05-18T05:25:29.1333768Z Generating XML reports...
2022-05-18T05:25:29.1367024Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518052527.xml
2022-05-18T05:25:30.2590595Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T05:25:30.2605138Z 
2022-05-18T05:25:30.2605392Z Running tests...
2022-05-18T05:25:30.2605814Z ----------------------------------------------------------------------
2022-05-18T05:25:31.8400026Z   test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:31.8804417Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104098
2022-05-18T05:25:31.8916006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104099
2022-05-18T05:25:31.9029691Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104100
2022-05-18T05:25:31.9143181Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104101
2022-05-18T05:25:32.7875929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:25:32.8358159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:32.8382884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:32.8425185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:25:33.0184126Z ok (2.758s)
2022-05-18T05:25:33.0184346Z 
2022-05-18T05:25:33.0184749Z ----------------------------------------------------------------------
2022-05-18T05:25:33.0185115Z Ran 1 test in 2.758s
2022-05-18T05:25:33.0185285Z 
2022-05-18T05:25:33.0185365Z OK
2022-05-18T05:25:33.0185508Z 
2022-05-18T05:25:33.0185644Z Generating XML reports...
2022-05-18T05:25:33.0229133Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518052530.xml
2022-05-18T05:25:34.1857723Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T05:25:34.1872818Z 
2022-05-18T05:25:34.1873144Z Running tests...
2022-05-18T05:25:34.1873588Z ----------------------------------------------------------------------
2022-05-18T05:25:35.7635910Z   test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:35.8030078Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104269
2022-05-18T05:25:35.8138803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104270
2022-05-18T05:25:35.8250631Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104271
2022-05-18T05:25:35.8364412Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104272
2022-05-18T05:25:36.7329429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:36.7334749Z [W socket.cpp:558] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-05-18T05:25:36.7387012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:36.7982404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:25:36.7992053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T05:25:36.8019051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:25:36.8029777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T05:25:37.7342665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:37.7394601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:37.7395392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:25:37.7442833Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:25:37.7445439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:25:37.7485601Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:25:39.5452237Z ok (5.358s)
2022-05-18T05:25:39.5452737Z 
2022-05-18T05:25:39.5453161Z ----------------------------------------------------------------------
2022-05-18T05:25:39.5453488Z Ran 1 test in 5.358s
2022-05-18T05:25:39.5453656Z 
2022-05-18T05:25:39.5453752Z OK
2022-05-18T05:25:39.5453888Z 
2022-05-18T05:25:39.5454022Z Generating XML reports...
2022-05-18T05:25:39.5497792Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518052534.xml
2022-05-18T05:25:40.7105876Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T05:25:40.7119407Z 
2022-05-18T05:25:40.7119924Z Running tests...
2022-05-18T05:25:40.7120432Z ----------------------------------------------------------------------
2022-05-18T05:25:42.2441423Z   test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:42.2839373Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104449
2022-05-18T05:25:42.2946915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104450
2022-05-18T05:25:42.3059208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104451
2022-05-18T05:25:42.3172094Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104452
2022-05-18T05:25:43.2737940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:25:43.2741341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:25:43.3073352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:43.3144668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:43.5217921Z ok (2.810s)
2022-05-18T05:25:43.5218166Z 
2022-05-18T05:25:43.5218705Z ----------------------------------------------------------------------
2022-05-18T05:25:43.5219168Z Ran 1 test in 2.810s
2022-05-18T05:25:43.5219357Z 
2022-05-18T05:25:43.5219456Z OK
2022-05-18T05:25:43.5219574Z 
2022-05-18T05:25:43.5219713Z Generating XML reports...
2022-05-18T05:25:43.5262697Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518052540.xml
2022-05-18T05:25:44.6804605Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T05:25:44.6818083Z 
2022-05-18T05:25:44.6818611Z Running tests...
2022-05-18T05:25:44.6819100Z ----------------------------------------------------------------------
2022-05-18T05:25:46.2156996Z   test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:46.2553161Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104620
2022-05-18T05:25:46.2662439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104621
2022-05-18T05:25:46.2774328Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104622
2022-05-18T05:25:46.2886389Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104623
2022-05-18T05:25:47.2092088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:47.2540426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:25:47.2549932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T05:25:47.2722180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:25:47.2732825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T05:25:47.2741350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:47.2751694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:47.2819083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:47.2819891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:25:47.2835868Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:25:47.2855351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:25:47.2856046Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T05:25:49.0957776Z ok (4.414s)
2022-05-18T05:25:49.0957964Z 
2022-05-18T05:25:49.0958631Z ----------------------------------------------------------------------
2022-05-18T05:25:49.0958985Z Ran 1 test in 4.414s
2022-05-18T05:25:49.0959153Z 
2022-05-18T05:25:49.0959250Z OK
2022-05-18T05:25:49.0959389Z 
2022-05-18T05:25:49.0959517Z Generating XML reports...
2022-05-18T05:25:49.1001905Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518052544.xml
2022-05-18T05:25:49.4981367Z Running distributed/fsdp/test_fsdp_exec_order ... [2022-05-18 05:25:49.497613]
2022-05-18T05:25:49.4982097Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:25:49.497724]
2022-05-18T05:25:50.4254504Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order
2022-05-18T05:25:50.4271553Z 
2022-05-18T05:25:50.4271873Z Running tests...
2022-05-18T05:25:50.4272311Z ----------------------------------------------------------------------
2022-05-18T05:25:50.4282359Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-05-18T05:25:51.9974199Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:25:52.0380845Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104800
2022-05-18T05:25:52.0494323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104801
2022-05-18T05:25:52.9657968Z dist init r=0, world=2
2022-05-18T05:25:52.9660282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:52.9676675Z dist init r=1, world=2
2022-05-18T05:25:52.9681993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:52.9683387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:52.9763242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:54.3212727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:54.3213274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:54.3427390Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:25:54.3428011Z   warnings.warn(
2022-05-18T05:25:54.3428766Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:25:54.3429311Z   warnings.warn(
2022-05-18T05:25:54.8565097Z ok (4.429s)
2022-05-18T05:25:54.8575477Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-05-18T05:25:54.8706692Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104879
2022-05-18T05:25:54.8813646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104880
2022-05-18T05:25:55.7990150Z dist init r=1, world=2
2022-05-18T05:25:55.7992996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:55.8009012Z dist init r=0, world=2
2022-05-18T05:25:55.8014167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:55.8015074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:55.8095564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:57.1386374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:57.1386935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:57.1585647Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:25:57.1586230Z   warnings.warn(
2022-05-18T05:25:57.1586982Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:25:57.1587526Z   warnings.warn(
2022-05-18T05:25:57.5875107Z ok (2.731s)
2022-05-18T05:25:57.5894262Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-05-18T05:25:57.6023853Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104958
2022-05-18T05:25:57.6137246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104959
2022-05-18T05:25:58.5762362Z dist init r=0, world=2
2022-05-18T05:25:58.5766317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:25:58.5820604Z dist init r=1, world=2
2022-05-18T05:25:58.5825913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:25:58.5827306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:58.5870407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:25:59.9113695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:25:59.9114720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:25:59.9347830Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:25:59.9348428Z   warnings.warn(
2022-05-18T05:25:59.9380606Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:25:59.9381729Z   warnings.warn(
2022-05-18T05:26:00.1219044Z [['layer0.weight', 'layer0.bias'], ['layer1.weight']] <class 'list'>
2022-05-18T05:26:00.1220010Z [['layer2.0.weight', 'layer2.2.weight']] <class 'list'>
2022-05-18T05:26:00.4199346Z ok (2.832s)
2022-05-18T05:26:00.4217800Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-05-18T05:26:00.4346962Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105041
2022-05-18T05:26:00.4456792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105042
2022-05-18T05:26:01.3669342Z dist init r=1, world=2
2022-05-18T05:26:01.3672734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:01.3705493Z dist init r=0, world=2
2022-05-18T05:26:01.3710786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:01.3712360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:01.3775819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:02.7214098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:02.7214634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:02.7465024Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:26:02.7465606Z   warnings.warn(
2022-05-18T05:26:02.7466354Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:26:02.7466894Z   warnings.warn(
2022-05-18T05:26:02.9351221Z [['layer0.weight', 'layer0.bias'], ['layer1.weight']] <class 'list'>
2022-05-18T05:26:02.9351725Z [['layer2.0.weight', 'layer2.2.weight']] <class 'list'>
2022-05-18T05:26:03.2520827Z ok (2.832s)
2022-05-18T05:26:03.2539353Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-05-18T05:26:03.2672078Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105124
2022-05-18T05:26:03.2783666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105125
2022-05-18T05:26:04.1813562Z dist init r=1, world=2
2022-05-18T05:26:04.1816857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:04.2438268Z dist init r=0, world=2
2022-05-18T05:26:04.2443304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:04.2444665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:04.2526814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:05.5884068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:05.5884775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:05.6105774Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:26:05.6106369Z   warnings.warn(
2022-05-18T05:26:05.6107142Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:26:05.6107981Z   warnings.warn(
2022-05-18T05:26:05.7922001Z [['layer0.weight', 'layer0.bias'], ['layer1.weight']] <class 'list'>
2022-05-18T05:26:05.7922676Z [['layer2.0.weight', 'layer2.2.weight']] <class 'list'>
2022-05-18T05:26:06.0845672Z ok (2.832s)
2022-05-18T05:26:06.0864250Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-05-18T05:26:06.0994827Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105207
2022-05-18T05:26:06.1106051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105208
2022-05-18T05:26:07.0293304Z dist init r=1, world=2
2022-05-18T05:26:07.0296332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:07.0527535Z dist init r=0, world=2
2022-05-18T05:26:07.0532053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:07.0532869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:07.0601881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:08.3704962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:08.3705493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:08.3904270Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:26:08.3904853Z   warnings.warn(
2022-05-18T05:26:08.3939129Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:26:08.3939684Z   warnings.warn(
2022-05-18T05:26:08.5789975Z [['layer0.weight', 'layer0.bias'], ['layer1.weight']] <class 'list'>
2022-05-18T05:26:08.5790524Z [['layer2.0.weight', 'layer2.2.weight']] <class 'list'>
2022-05-18T05:26:08.9167566Z ok (2.832s)
2022-05-18T05:26:08.9317058Z   test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105290
2022-05-18T05:26:08.9429115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105291
2022-05-18T05:26:09.8527334Z dist init r=0, world=2
2022-05-18T05:26:09.8530259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:09.8878441Z dist init r=1, world=2
2022-05-18T05:26:09.8883512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:09.8884759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:09.8936815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:11.2093196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:11.2093767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:11.2306091Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:26:11.2307030Z   warnings.warn(
2022-05-18T05:26:11.2307778Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:26:11.2308321Z   warnings.warn(
2022-05-18T05:26:11.7491598Z ok (2.832s)
2022-05-18T05:26:11.7636941Z   test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105373
2022-05-18T05:26:11.7747271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105374
2022-05-18T05:26:12.7297951Z dist init r=0, world=2
2022-05-18T05:26:12.7301198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:12.7497248Z dist init r=1, world=2
2022-05-18T05:26:12.7502009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:12.7503136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:12.7505458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:14.0920644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:14.0921203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:14.1146582Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:26:14.1147239Z   warnings.warn(
2022-05-18T05:26:14.1179911Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:26:14.1180475Z   warnings.warn(
2022-05-18T05:26:14.5808760Z ok (2.832s)
2022-05-18T05:26:14.5809073Z 
2022-05-18T05:26:14.5809751Z ----------------------------------------------------------------------
2022-05-18T05:26:14.5810093Z Ran 8 tests in 24.154s
2022-05-18T05:26:14.5810265Z 
2022-05-18T05:26:14.5810360Z OK
2022-05-18T05:26:14.5815360Z 
2022-05-18T05:26:14.5815921Z Generating XML reports...
2022-05-18T05:26:14.5874577Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20220518052550.xml
2022-05-18T05:26:14.8533834Z Running distributed/fsdp/test_fsdp_meta ... [2022-05-18 05:26:14.852808]
2022-05-18T05:26:14.8535034Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:26:14.852914]
2022-05-18T05:26:15.7809953Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta
2022-05-18T05:26:15.7827674Z 
2022-05-18T05:26:15.7828115Z Running tests...
2022-05-18T05:26:15.7828632Z ----------------------------------------------------------------------
2022-05-18T05:26:17.3675278Z   test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:26:17.4079378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105491
2022-05-18T05:26:17.4195328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105492
2022-05-18T05:26:18.3269854Z dist init r=0, world=2
2022-05-18T05:26:18.3273265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:18.3741648Z dist init r=1, world=2
2022-05-18T05:26:18.3745716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:18.3746761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:18.3781256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:19.7449645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:19.7450185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:20.0259400Z ok (4.243s)
2022-05-18T05:26:20.0267268Z   test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-05-18T05:26:20.0402574Z   test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105570
2022-05-18T05:26:20.0512580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105571
2022-05-18T05:26:20.9671379Z dist init r=0, world=2
2022-05-18T05:26:20.9674884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:20.9699583Z dist init r=1, world=2
2022-05-18T05:26:20.9703796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:20.9704985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:20.9778079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:22.2878190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:22.2878972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:22.7575665Z ok (2.731s)
2022-05-18T05:26:22.7714146Z   test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105653
2022-05-18T05:26:22.7825716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105654
2022-05-18T05:26:23.6926084Z dist init r=0, world=2
2022-05-18T05:26:23.6929445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:23.7205639Z dist init r=1, world=2
2022-05-18T05:26:23.7210175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:23.7211292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:23.7233958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:25.0375208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:25.0375804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:25.5889182Z ok (2.831s)
2022-05-18T05:26:25.6028893Z   test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105736
2022-05-18T05:26:25.6139423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105737
2022-05-18T05:26:26.5664088Z dist init r=0, world=2
2022-05-18T05:26:26.5667456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:26.5808032Z dist init r=1, world=2
2022-05-18T05:26:26.5812390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:26.5813190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:26.5871347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:27.9374101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:27.9374671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:28.4203260Z ok (2.831s)
2022-05-18T05:26:28.4341264Z   test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105819
2022-05-18T05:26:28.4452200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105820
2022-05-18T05:26:29.3567887Z dist init r=0, world=2
2022-05-18T05:26:29.3571275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:29.3590613Z dist init r=1, world=2
2022-05-18T05:26:29.3594910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:29.3595739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:29.3674293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:30.7011923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:30.7012461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:31.2516767Z ok (2.831s)
2022-05-18T05:26:31.2525253Z   test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-05-18T05:26:31.2531522Z   test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-05-18T05:26:31.2537648Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-05-18T05:26:31.2544146Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-05-18T05:26:31.2677636Z   test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105902
2022-05-18T05:26:31.2792055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105903
2022-05-18T05:26:32.2027660Z dist init r=0, world=2
2022-05-18T05:26:32.2030569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:32.2059032Z dist init r=1, world=2
2022-05-18T05:26:32.2063990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:32.2065007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:32.2133634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:33.5437004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:33.5437539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:34.0854402Z ok (2.831s)
2022-05-18T05:26:34.0990730Z   test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105985
2022-05-18T05:26:34.1100519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105986
2022-05-18T05:26:35.0182995Z dist init r=0, world=2
2022-05-18T05:26:35.0186426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:35.0747900Z dist init r=1, world=2
2022-05-18T05:26:35.0752962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:35.0754809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:35.0795539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:36.4004606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:26:36.4005161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:26:36.9163731Z ok (2.831s)
2022-05-18T05:26:36.9171070Z   test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-05-18T05:26:36.9176417Z   test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-05-18T05:26:36.9178084Z 
2022-05-18T05:26:36.9178606Z ----------------------------------------------------------------------
2022-05-18T05:26:36.9178960Z Ran 14 tests in 21.135s
2022-05-18T05:26:36.9181675Z 
2022-05-18T05:26:36.9182249Z OK (skipped=7)
2022-05-18T05:26:36.9182463Z 
2022-05-18T05:26:36.9182599Z Generating XML reports...
2022-05-18T05:26:36.9250575Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20220518052615.xml
2022-05-18T05:26:37.1924636Z Running distributed/elastic/multiprocessing/api_test ... [2022-05-18 05:26:37.191893]
2022-05-18T05:26:37.1925438Z Executing ['/opt/conda/bin/python', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:26:37.192000]
2022-05-18T05:26:38.1597635Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test
2022-05-18T05:26:38.1622398Z 
2022-05-18T05:26:38.1622945Z Running tests...
2022-05-18T05:26:38.1623449Z ----------------------------------------------------------------------
2022-05-18T05:26:39.7414482Z   test_get_failures (__main__.RunProcResultsTest) ... ok (1.579s)
2022-05-18T05:26:39.7425087Z   test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s)
2022-05-18T05:26:39.7444644Z   test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.002s)
2022-05-18T05:26:39.8045752Z   test_binary (__main__.StartProcessesListTest) ... hello stdout from 0
2022-05-18T05:26:39.8046107Z hello stderr from 0
2022-05-18T05:26:39.8257278Z hello stdout from 1
2022-05-18T05:26:39.8257549Z hello stderr from 1
2022-05-18T05:26:39.8931518Z ok (0.149s)
2022-05-18T05:26:39.9718494Z   test_binary_exit (__main__.StartProcessesListTest) ... bar stdout from 1
2022-05-18T05:26:39.9719117Z bar stderr from 1
2022-05-18T05:26:40.0393388Z failed (exitcode: 138) local_rank: 0 (pid: 106104) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2022-05-18T05:26:40.0408687Z ok (0.148s)
2022-05-18T05:26:40.0653524Z   test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.024s)
2022-05-18T05:26:40.1220098Z   test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last):
2022-05-18T05:26:40.1220640Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-05-18T05:26:40.1221048Z     raise RuntimeError(f"raised from {rank}")
2022-05-18T05:26:40.1221351Z RuntimeError: raised from 0
2022-05-18T05:26:40.1418961Z bar from 1
2022-05-18T05:26:40.2096420Z failed (exitcode: 1) local_rank: 0 (pid: 106107) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2022-05-18T05:26:40.2107522Z ok (0.145s)
2022-05-18T05:26:40.2891847Z   test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1
2022-05-18T05:26:40.3572183Z [trainer0]:hello stdout from 0
2022-05-18T05:26:40.3572502Z [trainer1]:world stderr from 1
2022-05-18T05:26:41.3603099Z ok (1.149s)
2022-05-18T05:26:42.3727160Z   test_function (__main__.StartProcessesListTest) ... hello stdout from 0
2022-05-18T05:26:42.3727767Z hello stderr from 0
2022-05-18T05:26:42.4048197Z hello stdout from 1
2022-05-18T05:26:42.4048755Z hello stderr from 1
2022-05-18T05:26:42.6253525Z Closing process 106114 via signal SIGTERM
2022-05-18T05:26:42.6311322Z ok (1.271s)
2022-05-18T05:26:44.2092332Z   test_function_large_ret_val (__main__.StartProcessesListTest) ... Closing process 106180 via signal SIGTERM
2022-05-18T05:26:44.2092829Z Closing process 106182 via signal SIGTERM
2022-05-18T05:26:44.2093155Z Closing process 106183 via signal SIGTERM
2022-05-18T05:26:44.2368805Z ok (1.605s)
2022-05-18T05:26:44.2390195Z   test_function_raise (__main__.StartProcessesListTest)
2022-05-18T05:26:45.3746525Z run 2x copies of echo2, raise an exception on the first ... failed (exitcode: 1) local_rank: 0 (pid: 106312) of fn: echo2 (start_method: spawn)
2022-05-18T05:26:45.3747015Z Traceback (most recent call last):
2022-05-18T05:26:45.3747737Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-05-18T05:26:45.3748183Z     self._pc.join(-1)
2022-05-18T05:26:45.3748657Z   File "/opt/conda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-05-18T05:26:45.3749117Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-05-18T05:26:45.3749575Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-05-18T05:26:45.3749851Z 
2022-05-18T05:26:45.3750118Z -- Process 0 terminated with the following error:
2022-05-18T05:26:45.3750461Z Traceback (most recent call last):
2022-05-18T05:26:45.3750942Z   File "/opt/conda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-05-18T05:26:45.3751302Z     fn(i, *args)
2022-05-18T05:26:45.3752047Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap
2022-05-18T05:26:45.3752450Z     ret = record(fn)(*args_)
2022-05-18T05:26:45.3752996Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
2022-05-18T05:26:45.3753404Z     return f(*args, **kwargs)
2022-05-18T05:26:45.3753784Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-05-18T05:26:45.3754167Z     raise RuntimeError(msg)
2022-05-18T05:26:45.3754443Z RuntimeError: hello
2022-05-18T05:26:45.3754614Z 
2022-05-18T05:26:45.3767277Z ok (1.140s)
2022-05-18T05:26:45.3793659Z   test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s)
2022-05-18T05:26:45.3809902Z   test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.002s)
2022-05-18T05:26:45.3963306Z   test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 106378 via signal SIGTERM
2022-05-18T05:26:45.4073813Z ok (0.026s)
2022-05-18T05:26:45.4120595Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-05-18T05:26:45.4121082Z Traceback (most recent call last):
2022-05-18T05:26:45.4121645Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-05-18T05:26:45.4122079Z     self._pc.join(-1)
2022-05-18T05:26:45.4122395Z   File "/opt/conda/lib/python3.9/unittest/mock.py", line 1092, in __call__
2022-05-18T05:26:45.4122747Z     return self._mock_call(*args, **kwargs)
2022-05-18T05:26:45.4123109Z   File "/opt/conda/lib/python3.9/unittest/mock.py", line 1096, in _mock_call
2022-05-18T05:26:45.4123705Z     return self._execute_mock_call(*args, **kwargs)
2022-05-18T05:26:45.4135090Z   File "/opt/conda/lib/python3.9/unittest/mock.py", line 1151, in _execute_mock_call
2022-05-18T05:26:45.4135434Z     raise effect
2022-05-18T05:26:45.4135836Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-05-18T05:26:45.4136216Z ok (0.006s)
2022-05-18T05:26:47.5415648Z   test_pcontext_wait (__main__.StartProcessesListTest) ... ok (2.128s)
2022-05-18T05:26:47.5664131Z   test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 106412 closing signal SIGTERM
2022-05-18T05:26:47.5682830Z ok (0.027s)
2022-05-18T05:26:47.5705262Z   test_to_map (__main__.StartProcessesListTest) ... ok (0.002s)
2022-05-18T05:26:47.5713073Z   test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s)
2022-05-18T05:26:48.5615754Z   test_void_function (__main__.StartProcessesListTest) ... world
2022-05-18T05:26:48.6150212Z hello
2022-05-18T05:26:48.8206701Z Closing process 106413 via signal SIGTERM
2022-05-18T05:26:48.8236593Z ok (1.252s)
2022-05-18T05:26:48.8261801Z   test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s)
2022-05-18T05:26:48.9056191Z   test_binary_exit (__main__.StartProcessesTest) ... bar stdout from 1
2022-05-18T05:26:48.9056569Z bar stderr from 1
2022-05-18T05:26:48.9725265Z failed (exitcode: 138) local_rank: 0 (pid: 106479) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2022-05-18T05:26:48.9738126Z ok (0.148s)
2022-05-18T05:26:48.9974070Z   test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.023s)
2022-05-18T05:26:49.0548824Z   test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last):
2022-05-18T05:26:49.0549348Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-05-18T05:26:49.0549773Z     raise RuntimeError(f"raised from {rank}")
2022-05-18T05:26:49.0550098Z RuntimeError: raised from 0
2022-05-18T05:26:49.0741462Z bar from 1
2022-05-18T05:26:49.1415310Z failed (exitcode: 1) local_rank: 0 (pid: 106482) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2022-05-18T05:26:49.1425743Z ok (0.145s)
2022-05-18T05:26:50.7189869Z   test_function_large_ret_val (__main__.StartProcessesTest) ... Closing process 106484 via signal SIGTERM
2022-05-18T05:26:50.7190329Z Closing process 106487 via signal SIGTERM
2022-05-18T05:26:50.7443467Z ok (1.601s)
2022-05-18T05:26:50.7461898Z   test_function_raise (__main__.StartProcessesTest)
2022-05-18T05:26:51.8816991Z run 2x copies of echo2, raise an exception on the first ... failed (exitcode: 1) local_rank: 0 (pid: 106616) of fn: echo2 (start_method: spawn)
2022-05-18T05:26:51.8817468Z Traceback (most recent call last):
2022-05-18T05:26:51.8818179Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-05-18T05:26:51.8818621Z     self._pc.join(-1)
2022-05-18T05:26:51.8819103Z   File "/opt/conda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-05-18T05:26:51.8819576Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-05-18T05:26:51.8820042Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-05-18T05:26:51.8820302Z 
2022-05-18T05:26:51.8820536Z -- Process 0 terminated with the following error:
2022-05-18T05:26:51.8820869Z Traceback (most recent call last):
2022-05-18T05:26:51.8821368Z   File "/opt/conda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-05-18T05:26:51.8821715Z     fn(i, *args)
2022-05-18T05:26:51.8822225Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap
2022-05-18T05:26:51.8822625Z     ret = record(fn)(*args_)
2022-05-18T05:26:51.8823177Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
2022-05-18T05:26:51.8823847Z     return f(*args, **kwargs)
2022-05-18T05:26:51.8824249Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-05-18T05:26:51.8824639Z     raise RuntimeError(msg)
2022-05-18T05:26:51.8824894Z RuntimeError: hello
2022-05-18T05:26:51.8825068Z 
2022-05-18T05:26:51.8840536Z ok (1.140s)
2022-05-18T05:26:51.8867573Z   test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.002s)
2022-05-18T05:26:51.8884153Z   test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.002s)
2022-05-18T05:26:51.9028609Z   test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 106682 via signal SIGTERM
2022-05-18T05:26:51.9136936Z ok (0.025s)
2022-05-18T05:26:51.9175256Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-05-18T05:26:51.9175759Z Traceback (most recent call last):
2022-05-18T05:26:51.9176289Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-05-18T05:26:51.9176722Z     self._pc.join(-1)
2022-05-18T05:26:51.9177057Z   File "/opt/conda/lib/python3.9/unittest/mock.py", line 1092, in __call__
2022-05-18T05:26:51.9177399Z     return self._mock_call(*args, **kwargs)
2022-05-18T05:26:51.9177763Z   File "/opt/conda/lib/python3.9/unittest/mock.py", line 1096, in _mock_call
2022-05-18T05:26:51.9178132Z     return self._execute_mock_call(*args, **kwargs)
2022-05-18T05:26:51.9178519Z   File "/opt/conda/lib/python3.9/unittest/mock.py", line 1151, in _execute_mock_call
2022-05-18T05:26:51.9178837Z     raise effect
2022-05-18T05:26:51.9179214Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-05-18T05:26:51.9183936Z ok (0.004s)
2022-05-18T05:26:54.0519616Z   test_pcontext_wait (__main__.StartProcessesTest) ... ok (2.133s)
2022-05-18T05:26:54.0765496Z   test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 106716 closing signal SIGTERM
2022-05-18T05:26:54.0782319Z ok (0.026s)
2022-05-18T05:26:54.0804122Z   test_to_map (__main__.StartProcessesTest) ... ok (0.002s)
2022-05-18T05:26:54.0812255Z   test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s)
2022-05-18T05:26:55.0817614Z   test_void_function (__main__.StartProcessesTest) ... world
2022-05-18T05:26:55.1121905Z hello
2022-05-18T05:26:55.3312797Z Closing process 106718 via signal SIGTERM
2022-05-18T05:26:55.3366872Z ok (1.255s)
2022-05-18T05:26:55.3385550Z   test_from_str_bad_input (__main__.StdTest) ... ok (0.002s)
2022-05-18T05:26:55.3396617Z   test_from_value (__main__.StdTest) ... ok (0.001s)
2022-05-18T05:26:55.3407008Z   test_from_value_map (__main__.StdTest) ... ok (0.001s)
2022-05-18T05:26:55.3407494Z 
2022-05-18T05:26:55.3408136Z ----------------------------------------------------------------------
2022-05-18T05:26:55.3408522Z Ran 38 tests in 17.179s
2022-05-18T05:26:55.3408721Z 
2022-05-18T05:26:55.3408817Z OK
2022-05-18T05:26:55.3408936Z 
2022-05-18T05:26:55.3409074Z Generating XML reports...
2022-05-18T05:26:55.3450504Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220518052638.xml
2022-05-18T05:26:55.3473850Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220518052638.xml
2022-05-18T05:26:55.3494199Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220518052638.xml
2022-05-18T05:26:55.3499070Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220518052638.xml
2022-05-18T05:26:55.6156335Z Running distributed/fsdp/test_fsdp_checkpoint ... [2022-05-18 05:26:55.615134]
2022-05-18T05:26:55.6157065Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:26:55.615240]
2022-05-18T05:26:56.5248588Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint
2022-05-18T05:26:56.5265961Z 
2022-05-18T05:26:56.5266278Z Running tests...
2022-05-18T05:26:56.5266745Z ----------------------------------------------------------------------
2022-05-18T05:26:58.1076825Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:26:58.1485422Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106818
2022-05-18T05:26:58.1604354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106819
2022-05-18T05:26:59.0847153Z dist init r=1, world=2
2022-05-18T05:26:59.0850879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:26:59.1082975Z dist init r=0, world=2
2022-05-18T05:26:59.1087322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:26:59.1088422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:26:59.1157080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:00.4586133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:01.0682337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:01.0682764Z ok (4.541s)
2022-05-18T05:27:01.0710800Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71418 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.003s)
2022-05-18T05:27:01.0869999Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106901
2022-05-18T05:27:01.0987399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106902
2022-05-18T05:27:02.0150815Z dist init r=0, world=2
2022-05-18T05:27:02.0153648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:02.0189770Z dist init r=1, world=2
2022-05-18T05:27:02.0194276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:02.0195100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:02.0256803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:03.3717912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:03.3718823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:04.0052628Z ok (2.934s)
2022-05-18T05:27:04.0081196Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/70368 for platform(s) win, linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.003s)
2022-05-18T05:27:04.0227881Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106984
2022-05-18T05:27:04.0337906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106985
2022-05-18T05:27:04.9487442Z dist init r=1, world=2
2022-05-18T05:27:04.9490280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:04.9952670Z dist init r=0, world=2
2022-05-18T05:27:04.9957281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:04.9958348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:04.9998450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:06.3249571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:06.3250131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:06.8400916Z ok (2.832s)
2022-05-18T05:27:06.8424739Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71009 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-05-18T05:27:06.8571875Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107067
2022-05-18T05:27:06.8686587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107068
2022-05-18T05:27:07.7943291Z dist init r=0, world=2
2022-05-18T05:27:07.7946358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:07.8391869Z dist init r=1, world=2
2022-05-18T05:27:07.8396506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:07.8397346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:07.8454157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:09.1820727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:09.1821278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:09.7752757Z ok (2.933s)
2022-05-18T05:27:09.7776607Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71349 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-05-18T05:27:09.7777357Z 
2022-05-18T05:27:09.7777640Z ----------------------------------------------------------------------
2022-05-18T05:27:09.7777976Z Ran 8 tests in 13.251s
2022-05-18T05:27:09.7778146Z 
2022-05-18T05:27:09.7778258Z OK (skipped=4)
2022-05-18T05:27:09.7778396Z 
2022-05-18T05:27:09.7778526Z Generating XML reports...
2022-05-18T05:27:09.7840641Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20220518052656.xml
2022-05-18T05:27:10.0508692Z Running distributed/fsdp/test_fsdp_overlap ... [2022-05-18 05:27:10.050365]
2022-05-18T05:27:10.0509768Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:27:10.050469]
2022-05-18T05:27:10.9822381Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap
2022-05-18T05:27:10.9841510Z 
2022-05-18T05:27:10.9841907Z Running tests...
2022-05-18T05:27:10.9842355Z ----------------------------------------------------------------------
2022-05-18T05:27:12.5684562Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:12.6090421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107185
2022-05-18T05:27:13.5213245Z dist init r=0, world=1
2022-05-18T05:27:13.5216350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:13.5217172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T05:27:14.8095267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:14.8950672Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:27:14.8951276Z   warnings.warn(
2022-05-18T05:27:18.6665382Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:27:18.6666744Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:27:18.6668323Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:27:18.6669627Z [W python_variable.cpp:205] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
2022-05-18T05:27:23.1335955Z 
2022-05-18T05:27:23.1336344Z rank0:
2022-05-18T05:27:23.1337219Z   e1: {'cpu_iter': 0.002030419200000022, 'cpu_wait': 3.549889999998612e-05, 'gpu_compute': 0.07052159998565913, 'gpu_total': 1.2292672038078307}
2022-05-18T05:27:23.1337834Z   e2: {'cpu_iter': 0.003710818700000118, 'cpu_wait': 3.484319999982333e-05, 'gpu_compute': 0.24235520027577878, 'gpu_total': 1.9469536185264587}
2022-05-18T05:27:23.1338421Z   e3: {'cpu_iter': 0.0021220295000000887, 'cpu_wait': 0.1917159844999996, 'gpu_compute': 193.396492767334, 'gpu_total': 193.86847381591798}
2022-05-18T05:27:23.1338989Z   e4: {'cpu_iter': 0.003771497499999832, 'cpu_wait': 0.19028192430000015, 'gpu_compute': 193.41612281799317, 'gpu_total': 193.98742065429687}
2022-05-18T05:27:23.3270915Z ok (12.343s)
2022-05-18T05:27:23.3283775Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-05-18T05:27:23.3284776Z 
2022-05-18T05:27:23.3285071Z ----------------------------------------------------------------------
2022-05-18T05:27:23.3285413Z Ran 2 tests in 12.344s
2022-05-18T05:27:23.3285580Z 
2022-05-18T05:27:23.3285693Z OK (skipped=1)
2022-05-18T05:27:23.3285832Z 
2022-05-18T05:27:23.3285966Z Generating XML reports...
2022-05-18T05:27:23.3337438Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220518052710.xml
2022-05-18T05:27:23.3341275Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220518052710.xml
2022-05-18T05:27:23.6008178Z Running distributed/fsdp/test_fsdp_ignored_modules ... [2022-05-18 05:27:23.600277]
2022-05-18T05:27:23.6009350Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_ignored_modules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:27:23.600379]
2022-05-18T05:27:24.5157749Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules
2022-05-18T05:27:24.5174479Z 
2022-05-18T05:27:24.5175119Z Running tests...
2022-05-18T05:27:24.5175643Z ----------------------------------------------------------------------
2022-05-18T05:27:24.5183764Z   test_ignored_modules_invalid (__main__.TestFSDPIgnoredModules)
2022-05-18T05:27:26.0472561Z Tests that passing an FSDP module as an ignored module or the ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:26.0869405Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107262
2022-05-18T05:27:26.0981907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107263
2022-05-18T05:27:27.0165894Z dist init r=1, world=2
2022-05-18T05:27:27.0166228Z dist init r=0, world=2
2022-05-18T05:27:27.0169205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:27.0170452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:27.0171032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:27.0172414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:28.3939756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:28.3940318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:28.4148251Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:27:28.4148842Z   warnings.warn(
2022-05-18T05:27:28.4181872Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:27:28.4182455Z   warnings.warn(
2022-05-18T05:27:28.7046298Z ok (4.187s)
2022-05-18T05:27:28.7061678Z   test_ignored_modules_nested (__main__.TestFSDPIgnoredModules)
2022-05-18T05:27:28.7192078Z Tests that passing a module with nested FSDP modules does not ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107341
2022-05-18T05:27:28.7301785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107342
2022-05-18T05:27:29.7129820Z dist init r=1, world=2
2022-05-18T05:27:29.7133058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:29.7199023Z dist init r=0, world=2
2022-05-18T05:27:29.7203988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:29.7205451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:29.7235758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:31.0797042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:31.0797584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:31.6364558Z ok (2.932s)
2022-05-18T05:27:31.6379483Z   test_ignored_modules_transformer (__main__.TestFSDPIgnoredModules)
2022-05-18T05:27:31.6509799Z Tests that ignored modules' parameters are not flattened for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107424
2022-05-18T05:27:31.6619197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107425
2022-05-18T05:27:32.5965104Z dist init r=0, world=2
2022-05-18T05:27:32.5968428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:32.6252821Z dist init r=1, world=2
2022-05-18T05:27:32.6257294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:32.6258361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:32.6273519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:33.9690019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:33.9690565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:34.0005806Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:27:34.0006387Z   warnings.warn(
2022-05-18T05:27:34.0007420Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:27:34.0007994Z   warnings.warn(
2022-05-18T05:27:35.1693406Z ok (3.533s)
2022-05-18T05:27:35.1693786Z 
2022-05-18T05:27:35.1694546Z ----------------------------------------------------------------------
2022-05-18T05:27:35.1695188Z Ran 3 tests in 10.652s
2022-05-18T05:27:35.1695364Z 
2022-05-18T05:27:35.1695442Z OK
2022-05-18T05:27:35.1695580Z 
2022-05-18T05:27:35.1695715Z Generating XML reports...
2022-05-18T05:27:35.1764897Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20220518052724.xml
2022-05-18T05:27:35.4428906Z Running distributed/fsdp/test_fsdp_apply ... [2022-05-18 05:27:35.442274]
2022-05-18T05:27:35.4429774Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:27:35.442383]
2022-05-18T05:27:36.3659979Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply
2022-05-18T05:27:36.3676771Z 
2022-05-18T05:27:36.3677135Z Running tests...
2022-05-18T05:27:36.3677659Z ----------------------------------------------------------------------
2022-05-18T05:27:36.3685319Z   test_apply_in_summon_raises_error (__main__.TestApply)
2022-05-18T05:27:37.9332798Z Ensures that if user calls apply() on FSDP instance within full param ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:37.9728056Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107542
2022-05-18T05:27:37.9843954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107543
2022-05-18T05:27:38.8934490Z dist init r=1, world=2
2022-05-18T05:27:38.8937345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:38.8992746Z dist init r=0, world=2
2022-05-18T05:27:38.8997471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:38.8998886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:38.9040932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:40.2556633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:40.2557227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:40.2880786Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:27:40.2881405Z   warnings.warn(
2022-05-18T05:27:40.2911896Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:27:40.2912449Z   warnings.warn(
2022-05-18T05:27:40.2993800Z Asserting FSDP instance is: FullyShardedDataParallel(
2022-05-18T05:27:40.2994591Z   (_fsdp_wrapped_module): FlattenParamsWrapper(
2022-05-18T05:27:40.2995378Z     (_fpw_module): TransformerWithSharedParams(
2022-05-18T05:27:40.2995770Z       (embed_tokens): Embedding(23, 16)
2022-05-18T05:27:40.2996072Z       (transformer): Transformer(
2022-05-18T05:27:40.2996376Z         (encoder): TransformerEncoder(
2022-05-18T05:27:40.2996648Z           (layers): ModuleList(
2022-05-18T05:27:40.2996952Z             (0): TransformerEncoderLayer(
2022-05-18T05:27:40.2997583Z               (self_attn): MultiheadAttention(
2022-05-18T05:27:40.2998712Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-05-18T05:27:40.2999471Z               )
2022-05-18T05:27:40.3000305Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-05-18T05:27:40.3001160Z               (dropout): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3001784Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-05-18T05:27:40.3002281Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-05-18T05:27:40.3002751Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-05-18T05:27:40.3003095Z               (dropout1): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3003437Z               (dropout2): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3003729Z             )
2022-05-18T05:27:40.3004004Z             (1): TransformerEncoderLayer(
2022-05-18T05:27:40.3004307Z               (self_attn): MultiheadAttention(
2022-05-18T05:27:40.3004742Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-05-18T05:27:40.3005106Z               )
2022-05-18T05:27:40.3005401Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-05-18T05:27:40.3005758Z               (dropout): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3006336Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-05-18T05:27:40.3006979Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-05-18T05:27:40.3007736Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-05-18T05:27:40.3008350Z               (dropout1): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3008824Z               (dropout2): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3009553Z             )
2022-05-18T05:27:40.3010046Z           )
2022-05-18T05:27:40.3010639Z           (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-05-18T05:27:40.3010923Z         )
2022-05-18T05:27:40.3011191Z         (decoder): TransformerDecoder(
2022-05-18T05:27:40.3011517Z           (layers): ModuleList(
2022-05-18T05:27:40.3011823Z             (0): TransformerDecoderLayer(
2022-05-18T05:27:40.3012128Z               (self_attn): MultiheadAttention(
2022-05-18T05:27:40.3012547Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-05-18T05:27:40.3012905Z               )
2022-05-18T05:27:40.3013171Z               (multihead_attn): MultiheadAttention(
2022-05-18T05:27:40.3013594Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-05-18T05:27:40.3013948Z               )
2022-05-18T05:27:40.3014236Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-05-18T05:27:40.3014597Z               (dropout): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3014950Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-05-18T05:27:40.3015417Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-05-18T05:27:40.3015860Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-05-18T05:27:40.3016313Z               (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-05-18T05:27:40.3016662Z               (dropout1): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3016978Z               (dropout2): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3017306Z               (dropout3): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3017583Z             )
2022-05-18T05:27:40.3017840Z             (1): TransformerDecoderLayer(
2022-05-18T05:27:40.3018158Z               (self_attn): MultiheadAttention(
2022-05-18T05:27:40.3018572Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-05-18T05:27:40.3018926Z               )
2022-05-18T05:27:40.3019190Z               (multihead_attn): MultiheadAttention(
2022-05-18T05:27:40.3019608Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2022-05-18T05:27:40.3020037Z               )
2022-05-18T05:27:40.3020340Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2022-05-18T05:27:40.3020698Z               (dropout): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3021056Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2022-05-18T05:27:40.3021502Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-05-18T05:27:40.3021958Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-05-18T05:27:40.3022411Z               (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-05-18T05:27:40.3022756Z               (dropout1): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3023082Z               (dropout2): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3023410Z               (dropout3): Dropout(p=0.1, inplace=False)
2022-05-18T05:27:40.3023683Z             )
2022-05-18T05:27:40.3023891Z           )
2022-05-18T05:27:40.3024275Z           (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2022-05-18T05:27:40.3024570Z         )
2022-05-18T05:27:40.3024774Z       )
2022-05-18T05:27:40.3025078Z       (output_proj): Linear(in_features=16, out_features=23, bias=True)
2022-05-18T05:27:40.3025578Z       (bn): BatchNorm1d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
2022-05-18T05:27:40.3025949Z     )
2022-05-18T05:27:40.3026331Z   )
2022-05-18T05:27:40.3026709Z )
2022-05-18T05:27:40.3027353Z ERROR: expected to be in states [<TrainingState_.IDLE: 1>] but current state is TrainingState_.SUMMON_FULL_PARAMS
2022-05-18T05:27:40.3027771Z   File "<string>", line 1, in <module>
2022-05-18T05:27:40.3028254Z   File "/opt/conda/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
2022-05-18T05:27:40.3028630Z     exitcode = _main(fd, parent_sentinel)
2022-05-18T05:27:40.3028983Z   File "/opt/conda/lib/python3.9/multiprocessing/spawn.py", line 129, in _main
2022-05-18T05:27:40.3029353Z     return self._bootstrap(parent_sentinel)
2022-05-18T05:27:40.3029752Z   File "/opt/conda/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2022-05-18T05:27:40.3030073Z     self.run()
2022-05-18T05:27:40.3030404Z   File "/opt/conda/lib/python3.9/multiprocessing/process.py", line 108, in run
2022-05-18T05:27:40.3030771Z     self._target(*self._args, **self._kwargs)
2022-05-18T05:27:40.3031288Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_fsdp.py", line 429, in _run
2022-05-18T05:27:40.3031687Z     self.run_test(test_name, pipe)
2022-05-18T05:27:40.3032216Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 618, in run_test
2022-05-18T05:27:40.3032619Z     getattr(self, test_name)()
2022-05-18T05:27:40.3033115Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 499, in wrapper
2022-05-18T05:27:40.3033483Z     fn()
2022-05-18T05:27:40.3033961Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 141, in wrapper
2022-05-18T05:27:40.3034343Z     return func(*args, **kwargs)
2022-05-18T05:27:40.3034762Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_apply.py", line 100, in test_apply_in_summon_raises_error
2022-05-18T05:27:40.3035200Z     transformer.apply(self._init_linear_weights)
2022-05-18T05:27:40.3035767Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1104, in apply
2022-05-18T05:27:40.3036186Z     self._assert_state(TrainingState_.IDLE)
2022-05-18T05:27:40.3036753Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3298, in _assert_state
2022-05-18T05:27:40.3037171Z     traceback.print_stack()
2022-05-18T05:27:40.5908673Z ok (4.223s)
2022-05-18T05:27:40.5915979Z   test_nested_module_apply (__main__.TestApply)
2022-05-18T05:27:40.6047386Z Checks apply() modifies weights appropriately on a nested FSDP instance. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107621
2022-05-18T05:27:40.6157260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107622
2022-05-18T05:27:41.5375399Z dist init r=0, world=2
2022-05-18T05:27:41.5378677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:41.5531448Z dist init r=1, world=2
2022-05-18T05:27:41.5536346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:41.5537145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:41.5583355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:42.8877910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:42.8878758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:42.9100656Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:27:42.9101331Z   warnings.warn(
2022-05-18T05:27:42.9135690Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:27:42.9136229Z   warnings.warn(
2022-05-18T05:27:43.3218277Z ok (2.731s)
2022-05-18T05:27:43.3224463Z   test_transformer_module_apply (__main__.TestApply)
2022-05-18T05:27:43.3355422Z Checks apply() modifies weights appropriately on a wrapped Transformer ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107700
2022-05-18T05:27:43.3470897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107701
2022-05-18T05:27:44.2742047Z dist init r=1, world=2
2022-05-18T05:27:44.2745171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:44.2898015Z dist init r=0, world=2
2022-05-18T05:27:44.2902707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:44.2903759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:44.2949207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:45.6357405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:45.6357946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:45.6676502Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:27:45.6677061Z   warnings.warn(
2022-05-18T05:27:45.6707353Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:27:45.6707906Z   warnings.warn(
2022-05-18T05:27:46.3536464Z ok (3.032s)
2022-05-18T05:27:46.3536785Z 
2022-05-18T05:27:46.3537365Z ----------------------------------------------------------------------
2022-05-18T05:27:46.3537717Z Ran 3 tests in 9.986s
2022-05-18T05:27:46.3537885Z 
2022-05-18T05:27:46.3537983Z OK
2022-05-18T05:27:46.3538124Z 
2022-05-18T05:27:46.3538241Z Generating XML reports...
2022-05-18T05:27:46.3598623Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20220518052736.xml
2022-05-18T05:27:46.6271462Z Running distributed/fsdp/test_distributed_checkpoint ... [2022-05-18 05:27:46.626640]
2022-05-18T05:27:46.6272251Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:27:46.626746]
2022-05-18T05:27:47.5577672Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint
2022-05-18T05:27:47.5594236Z 
2022-05-18T05:27:47.5594772Z Running tests...
2022-05-18T05:27:47.5595444Z ----------------------------------------------------------------------
2022-05-18T05:27:49.1451986Z   test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:49.1865234Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107814
2022-05-18T05:27:49.1983845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107815
2022-05-18T05:27:50.1472850Z dist init r=0, world=2
2022-05-18T05:27:50.1476276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:50.1605705Z dist init r=1, world=2
2022-05-18T05:27:50.1609886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:50.1610706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:50.1680429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:51.5182654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:51.5183386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:51.8047216Z ok (4.245s)
2022-05-18T05:27:51.8195584Z   test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107893
2022-05-18T05:27:51.8308032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107894
2022-05-18T05:27:52.7451437Z dist init r=1, world=2
2022-05-18T05:27:52.7454157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:27:52.7531924Z dist init r=0, world=2
2022-05-18T05:27:52.7536424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:27:52.7537240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:52.7556512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:27:54.0866501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:54.0867071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:54.4370697Z ok (2.632s)
2022-05-18T05:27:54.4371070Z 
2022-05-18T05:27:54.4371822Z ----------------------------------------------------------------------
2022-05-18T05:27:54.4372400Z Ran 2 tests in 6.878s
2022-05-18T05:27:54.4372572Z 
2022-05-18T05:27:54.4372667Z OK
2022-05-18T05:27:54.4372805Z 
2022-05-18T05:27:54.4416489Z Generating XML reports...
2022-05-18T05:27:54.4417190Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20220518052747.xml
2022-05-18T05:27:54.7118640Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp ... [2022-05-18 05:27:54.711311]
2022-05-18T05:27:54.7119727Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:27:54.711415]
2022-05-18T05:27:55.6118696Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp
2022-05-18T05:27:55.6135287Z 
2022-05-18T05:27:55.6135523Z Running tests...
2022-05-18T05:27:55.6136230Z ----------------------------------------------------------------------
2022-05-18T05:27:55.6146688Z   test_torch_allclose (__main__.TestShardedTensorBinaryOps)
2022-05-18T05:27:57.2088857Z Test torch.allclose(ShardedTensor, ShardedTensor) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:27:57.2493748Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108007
2022-05-18T05:27:57.2608945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108008
2022-05-18T05:27:57.2729798Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108009
2022-05-18T05:27:57.2861521Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108010
2022-05-18T05:27:58.1797572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:58.2106539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:58.2312353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:27:58.2502626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:27:58.4902305Z skip: Need at least 4 CUDA devices (2.876s)
2022-05-18T05:27:58.5039443Z   test_torch_allclose_tensor_specs (__main__.TestShardedTensorBinaryOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108143
2022-05-18T05:27:58.5152238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108144
2022-05-18T05:27:58.5267716Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108145
2022-05-18T05:27:58.5388198Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108146
2022-05-18T05:27:59.5420961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:27:59.5452469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:27:59.5758578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:27:59.6065591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:27:59.8427442Z skip: Need at least 4 CUDA devices (1.352s)
2022-05-18T05:27:59.8434844Z   test_torch_equal (__main__.TestShardedTensorBinaryOps)
2022-05-18T05:27:59.8567486Z Test torch.equal(ShardedTensor, ShardedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108279
2022-05-18T05:27:59.8678395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108280
2022-05-18T05:27:59.8794580Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108281
2022-05-18T05:27:59.8915129Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108282
2022-05-18T05:28:00.8570177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:00.8624056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:00.9124258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:00.9186639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:01.0953198Z skip: Need at least 4 CUDA devices (1.252s)
2022-05-18T05:28:01.1089913Z   test_torch_equal_tensor_specs (__main__.TestShardedTensorBinaryOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108415
2022-05-18T05:28:01.1199728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108416
2022-05-18T05:28:01.1313779Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108417
2022-05-18T05:28:01.1432183Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108418
2022-05-18T05:28:02.0373784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:02.0493628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:02.1107128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:02.1107669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:02.3468729Z skip: Need at least 4 CUDA devices (1.251s)
2022-05-18T05:28:02.3468984Z 
2022-05-18T05:28:02.3469382Z ----------------------------------------------------------------------
2022-05-18T05:28:02.3469724Z Ran 4 tests in 6.733s
2022-05-18T05:28:02.3469872Z 
2022-05-18T05:28:02.3469999Z OK (skipped=4)
2022-05-18T05:28:02.3470156Z 
2022-05-18T05:28:02.3470289Z Generating XML reports...
2022-05-18T05:28:02.3517235Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20220518052755.xml
2022-05-18T05:28:02.6212810Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-05-18 05:28:02.620798]
2022-05-18T05:28:02.6213602Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:28:02.620901]
2022-05-18T05:28:03.5078463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0lkm8z9
2022-05-18T05:28:03.5080203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0lkm8z9/_remote_module_non_scriptable.py
2022-05-18T05:28:03.5258806Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec
2022-05-18T05:28:03.5278643Z 
2022-05-18T05:28:03.5279121Z Running tests...
2022-05-18T05:28:03.5279654Z ----------------------------------------------------------------------
2022-05-18T05:28:05.1041056Z   test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:05.1440934Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108586
2022-05-18T05:28:05.1553943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108587
2022-05-18T05:28:05.1673076Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108588
2022-05-18T05:28:05.1793313Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108589
2022-05-18T05:28:06.0633225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7w6nbo2
2022-05-18T05:28:06.0634458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7w6nbo2/_remote_module_non_scriptable.py
2022-05-18T05:28:06.0784012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:06.1261213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3mkvlk3
2022-05-18T05:28:06.1262204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3mkvlk3/_remote_module_non_scriptable.py
2022-05-18T05:28:06.1335957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1jx108c
2022-05-18T05:28:06.1338630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1jx108c/_remote_module_non_scriptable.py
2022-05-18T05:28:06.1366116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2fos982
2022-05-18T05:28:06.1369160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2fos982/_remote_module_non_scriptable.py
2022-05-18T05:28:06.1418939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:06.1487198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:06.1524386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:06.3835258Z ok (2.855s)
2022-05-18T05:28:06.3847106Z   test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec)
2022-05-18T05:28:06.3979594Z Test custom spec can be invoked from the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108722
2022-05-18T05:28:06.4088097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108723
2022-05-18T05:28:06.4201782Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108724
2022-05-18T05:28:06.4319404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108725
2022-05-18T05:28:07.3249609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2my68shf
2022-05-18T05:28:07.3250237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2my68shf/_remote_module_non_scriptable.py
2022-05-18T05:28:07.3398146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:07.3864880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4g691p7_
2022-05-18T05:28:07.3865470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4g691p7_/_remote_module_non_scriptable.py
2022-05-18T05:28:07.4010030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwe_jzcp7
2022-05-18T05:28:07.4012509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwe_jzcp7/_remote_module_non_scriptable.py
2022-05-18T05:28:07.4013308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:07.4151642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3alzf4pj
2022-05-18T05:28:07.4153564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3alzf4pj/_remote_module_non_scriptable.py
2022-05-18T05:28:07.4162076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:07.4312346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:07.6357062Z skip: Need at least 4 CUDA devices (1.252s)
2022-05-18T05:28:07.6368356Z   test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec)
2022-05-18T05:28:07.6499072Z Test sharded_tensor.ones(...) with the custom ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108858
2022-05-18T05:28:07.6610277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108859
2022-05-18T05:28:07.6724263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108860
2022-05-18T05:28:07.6840764Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108861
2022-05-18T05:28:08.5708717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2b0zyvh
2022-05-18T05:28:08.5709980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2b0zyvh/_remote_module_non_scriptable.py
2022-05-18T05:28:08.5857229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:08.5965259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7pz80rmn
2022-05-18T05:28:08.5967604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7pz80rmn/_remote_module_non_scriptable.py
2022-05-18T05:28:08.6117431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:08.6276853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp9c8d0ft
2022-05-18T05:28:08.6278643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp9c8d0ft/_remote_module_non_scriptable.py
2022-05-18T05:28:08.6412034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxcdkwr4
2022-05-18T05:28:08.6414554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxcdkwr4/_remote_module_non_scriptable.py
2022-05-18T05:28:08.6427646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:08.6568262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:08.8877932Z skip: Need at least 4 CUDA devices (1.252s)
2022-05-18T05:28:08.9018261Z   test_chunked_sharding_spec (__main__.TestShardingSpec) ... ok (0.014s)
2022-05-18T05:28:08.9096092Z   test_device_placement (__main__.TestShardingSpec) ... ok (0.008s)
2022-05-18T05:28:08.9182804Z   test_enumerable_sharding_spec (__main__.TestShardingSpec) ... ok (0.009s)
2022-05-18T05:28:08.9203930Z   test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s)
2022-05-18T05:28:08.9215949Z   test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-05-18T05:28:08.9228496Z   test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-05-18T05:28:08.9332894Z   test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.010s)
2022-05-18T05:28:08.9333528Z 
2022-05-18T05:28:08.9334250Z ----------------------------------------------------------------------
2022-05-18T05:28:08.9334652Z Ran 10 tests in 5.405s
2022-05-18T05:28:08.9334804Z 
2022-05-18T05:28:08.9334922Z OK (skipped=2)
2022-05-18T05:28:08.9335082Z 
2022-05-18T05:28:08.9335212Z Generating XML reports...
2022-05-18T05:28:08.9378968Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220518052803.xml
2022-05-18T05:28:08.9389204Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220518052803.xml
2022-05-18T05:28:09.2167564Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2022-05-18 05:28:09.216184]
2022-05-18T05:28:09.2168395Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:28:09.216287]
2022-05-18T05:28:10.1233916Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops
2022-05-18T05:28:10.1251066Z 
2022-05-18T05:28:10.1251399Z Running tests...
2022-05-18T05:28:10.1251863Z ----------------------------------------------------------------------
2022-05-18T05:28:11.7176932Z   test_sharded_dropout (__main__.TestShardedTensorElementWiseOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:11.7582285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109029
2022-05-18T05:28:11.7699764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109030
2022-05-18T05:28:11.7818996Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109031
2022-05-18T05:28:11.7939963Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109032
2022-05-18T05:28:12.7770648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:12.7791709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:12.7813729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:12.8059184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:12.9983909Z skip: Need at least 4 CUDA devices (2.873s)
2022-05-18T05:28:13.0125802Z   test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109165
2022-05-18T05:28:13.0236775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109166
2022-05-18T05:28:13.0351347Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109167
2022-05-18T05:28:13.0467895Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109168
2022-05-18T05:28:14.0136815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:14.0342565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:14.0469073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:14.0603535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:14.2508374Z skip: Need at least 4 CUDA devices (1.252s)
2022-05-18T05:28:14.2647482Z   test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109301
2022-05-18T05:28:14.2757524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109302
2022-05-18T05:28:14.2873752Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109303
2022-05-18T05:28:14.2993621Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109304
2022-05-18T05:28:15.2757034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:15.2811035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:15.3276714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:15.3557779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:15.5029433Z skip: Need at least 4 CUDA devices (1.252s)
2022-05-18T05:28:15.5030136Z 
2022-05-18T05:28:15.5030532Z ----------------------------------------------------------------------
2022-05-18T05:28:15.5030859Z Ran 3 tests in 5.378s
2022-05-18T05:28:15.5031026Z 
2022-05-18T05:28:15.5031141Z OK (skipped=3)
2022-05-18T05:28:15.5031298Z 
2022-05-18T05:28:15.5031427Z Generating XML reports...
2022-05-18T05:28:15.5077092Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220518052810.xml
2022-05-18T05:28:15.7705902Z Running distributed/_shard/sharded_tensor/ops/test_init ... [2022-05-18 05:28:15.770088]
2022-05-18T05:28:15.7706676Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_init.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:28:15.770190]
2022-05-18T05:28:16.6635273Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init
2022-05-18T05:28:16.6652081Z 
2022-05-18T05:28:16.6652423Z Running tests...
2022-05-18T05:28:16.6652882Z ----------------------------------------------------------------------
2022-05-18T05:28:16.6667656Z   test_init_sharded_tensor_with_kaiming_uniform (__main__.TestShardedTensorNNInit)
2022-05-18T05:28:18.2028058Z Test torch.nn.init.kaiming_uniform_(ShardedTensor, a, mode, nonlinearit) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:18.2425899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109472
2022-05-18T05:28:18.2541472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109473
2022-05-18T05:28:18.2658207Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109474
2022-05-18T05:28:18.2777753Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109475
2022-05-18T05:28:19.2433610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:19.2916332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:19.3068130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:19.3102420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:19.4820688Z skip: Need at least 4 CUDA devices (2.816s)
2022-05-18T05:28:19.4834183Z   test_init_sharded_tensor_with_normal (__main__.TestShardedTensorNNInit)
2022-05-18T05:28:19.4964984Z Test torch.nn.init.normal_(ShardedTensor, mean, std) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109608
2022-05-18T05:28:19.5076366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109609
2022-05-18T05:28:19.5193041Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109610
2022-05-18T05:28:19.5311290Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109611
2022-05-18T05:28:20.4371606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:20.4631179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:20.4888520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:20.4890048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:20.6348233Z skip: Need at least 4 CUDA devices (1.153s)
2022-05-18T05:28:20.6363187Z   test_init_sharded_tensor_with_uniform (__main__.TestShardedTensorNNInit)
2022-05-18T05:28:20.6496260Z Test torch.nn.init.uniform_(ShardedTensor, a, b) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109744
2022-05-18T05:28:20.6609049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109745
2022-05-18T05:28:20.6725394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109746
2022-05-18T05:28:20.6848024Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109747
2022-05-18T05:28:21.5943828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:21.6438471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:21.6551802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:21.6653679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:21.8886051Z skip: Need at least 4 CUDA devices (1.254s)
2022-05-18T05:28:21.8886558Z 
2022-05-18T05:28:21.8887305Z ----------------------------------------------------------------------
2022-05-18T05:28:21.8887709Z Ran 3 tests in 5.223s
2022-05-18T05:28:21.8887879Z 
2022-05-18T05:28:21.8887998Z OK (skipped=3)
2022-05-18T05:28:21.8888158Z 
2022-05-18T05:28:21.8888292Z Generating XML reports...
2022-05-18T05:28:21.8934279Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20220518052816.xml
2022-05-18T05:28:22.1651677Z Running distributed/elastic/utils/distributed_test ... [2022-05-18 05:28:22.164642]
2022-05-18T05:28:22.1652498Z Executing ['/opt/conda/bin/python', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:28:22.164746]
2022-05-18T05:28:23.0881637Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test
2022-05-18T05:28:23.0898912Z 
2022-05-18T05:28:23.0899184Z Running tests...
2022-05-18T05:28:23.0899641Z ----------------------------------------------------------------------
2022-05-18T05:28:24.7107046Z   test_create_store_multi (__main__.DistributedUtilTest) ... ok (1.620s)
2022-05-18T05:28:24.7118273Z   test_create_store_no_port_multi (__main__.DistributedUtilTest) ... ok (0.001s)
2022-05-18T05:28:24.7124920Z   test_create_store_single_server (__main__.DistributedUtilTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/66207 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.000s)
2022-05-18T05:28:27.7280237Z   test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... ok (3.015s)
2022-05-18T05:28:27.7288010Z   test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (4e1873759d18, 0).
2022-05-18T05:28:27.7289831Z ok (0.001s)
2022-05-18T05:28:27.7305267Z   test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... [W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:41263 (errno: 98 - Address already in use).
2022-05-18T05:28:27.7325008Z [W socket.cpp:401] [c10d] The server socket has failed to bind to 0.0.0.0:41263 (errno: 98 - Address already in use).
2022-05-18T05:28:27.7325512Z [E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address.
2022-05-18T05:28:27.7328298Z ok (0.004s)
2022-05-18T05:28:27.7357283Z   test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (4e1873759d18, 45533).
2022-05-18T05:28:27.7359463Z ok (0.003s)
2022-05-18T05:28:27.7361212Z 
2022-05-18T05:28:27.7361959Z ----------------------------------------------------------------------
2022-05-18T05:28:27.7362337Z Ran 7 tests in 4.646s
2022-05-18T05:28:27.7362510Z 
2022-05-18T05:28:27.7362639Z OK (skipped=1)
2022-05-18T05:28:27.7362799Z 
2022-05-18T05:28:27.7362911Z Generating XML reports...
2022-05-18T05:28:27.7405385Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220518052823.xml
2022-05-18T05:28:28.0029848Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-05-18 05:28:28.002496]
2022-05-18T05:28:28.0030982Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:28:28.002605]
2022-05-18T05:28:28.9131753Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping
2022-05-18T05:28:28.9149023Z 
2022-05-18T05:28:28.9149236Z Running tests...
2022-05-18T05:28:28.9149685Z ----------------------------------------------------------------------
2022-05-18T05:28:28.9163341Z   test_multiple_wrapping (__main__.TestMultipleWrapping)
2022-05-18T05:28:30.5117004Z This test simulates wrapping the module after training to run inference. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:30.5512076Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109959
2022-05-18T05:28:30.5626169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109960
2022-05-18T05:28:31.4744265Z dist init r=0, world=2
2022-05-18T05:28:31.4748775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:28:31.4965560Z dist init r=1, world=2
2022-05-18T05:28:31.4971022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:28:31.4972138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:31.5054272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:32.8690874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:32.8691428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:32.8900378Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:28:32.8901063Z   warnings.warn(
2022-05-18T05:28:32.8902089Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:28:32.8902641Z   warnings.warn(
2022-05-18T05:28:33.3704275Z ok (4.455s)
2022-05-18T05:28:33.3704502Z 
2022-05-18T05:28:33.3704903Z ----------------------------------------------------------------------
2022-05-18T05:28:33.3705229Z Ran 1 test in 4.455s
2022-05-18T05:28:33.3705397Z 
2022-05-18T05:28:33.3705502Z OK
2022-05-18T05:28:33.3705641Z 
2022-05-18T05:28:33.3705777Z Generating XML reports...
2022-05-18T05:28:33.3760644Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220518052828.xml
2022-05-18T05:28:33.6409707Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2022-05-18 05:28:33.640452]
2022-05-18T05:28:33.6410489Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:28:33.640555]
2022-05-18T05:28:34.5683770Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward
2022-05-18T05:28:34.5700304Z 
2022-05-18T05:28:34.5700710Z Running tests...
2022-05-18T05:28:34.5701209Z ----------------------------------------------------------------------
2022-05-18T05:28:36.1568646Z   test_multi_forward (__main__.TestMultiForward) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:36.1962393Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110077
2022-05-18T05:28:36.2075963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110078
2022-05-18T05:28:37.1196045Z dist init r=1, world=2
2022-05-18T05:28:37.1200472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T05:28:37.1269670Z dist init r=0, world=2
2022-05-18T05:28:37.1275599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T05:28:37.1276816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:37.1303329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T05:28:38.4839761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:38.4840307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:38.6870088Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:38.6870632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T05:28:38.6907634Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 1 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:28:38.6908226Z   warnings.warn(
2022-05-18T05:28:38.6909001Z /opt/conda/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:911: UserWarning: Module is input on CPU, we are moving it to 0 to perform parameter verification, flattening, sharding, and will move it back after.
2022-05-18T05:28:38.6909521Z   warnings.warn(
2022-05-18T05:28:39.0143519Z ok (4.444s)
2022-05-18T05:28:39.0143869Z 
2022-05-18T05:28:39.0144490Z ----------------------------------------------------------------------
2022-05-18T05:28:39.0144989Z Ran 1 test in 4.444s
2022-05-18T05:28:39.0145159Z 
2022-05-18T05:28:39.0145257Z OK
2022-05-18T05:28:39.0145413Z 
2022-05-18T05:28:39.0145552Z Generating XML reports...
2022-05-18T05:28:39.0201363Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20220518052834.xml
2022-05-18T05:28:39.2923158Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2022-05-18 05:28:39.291761]
2022-05-18T05:28:39.2924264Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:28:39.291887]
2022-05-18T05:28:40.2199805Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag
2022-05-18T05:28:40.2217503Z 
2022-05-18T05:28:40.2217802Z Running tests...
2022-05-18T05:28:40.2218247Z ----------------------------------------------------------------------
2022-05-18T05:28:41.8260591Z   test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:41.8665033Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110195
2022-05-18T05:28:41.8781200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110196
2022-05-18T05:28:41.8900155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110197
2022-05-18T05:28:41.9021313Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110198
2022-05-18T05:28:42.8111959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:42.8112483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:42.8241952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:42.8919606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:43.1062349Z skip: Need at least 4 CUDA devices (2.884s)
2022-05-18T05:28:43.1197983Z   test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110331
2022-05-18T05:28:43.1307049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110332
2022-05-18T05:28:43.1420989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110333
2022-05-18T05:28:43.1535799Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110334
2022-05-18T05:28:44.0535524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:44.0536105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:44.0730380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:44.0767339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:44.2571636Z skip: Need at least 4 CUDA devices (1.151s)
2022-05-18T05:28:44.2571910Z 
2022-05-18T05:28:44.2572315Z ----------------------------------------------------------------------
2022-05-18T05:28:44.2572642Z Ran 2 tests in 4.035s
2022-05-18T05:28:44.2572809Z 
2022-05-18T05:28:44.2572923Z OK (skipped=2)
2022-05-18T05:28:44.2573081Z 
2022-05-18T05:28:44.2573219Z Generating XML reports...
2022-05-18T05:28:44.2618197Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220518052840.xml
2022-05-18T05:28:44.5260689Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard ... [2022-05-18 05:28:44.525580]
2022-05-18T05:28:44.5261543Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:28:44.525681]
2022-05-18T05:28:45.4232916Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard
2022-05-18T05:28:45.4250903Z 
2022-05-18T05:28:45.4251253Z Running tests...
2022-05-18T05:28:45.4251683Z ----------------------------------------------------------------------
2022-05-18T05:28:46.9967654Z   test_sharded_tensor_reshard (__main__.TestReshard) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:47.0374560Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110502
2022-05-18T05:28:47.0490006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110503
2022-05-18T05:28:47.0609288Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110504
2022-05-18T05:28:47.0729821Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110505
2022-05-18T05:28:47.9574340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:47.9763547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:48.0263298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:48.0290119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:48.1770625Z skip: Need at least 4 CUDA devices (2.752s)
2022-05-18T05:28:48.1916191Z   test_sharded_tensor_reshard_errors (__main__.TestReshard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110638
2022-05-18T05:28:48.2032938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110639
2022-05-18T05:28:48.2148195Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110640
2022-05-18T05:28:48.2266293Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110641
2022-05-18T05:28:49.1033307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:49.1448083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:49.1603109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:49.1804194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:49.3302614Z skip: Need at least 4 CUDA devices (1.153s)
2022-05-18T05:28:49.3302932Z 
2022-05-18T05:28:49.3303637Z ----------------------------------------------------------------------
2022-05-18T05:28:49.3304308Z Ran 2 tests in 3.905s
2022-05-18T05:28:49.3304480Z 
2022-05-18T05:28:49.3304592Z OK (skipped=2)
2022-05-18T05:28:49.3304755Z 
2022-05-18T05:28:49.3304883Z Generating XML reports...
2022-05-18T05:28:49.3350107Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20220518052845.xml
2022-05-18T05:28:49.6002910Z Running distributed/_shard/sharded_optim/test_sharded_optim ... [2022-05-18 05:28:49.599716]
2022-05-18T05:28:49.6004015Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:28:49.599843]
2022-05-18T05:28:50.4992330Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim
2022-05-18T05:28:50.5009747Z 
2022-05-18T05:28:50.5010143Z Running tests...
2022-05-18T05:28:50.5010617Z ----------------------------------------------------------------------
2022-05-18T05:28:52.0761350Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:52.1166826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110809
2022-05-18T05:28:52.1283201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110810
2022-05-18T05:28:52.1401924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110811
2022-05-18T05:28:52.1523702Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110812
2022-05-18T05:28:53.0377153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:53.0519274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:53.0714289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:53.1143578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:53.2563678Z skip: Need at least 4 CUDA devices (2.755s)
2022-05-18T05:28:53.2720677Z   test_sharded_optim (__main__.TestShardedOptimizer) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110945
2022-05-18T05:28:53.2830541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110946
2022-05-18T05:28:53.2946918Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110947
2022-05-18T05:28:53.3066482Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110948
2022-05-18T05:28:54.2564500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:54.2580155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:54.2711518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:54.2787017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:54.5102455Z skip: Need at least 4 CUDA devices (1.254s)
2022-05-18T05:28:54.5102755Z 
2022-05-18T05:28:54.5103335Z ----------------------------------------------------------------------
2022-05-18T05:28:54.5103837Z Ran 2 tests in 4.009s
2022-05-18T05:28:54.5104010Z 
2022-05-18T05:28:54.5104123Z OK (skipped=2)
2022-05-18T05:28:54.5104286Z 
2022-05-18T05:28:54.5104396Z Generating XML reports...
2022-05-18T05:28:54.5148718Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20220518052850.xml
2022-05-18T05:28:54.7816097Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2022-05-18 05:28:54.781051]
2022-05-18T05:28:54.7816934Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:28:54.781157]
2022-05-18T05:28:55.6835681Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype
2022-05-18T05:28:55.6853129Z 
2022-05-18T05:28:55.6853531Z Running tests...
2022-05-18T05:28:55.6854018Z ----------------------------------------------------------------------
2022-05-18T05:28:57.2711232Z   test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T05:28:57.3106731Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111116
2022-05-18T05:28:57.3221539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111117
2022-05-18T05:28:57.3336691Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 111118
2022-05-18T05:28:57.3455785Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 111119
2022-05-18T05:28:58.2360857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T05:28:58.2702014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T05:28:58.3018874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T05:28:58.3135371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T05:28:58.4497904Z skip: Need at least 4 CUDA devices (2.764s)
2022-05-18T05:28:58.4498471Z 
2022-05-18T05:28:58.4499193Z ----------------------------------------------------------------------
2022-05-18T05:28:58.4499586Z Ran 1 test in 2.764s
2022-05-18T05:28:58.4499751Z 
2022-05-18T05:28:58.4499861Z OK (skipped=1)
2022-05-18T05:28:58.4500021Z 
2022-05-18T05:28:58.4500131Z Generating XML reports...
2022-05-18T05:28:58.4543779Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20220518052855.xml
2022-05-18T05:28:58.7265135Z Running distributed/test_launcher ... [2022-05-18 05:28:58.725931]
2022-05-18T05:28:58.7266126Z Executing ['/opt/conda/bin/python', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:28:58.726034]
2022-05-18T05:28:59.8959144Z Test results will be stored in test-reports/python-unittest/distributed.test_launcher
2022-05-18T05:28:59.8974583Z 
2022-05-18T05:28:59.8975150Z Running tests...
2022-05-18T05:28:59.8976084Z ----------------------------------------------------------------------
2022-05-18T05:29:01.5053810Z   test_launch_user_script (__main__.TestDistributedLaunch) ... /opt/conda/lib/python3.9/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
2022-05-18T05:29:01.5054800Z and will be removed in future. Use torchrun.
2022-05-18T05:29:01.5055209Z Note that --use_env is set by default in torchrun.
2022-05-18T05:29:01.5055659Z If your script expects `--local_rank` argument to be set, please
2022-05-18T05:29:01.5056095Z change it to read from `os.environ['LOCAL_RANK']` instead. See 
2022-05-18T05:29:01.5056598Z https://pytorch.org/docs/stable/distributed.html#launch-utility for 
2022-05-18T05:29:01.5056946Z further instructions
2022-05-18T05:29:01.5057110Z 
2022-05-18T05:29:01.5057223Z   warnings.warn(
2022-05-18T05:29:01.5067366Z WARNING:torch.distributed.run:
2022-05-18T05:29:01.5067662Z *****************************************
2022-05-18T05:29:01.5068473Z Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
2022-05-18T05:29:01.5069288Z *****************************************
2022-05-18T05:29:01.5596190Z Success, smoke test
2022-05-18T05:29:01.5795524Z Success, smoke test
2022-05-18T05:29:01.6007467Z Success, smoke test
2022-05-18T05:29:01.6219645Z Success, smoke test
2022-05-18T05:29:02.6025438Z ok (2.705s)
2022-05-18T05:29:02.6026711Z 
2022-05-18T05:29:02.6027269Z ----------------------------------------------------------------------
2022-05-18T05:29:02.6028038Z Ran 1 test in 2.705s
2022-05-18T05:29:02.6028364Z 
2022-05-18T05:29:02.6028534Z OK
2022-05-18T05:29:02.6028676Z 
2022-05-18T05:29:02.6028809Z Generating XML reports...
2022-05-18T05:29:02.6072214Z Generated XML report: test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20220518052859.xml
2022-05-18T05:29:02.8831233Z Running distributed/elastic/utils/util_test ... [2022-05-18 05:29:02.882560]
2022-05-18T05:29:02.8832141Z Executing ['/opt/conda/bin/python', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:29:02.882662]
2022-05-18T05:29:03.7526722Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test
2022-05-18T05:29:03.7544741Z 
2022-05-18T05:29:03.7545232Z Running tests...
2022-05-18T05:29:03.7545788Z ----------------------------------------------------------------------
2022-05-18T05:29:05.3554560Z   test_get_all_rank_0 (__main__.StoreUtilTest) ... ok (1.601s)
2022-05-18T05:29:05.3577417Z   test_get_all_rank_n (__main__.StoreUtilTest) ... ok (0.002s)
2022-05-18T05:29:05.3609535Z   test_synchronize (__main__.StoreUtilTest) ... ok (0.003s)
2022-05-18T05:29:05.4303699Z   test_get_logger (__main__.UtilTest) ... ok (0.069s)
2022-05-18T05:29:05.4311707Z   test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s)
2022-05-18T05:29:05.4322427Z   test_get_logger_different (__main__.UtilTest) ... ok (0.001s)
2022-05-18T05:29:05.4337410Z   test_get_logger_none (__main__.UtilTest) ... ok (0.001s)
2022-05-18T05:29:05.4337743Z 
2022-05-18T05:29:05.4338144Z ----------------------------------------------------------------------
2022-05-18T05:29:05.4338505Z Ran 7 tests in 1.679s
2022-05-18T05:29:05.4338675Z 
2022-05-18T05:29:05.4338772Z OK
2022-05-18T05:29:05.4338907Z 
2022-05-18T05:29:05.4339313Z Generating XML reports...
2022-05-18T05:29:05.4372808Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220518052903.xml
2022-05-18T05:29:05.4379960Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220518052903.xml
2022-05-18T05:29:05.6664590Z Running distributed/fsdp/test_utils ... [2022-05-18 05:29:05.665917]
2022-05-18T05:29:05.6665298Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:29:05.666022]
2022-05-18T05:29:06.5721508Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils
2022-05-18T05:29:06.5737951Z 
2022-05-18T05:29:06.5738110Z Running tests...
2022-05-18T05:29:06.5738966Z ----------------------------------------------------------------------
2022-05-18T05:29:08.1794367Z   test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... ok (1.605s)
2022-05-18T05:29:08.1823434Z   test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.003s)
2022-05-18T05:29:08.1852821Z   test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... ok (0.003s)
2022-05-18T05:29:08.1861313Z   test_packed_sequence (__main__.TestUtils)
2022-05-18T05:29:08.1882565Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.003s)
2022-05-18T05:29:08.1894254Z   test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s)
2022-05-18T05:29:08.1894801Z 
2022-05-18T05:29:08.1895122Z ----------------------------------------------------------------------
2022-05-18T05:29:08.1895461Z Ran 5 tests in 1.616s
2022-05-18T05:29:08.1895963Z 
2022-05-18T05:29:08.1896063Z OK
2022-05-18T05:29:08.1896208Z 
2022-05-18T05:29:08.1896342Z Generating XML reports...
2022-05-18T05:29:08.1940547Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220518052906.xml
2022-05-18T05:29:08.4363284Z Running distributed/nn/jit/test_instantiator ... [2022-05-18 05:29:08.435796]
2022-05-18T05:29:08.4364039Z Executing ['/opt/conda/bin/python', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:29:08.435923]
2022-05-18T05:29:09.3181794Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator
2022-05-18T05:29:09.3198628Z 
2022-05-18T05:29:09.3198956Z Running tests...
2022-05-18T05:29:09.3199405Z ----------------------------------------------------------------------
2022-05-18T05:29:10.9084518Z   test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (1.588s)
2022-05-18T05:29:10.9105629Z   test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s)
2022-05-18T05:29:10.9261062Z   test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.015s)
2022-05-18T05:29:10.9261372Z 
2022-05-18T05:29:10.9261766Z ----------------------------------------------------------------------
2022-05-18T05:29:10.9262106Z Ran 3 tests in 1.606s
2022-05-18T05:29:10.9262273Z 
2022-05-18T05:29:10.9262375Z OK
2022-05-18T05:29:10.9262515Z 
2022-05-18T05:29:10.9262645Z Generating XML reports...
2022-05-18T05:29:10.9297971Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220518052909.xml
2022-05-18T05:29:11.1760617Z Running distributed/_shard/sharding_plan/test_sharding_plan ... [2022-05-18 05:29:11.175546]
2022-05-18T05:29:11.1761418Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:29:11.175649]
2022-05-18T05:29:12.1647901Z Running distributed/_shard/test_sharder ... [2022-05-18 05:29:12.164300]
2022-05-18T05:29:12.1648629Z Executing ['/opt/conda/bin/python', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:29:12.164402]
2022-05-18T05:29:13.0453119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54sbofb4
2022-05-18T05:29:13.0454459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54sbofb4/_remote_module_non_scriptable.py
2022-05-18T05:29:13.1770875Z Running distributed/elastic/timer/api_test ... [2022-05-18 05:29:13.176591]
2022-05-18T05:29:13.1771631Z Executing ['/opt/conda/bin/python', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:29:13.176693]
2022-05-18T05:29:14.0884178Z Running distributed/pipeline/sync/skip/test_api ... [2022-05-18 05:29:14.087925]
2022-05-18T05:29:14.0884878Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2022-05-18 05:29:14.088029]
2022-05-18T05:29:15.6494668Z ============================= test session starts ==============================
2022-05-18T05:29:15.6495244Z platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:29:15.6559336Z cachedir: .pytest_cache
2022-05-18T05:29:15.6559956Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:29:15.6560411Z torch: 1.12.0a0+git3b23752
2022-05-18T05:29:15.6560728Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:29:15.6561112Z plugins: hypothesis-4.53.2
2022-05-18T05:29:15.6713951Z collecting ... 
2022-05-18T05:29:15.6714344Z collected 3 items                                                              
2022-05-18T05:29:15.6718684Z 
2022-05-18T05:29:15.6750656Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%]
2022-05-18T05:29:15.6767536Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED   [ 66%]
2022-05-18T05:29:15.6805695Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED   [100%]
2022-05-18T05:29:15.6807410Z 
2022-05-18T05:29:15.6807763Z ============================== 3 passed in 0.03s ===============================
2022-05-18T05:29:15.8271748Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-05-18 05:29:15.826623]
2022-05-18T05:29:15.8272962Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-05-18 05:29:15.826731]
2022-05-18T05:29:17.0653861Z ============================= test session starts ==============================
2022-05-18T05:29:17.0654482Z platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:29:17.0673369Z cachedir: .pytest_cache
2022-05-18T05:29:17.0673979Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:29:17.0674403Z torch: 1.12.0a0+git3b23752
2022-05-18T05:29:17.0674750Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:29:17.0675139Z plugins: hypothesis-4.53.2
2022-05-18T05:29:17.0852879Z collecting ... 
2022-05-18T05:29:17.0853300Z collected 6 items                                                              
2022-05-18T05:29:17.0857374Z 
2022-05-18T05:29:17.0890587Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%]
2022-05-18T05:29:17.0909252Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%]
2022-05-18T05:29:17.0927803Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%]
2022-05-18T05:29:17.0946340Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%]
2022-05-18T05:29:17.0968019Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%]
2022-05-18T05:29:17.1009497Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%]
2022-05-18T05:29:17.1010535Z 
2022-05-18T05:29:17.1010869Z ============================== 6 passed in 0.04s ===============================
2022-05-18T05:29:17.2442609Z Running distributed/pipeline/sync/skip/test_portal ... [2022-05-18 05:29:17.243654]
2022-05-18T05:29:17.2443733Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-05-18 05:29:17.243760]
2022-05-18T05:29:18.4991805Z ============================= test session starts ==============================
2022-05-18T05:29:18.4992431Z platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:29:18.5012186Z cachedir: .pytest_cache
2022-05-18T05:29:18.5013270Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:29:18.5013761Z torch: 1.12.0a0+git3b23752
2022-05-18T05:29:18.5014088Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:29:18.5014481Z plugins: hypothesis-4.53.2
2022-05-18T05:29:18.5429076Z collecting ... 
2022-05-18T05:29:18.5429500Z collected 10 items                                                             
2022-05-18T05:29:18.5433674Z 
2022-05-18T05:29:19.7748079Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device PASSED [ 10%]
2022-05-18T05:29:19.7774970Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED   [ 20%]
2022-05-18T05:29:19.7793332Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%]
2022-05-18T05:29:19.7808875Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED      [ 40%]
2022-05-18T05:29:19.7826441Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%]
2022-05-18T05:29:19.7842434Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%]
2022-05-18T05:29:19.7858247Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%]
2022-05-18T05:29:19.7874007Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%]
2022-05-18T05:29:19.7890029Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%]
2022-05-18T05:29:19.7909894Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%]
2022-05-18T05:29:19.7911857Z 
2022-05-18T05:29:19.7912510Z ============================== 10 passed in 1.29s ==============================
2022-05-18T05:29:20.0068579Z Running distributed/pipeline/sync/skip/test_tracker ... [2022-05-18 05:29:20.006368]
2022-05-18T05:29:20.0069239Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2022-05-18 05:29:20.006475]
2022-05-18T05:29:21.2599602Z ============================= test session starts ==============================
2022-05-18T05:29:21.2600232Z platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:29:21.2619948Z cachedir: .pytest_cache
2022-05-18T05:29:21.2620906Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:29:21.2621393Z torch: 1.12.0a0+git3b23752
2022-05-18T05:29:21.2621731Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:29:21.2622125Z plugins: hypothesis-4.53.2
2022-05-18T05:29:21.2921985Z collecting ... 
2022-05-18T05:29:21.2922420Z collected 6 items                                                              
2022-05-18T05:29:21.2927016Z 
2022-05-18T05:29:21.2962569Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%]
2022-05-18T05:29:22.5199406Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel PASSED [ 33%]
2022-05-18T05:29:22.5215761Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%]
2022-05-18T05:29:22.5230194Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%]
2022-05-18T05:29:22.5245306Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%]
2022-05-18T05:29:22.5264417Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%]
2022-05-18T05:29:22.5266480Z 
2022-05-18T05:29:22.5267011Z ============================== 6 passed in 1.27s ===============================
2022-05-18T05:29:22.7400477Z Running distributed/pipeline/sync/test_balance ... [2022-05-18 05:29:22.739524]
2022-05-18T05:29:22.7401167Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-05-18 05:29:22.739631]
2022-05-18T05:29:24.0286521Z ============================= test session starts ==============================
2022-05-18T05:29:24.0287107Z platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:29:24.0306972Z cachedir: .pytest_cache
2022-05-18T05:29:24.0307932Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:29:24.0308720Z torch: 1.12.0a0+git3b23752
2022-05-18T05:29:24.0309037Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:29:24.0309431Z plugins: hypothesis-4.53.2
2022-05-18T05:29:24.0684884Z collecting ... 
2022-05-18T05:29:24.0685375Z collected 18 items                                                             
2022-05-18T05:29:24.0690005Z 
2022-05-18T05:29:24.0721514Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED    [  5%]
2022-05-18T05:29:24.0739033Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 11%]
2022-05-18T05:29:24.0757087Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 16%]
2022-05-18T05:29:24.0774764Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 22%]
2022-05-18T05:29:24.0788030Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 27%]
2022-05-18T05:29:24.0800116Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda] SKIPPED [ 33%]
2022-05-18T05:29:25.0846992Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 38%]
2022-05-18T05:29:26.3413150Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent PASSED [ 44%]
2022-05-18T05:29:26.5181967Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param PASSED [ 50%]
2022-05-18T05:29:26.5318246Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale PASSED [ 55%]
2022-05-18T05:29:26.5350135Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 61%]
2022-05-18T05:29:26.5381355Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda] PASSED [ 66%]
2022-05-18T05:29:27.5416555Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 72%]
2022-05-18T05:29:28.5453689Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] PASSED [ 77%]
2022-05-18T05:29:29.5486636Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED      [ 83%]
2022-05-18T05:29:30.5513038Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 88%]
2022-05-18T05:29:30.5545642Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple PASSED [ 94%]
2022-05-18T05:29:30.5575053Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED  [100%]
2022-05-18T05:29:30.5578030Z 
2022-05-18T05:29:30.5578578Z =========================== short test summary info ============================
2022-05-18T05:29:30.5579147Z SKIPPED [2] distributed/pipeline/sync/test_balance.py:47: Flaky due to time.sleep()
2022-05-18T05:29:30.5580058Z ======================== 16 passed, 2 skipped in 6.53s =========================
2022-05-18T05:29:30.8167129Z Running distributed/pipeline/sync/test_checkpoint ... [2022-05-18 05:29:30.816096]
2022-05-18T05:29:30.8168282Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-05-18 05:29:30.816206]
2022-05-18T05:29:32.0688303Z ============================= test session starts ==============================
2022-05-18T05:29:32.0688862Z platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:29:32.0708756Z cachedir: .pytest_cache
2022-05-18T05:29:32.0710146Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:29:32.0710679Z torch: 1.12.0a0+git3b23752
2022-05-18T05:29:32.0711000Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:29:32.0711715Z plugins: hypothesis-4.53.2
2022-05-18T05:29:32.0976454Z collecting ... 
2022-05-18T05:29:32.0977411Z collected 9 items                                                              
2022-05-18T05:29:32.0981058Z 
2022-05-18T05:29:32.1046539Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 11%]
2022-05-18T05:29:33.3308297Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda] PASSED [ 22%]
2022-05-18T05:29:33.3326203Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 33%]
2022-05-18T05:29:33.3343864Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 44%]
2022-05-18T05:29:33.3374885Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 55%]
2022-05-18T05:29:33.3409601Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda] PASSED [ 66%]
2022-05-18T05:29:33.3427742Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 77%]
2022-05-18T05:29:33.3442868Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 88%]
2022-05-18T05:29:33.3466669Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%]
2022-05-18T05:29:33.3467591Z 
2022-05-18T05:29:33.3468201Z ============================== 9 passed in 1.28s ===============================
2022-05-18T05:29:33.5613322Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2022-05-18 05:29:33.560757]
2022-05-18T05:29:33.5614077Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2022-05-18 05:29:33.560862]
2022-05-18T05:29:34.8230373Z ============================= test session starts ==============================
2022-05-18T05:29:34.8230965Z platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:29:34.8250246Z cachedir: .pytest_cache
2022-05-18T05:29:34.8251492Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:29:34.8251973Z torch: 1.12.0a0+git3b23752
2022-05-18T05:29:34.8252313Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:29:34.8252690Z plugins: hypothesis-4.53.2
2022-05-18T05:29:34.8565143Z collecting ... 
2022-05-18T05:29:34.8565754Z collected 11 items                                                             
2022-05-18T05:29:34.8570042Z 
2022-05-18T05:29:34.9286229Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [  9%]
2022-05-18T05:29:34.9752314Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%]
2022-05-18T05:29:35.0138246Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%]
2022-05-18T05:29:35.0590715Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%]
2022-05-18T05:29:35.0872413Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%]
2022-05-18T05:29:35.1161016Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%]
2022-05-18T05:29:35.1182359Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%]
2022-05-18T05:29:35.1507112Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED  [ 72%]
2022-05-18T05:29:35.3041189Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%]
2022-05-18T05:29:35.3953149Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%]
2022-05-18T05:29:35.4199095Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%]
2022-05-18T05:29:35.4199434Z 
2022-05-18T05:29:35.4200226Z ============================== 11 passed in 0.60s ==============================
2022-05-18T05:29:35.5758820Z Running distributed/pipeline/sync/test_inplace ... [2022-05-18 05:29:35.575342]
2022-05-18T05:29:35.5759553Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-05-18 05:29:35.575446]
2022-05-18T05:29:36.7944615Z ============================= test session starts ==============================
2022-05-18T05:29:36.7945235Z platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:29:36.7965699Z cachedir: .pytest_cache
2022-05-18T05:29:36.7966606Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:29:36.7967054Z torch: 1.12.0a0+git3b23752
2022-05-18T05:29:36.7967395Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:29:36.7967782Z plugins: hypothesis-4.53.2
2022-05-18T05:29:36.8098960Z collecting ... 
2022-05-18T05:29:36.8099774Z collected 3 items                                                              
2022-05-18T05:29:36.8104090Z 
2022-05-18T05:29:36.9164161Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%]
2022-05-18T05:29:36.9422311Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%]
2022-05-18T05:29:36.9580657Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%]
2022-05-18T05:29:36.9582809Z 
2022-05-18T05:29:36.9583038Z =========================== short test summary info ============================
2022-05-18T05:29:36.9583479Z XFAIL distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad
2022-05-18T05:29:36.9583925Z XFAIL distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad
2022-05-18T05:29:36.9585110Z ========================= 1 passed, 2 xfailed in 0.16s =========================
2022-05-18T05:29:37.1019266Z Running distributed/pipeline/sync/test_phony ... [2022-05-18 05:29:37.101454]
2022-05-18T05:29:37.1019904Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-05-18 05:29:37.101562]
2022-05-18T05:29:38.3474405Z ============================= test session starts ==============================
2022-05-18T05:29:38.3475522Z platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:29:38.3496681Z cachedir: .pytest_cache
2022-05-18T05:29:38.3497952Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:29:38.3498846Z torch: 1.12.0a0+git3b23752
2022-05-18T05:29:38.3499494Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:29:38.3500254Z plugins: hypothesis-4.53.2
2022-05-18T05:29:38.3682514Z collecting ... 
2022-05-18T05:29:38.3682957Z collected 4 items                                                              
2022-05-18T05:29:38.3686842Z 
2022-05-18T05:29:38.3719257Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED          [ 25%]
2022-05-18T05:29:38.3734995Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%]
2022-05-18T05:29:38.3750017Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED        [ 75%]
2022-05-18T05:29:38.3780593Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%]
2022-05-18T05:29:38.3781709Z 
2022-05-18T05:29:38.3782072Z ============================== 4 passed in 0.03s ===============================
2022-05-18T05:29:38.5182368Z Running distributed/pipeline/sync/test_pipeline ... [2022-05-18 05:29:38.517764]
2022-05-18T05:29:38.5183047Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2022-05-18 05:29:38.517869]
2022-05-18T05:29:39.7547420Z ============================= test session starts ==============================
2022-05-18T05:29:39.7548047Z platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:29:39.7567637Z cachedir: .pytest_cache
2022-05-18T05:29:39.7568651Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:29:39.7569264Z torch: 1.12.0a0+git3b23752
2022-05-18T05:29:39.7569927Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:29:39.7570772Z plugins: hypothesis-4.53.2
2022-05-18T05:29:39.7736215Z collecting ... 
2022-05-18T05:29:39.7737139Z collected 1 item                                                               
2022-05-18T05:29:39.7740686Z 
2022-05-18T05:29:39.7775642Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED     [100%]
2022-05-18T05:29:39.7776427Z 
2022-05-18T05:29:39.7777238Z ============================== 1 passed in 0.02s ===============================
2022-05-18T05:29:39.9139082Z Running distributed/pipeline/sync/test_transparency ... [2022-05-18 05:29:39.913359]
2022-05-18T05:29:39.9139883Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-05-18 05:29:39.913464]
2022-05-18T05:29:41.1590221Z ============================= test session starts ==============================
2022-05-18T05:29:41.1591000Z platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T05:29:41.1610310Z cachedir: .pytest_cache
2022-05-18T05:29:41.1611570Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T05:29:41.1612501Z torch: 1.12.0a0+git3b23752
2022-05-18T05:29:41.1612843Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T05:29:41.1613244Z plugins: hypothesis-4.53.2
2022-05-18T05:29:41.1728698Z collecting ... 
2022-05-18T05:29:41.1729609Z collected 1 item                                                               
2022-05-18T05:29:41.1733628Z 
2022-05-18T05:29:41.2859281Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%]
2022-05-18T05:29:41.2859961Z 
2022-05-18T05:29:41.2860353Z ============================== 1 passed in 0.13s ===============================
2022-05-18T05:29:41.4307863Z Running distributed/rpc/test_faulty_agent ... [2022-05-18 05:29:41.430273]
2022-05-18T05:29:41.4308784Z Executing ['/opt/conda/bin/python', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 05:29:41.430377]
2022-05-18T05:29:42.3156675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0bdzpyl
2022-05-18T05:29:42.3157991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0bdzpyl/_remote_module_non_scriptable.py
2022-05-18T05:29:43.6247797Z 
2022-05-18T05:29:43.6248476Z real	84m54.023s
2022-05-18T05:29:43.6249059Z user	130m51.297s
2022-05-18T05:29:43.6249591Z sys	109m55.022s
2022-05-18T05:29:43.6249864Z + assert_git_not_dirty
2022-05-18T05:29:43.6250428Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed != *rocm* ]]
2022-05-18T05:29:43.6250939Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed != *xla* ]]
2022-05-18T05:29:43.6255738Z ++ git status --porcelain
2022-05-18T05:29:44.3763235Z + git_status=
2022-05-18T05:29:44.3763678Z + [[ -n '' ]]
2022-05-18T05:29:44.3764123Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed == *cuda* ]]
2022-05-18T05:29:44.3764460Z + [[ 1 == 1 ]]
2022-05-18T05:29:44.3764765Z + echo 'Testing distributed C++ tests'
2022-05-18T05:29:44.3765061Z Testing distributed C++ tests
2022-05-18T05:29:44.3766425Z + ln -sf /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.9/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.9/site-packages/torch/bin
2022-05-18T05:29:44.3779193Z + ln -sf /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.9/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.9/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.9/site-packages/torch/bin
2022-05-18T05:29:44.3789534Z + TEST_REPORTS_DIR=test/test-reports/cpp-distributed/test_distributed
2022-05-18T05:29:44.3790040Z + mkdir -p test/test-reports/cpp-distributed/test_distributed
2022-05-18T05:29:44.3805227Z + /opt/conda/lib/python3.9/site-packages/torch/bin/FileStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml
2022-05-18T05:29:44.6001069Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-05-18T05:29:44.6001642Z [==========] Running 4 tests from 1 test suite.
2022-05-18T05:29:44.6002063Z [----------] Global test environment set-up.
2022-05-18T05:29:44.6002463Z [----------] 4 tests from FileStoreTest
2022-05-18T05:29:44.6002843Z [ RUN      ] FileStoreTest.testGetAndSet
2022-05-18T05:29:44.6006275Z [       OK ] FileStoreTest.testGetAndSet (0 ms)
2022-05-18T05:29:44.6006751Z [ RUN      ] FileStoreTest.testGetAndSetWithPrefix
2022-05-18T05:29:44.6010653Z [       OK ] FileStoreTest.testGetAndSetWithPrefix (0 ms)
2022-05-18T05:29:44.6011110Z [ RUN      ] FileStoreTest.testStressStore
2022-05-18T05:29:44.6215095Z [       OK ] FileStoreTest.testStressStore (20 ms)
2022-05-18T05:29:44.6215613Z [ RUN      ] FileStoreTest.testStressStoreWithPrefix
2022-05-18T05:29:44.6418925Z [       OK ] FileStoreTest.testStressStoreWithPrefix (20 ms)
2022-05-18T05:29:44.6419670Z [----------] 4 tests from FileStoreTest (41 ms total)
2022-05-18T05:29:44.6420061Z 
2022-05-18T05:29:44.6420306Z [----------] Global test environment tear-down
2022-05-18T05:29:44.6421074Z [==========] 4 tests from 1 test suite ran. (41 ms total)
2022-05-18T05:29:44.6421419Z [  PASSED  ] 4 tests.
2022-05-18T05:29:44.6810436Z + /opt/conda/lib/python3.9/site-packages/torch/bin/HashStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml
2022-05-18T05:29:44.8960866Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-05-18T05:29:44.8961467Z [==========] Running 4 tests from 1 test suite.
2022-05-18T05:29:44.8961872Z [----------] Global test environment set-up.
2022-05-18T05:29:44.8962272Z [----------] 4 tests from HashStoreTest
2022-05-18T05:29:44.8962665Z [ RUN      ] HashStoreTest.testGetAndSet
2022-05-18T05:29:44.9964632Z [       OK ] HashStoreTest.testGetAndSet (100 ms)
2022-05-18T05:29:44.9965111Z [ RUN      ] HashStoreTest.testGetAndSetWithPrefix
2022-05-18T05:29:45.0966932Z [       OK ] HashStoreTest.testGetAndSetWithPrefix (100 ms)
2022-05-18T05:29:45.0967424Z [ RUN      ] HashStoreTest.testStressStore
2022-05-18T05:29:45.0973695Z [       OK ] HashStoreTest.testStressStore (0 ms)
2022-05-18T05:29:45.0974434Z [ RUN      ] HashStoreTest.testStressStoreWithPrefix
2022-05-18T05:29:45.0980414Z [       OK ] HashStoreTest.testStressStoreWithPrefix (0 ms)
2022-05-18T05:29:45.0980922Z [----------] 4 tests from HashStoreTest (202 ms total)
2022-05-18T05:29:45.0981253Z 
2022-05-18T05:29:45.0981707Z [----------] Global test environment tear-down
2022-05-18T05:29:45.0982918Z [==========] 4 tests from 1 test suite ran. (202 ms total)
2022-05-18T05:29:45.0983277Z [  PASSED  ] 4 tests.
2022-05-18T05:29:45.1413248Z + /opt/conda/lib/python3.9/site-packages/torch/bin/TCPStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml
2022-05-18T05:29:45.3608719Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-05-18T05:29:45.3609338Z [==========] Running 11 tests from 1 test suite.
2022-05-18T05:29:45.3609745Z [----------] Global test environment set-up.
2022-05-18T05:29:45.3610150Z [----------] 11 tests from TCPStoreTest
2022-05-18T05:29:45.3610538Z [ RUN      ] TCPStoreTest.testHelper
2022-05-18T05:29:46.3405608Z [       OK ] TCPStoreTest.testHelper (979 ms)
2022-05-18T05:29:46.3406080Z [ RUN      ] TCPStoreTest.testHelperPrefix
2022-05-18T05:29:47.3144438Z [       OK ] TCPStoreTest.testHelperPrefix (973 ms)
2022-05-18T05:29:47.3144963Z [ RUN      ] TCPStoreTest.testWatchKeyCallback
2022-05-18T05:29:47.3289855Z [       OK ] TCPStoreTest.testWatchKeyCallback (14 ms)
2022-05-18T05:29:47.3290412Z [ RUN      ] TCPStoreTest.testWatchKeyCallbackWithPrefix
2022-05-18T05:29:47.3429548Z [       OK ] TCPStoreTest.testWatchKeyCallbackWithPrefix (13 ms)
2022-05-18T05:29:47.3430076Z [ RUN      ] TCPStoreTest.testKeyEmptyUpdate
2022-05-18T05:29:47.5486587Z [       OK ] TCPStoreTest.testKeyEmptyUpdate (205 ms)
2022-05-18T05:29:47.5487069Z [ RUN      ] TCPStoreTest.testKeyUpdate
2022-05-18T05:29:47.5492688Z [       OK ] TCPStoreTest.testKeyUpdate (0 ms)
2022-05-18T05:29:47.5493126Z [ RUN      ] TCPStoreTest.testKeyCreate
2022-05-18T05:29:47.5497741Z [       OK ] TCPStoreTest.testKeyCreate (0 ms)
2022-05-18T05:29:47.5498391Z [ RUN      ] TCPStoreTest.testKeyAdd
2022-05-18T05:29:47.5502758Z [       OK ] TCPStoreTest.testKeyAdd (0 ms)
2022-05-18T05:29:47.5503174Z [ RUN      ] TCPStoreTest.testKeyDelete
2022-05-18T05:29:47.7565101Z [       OK ] TCPStoreTest.testKeyDelete (206 ms)
2022-05-18T05:29:47.7565598Z [ RUN      ] TCPStoreTest.testCleanShutdown
2022-05-18T05:29:47.7573737Z [       OK ] TCPStoreTest.testCleanShutdown (0 ms)
2022-05-18T05:29:47.7574225Z [ RUN      ] TCPStoreTest.testMultiTenantStores
2022-05-18T05:29:47.7587360Z [       OK ] TCPStoreTest.testMultiTenantStores (1 ms)
2022-05-18T05:29:47.7587885Z [----------] 11 tests from TCPStoreTest (2397 ms total)
2022-05-18T05:29:47.7588111Z 
2022-05-18T05:29:47.7588347Z [----------] Global test environment tear-down
2022-05-18T05:29:47.7591233Z [==========] 11 tests from 1 test suite ran. (2397 ms total)
2022-05-18T05:29:47.7591591Z [  PASSED  ] 11 tests.
2022-05-18T05:29:47.8036839Z ++ command -v mpiexec
2022-05-18T05:29:47.8039502Z + MPIEXEC=/usr/bin/mpiexec
2022-05-18T05:29:47.8039826Z + [[ -n /usr/bin/mpiexec ]]
2022-05-18T05:29:47.8040118Z + [[ -z true ]]
2022-05-18T05:29:47.8040778Z + /opt/conda/lib/python3.9/site-packages/torch/bin/ProcessGroupGlooTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml
2022-05-18T05:29:48.0199034Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-05-18T05:29:48.0199648Z [==========] Running 12 tests from 1 test suite.
2022-05-18T05:29:48.0200067Z [----------] Global test environment set-up.
2022-05-18T05:29:48.0200846Z [----------] 12 tests from ProcessGroupGlooTest
2022-05-18T05:29:48.0201347Z [ RUN      ] ProcessGroupGlooTest.testSIGSTOPException
2022-05-18T05:29:49.0848227Z [       OK ] ProcessGroupGlooTest.testSIGSTOPException (1064 ms)
2022-05-18T05:29:49.0848789Z [ RUN      ] ProcessGroupGlooTest.testSIGKILLException
2022-05-18T05:29:49.1175420Z [       OK ] ProcessGroupGlooTest.testSIGKILLException (32 ms)
2022-05-18T05:29:49.1175953Z [ RUN      ] ProcessGroupGlooTest.testAllReduceCPU
2022-05-18T05:29:49.4261390Z [       OK ] ProcessGroupGlooTest.testAllReduceCPU (308 ms)
2022-05-18T05:29:49.4261896Z [ RUN      ] ProcessGroupGlooTest.testBroadcastCPU
2022-05-18T05:29:49.4685171Z [       OK ] ProcessGroupGlooTest.testBroadcastCPU (42 ms)
2022-05-18T05:29:49.4685684Z [ RUN      ] ProcessGroupGlooTest.testAllToAllCPU
2022-05-18T05:29:49.6119316Z [       OK ] ProcessGroupGlooTest.testAllToAllCPU (143 ms)
2022-05-18T05:29:49.6119820Z [ RUN      ] ProcessGroupGlooTest.testBarrier
2022-05-18T05:29:49.6633413Z [       OK ] ProcessGroupGlooTest.testBarrier (51 ms)
2022-05-18T05:29:49.6633926Z [ RUN      ] ProcessGroupGlooTest.testMonitoredBarrier
2022-05-18T05:29:50.6943492Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 1000 ms
2022-05-18T05:29:50.7145654Z [       OK ] ProcessGroupGlooTest.testMonitoredBarrier (1051 ms)
2022-05-18T05:29:50.7146200Z [ RUN      ] ProcessGroupGlooTest.testSequenceNumInit
2022-05-18T05:29:50.8567812Z [       OK ] ProcessGroupGlooTest.testSequenceNumInit (142 ms)
2022-05-18T05:29:50.8568292Z [ RUN      ] ProcessGroupGlooTest.testSend
2022-05-18T05:29:50.8983761Z [       OK ] ProcessGroupGlooTest.testSend (41 ms)
2022-05-18T05:29:50.8984229Z [ RUN      ] ProcessGroupGlooTest.testRecv
2022-05-18T05:29:50.9401396Z [       OK ] ProcessGroupGlooTest.testRecv (41 ms)
2022-05-18T05:29:50.9401895Z [ RUN      ] ProcessGroupGlooTest.testStoreSetGet
2022-05-18T05:29:50.9812285Z [       OK ] ProcessGroupGlooTest.testStoreSetGet (41 ms)
2022-05-18T05:29:50.9812782Z [ RUN      ] ProcessGroupGlooTest.testWaitDelay
2022-05-18T05:29:51.1230367Z [       OK ] ProcessGroupGlooTest.testWaitDelay (141 ms)
2022-05-18T05:29:51.1230942Z [----------] 12 tests from ProcessGroupGlooTest (3103 ms total)
2022-05-18T05:29:51.1231188Z 
2022-05-18T05:29:51.1231422Z [----------] Global test environment tear-down
2022-05-18T05:29:51.1233798Z [==========] 12 tests from 1 test suite ran. (3103 ms total)
2022-05-18T05:29:51.1234151Z [  PASSED  ] 12 tests.
2022-05-18T05:29:51.1749515Z + /opt/conda/lib/python3.9/site-packages/torch/bin/ProcessGroupNCCLTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml
2022-05-18T05:29:51.4350555Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-05-18T05:29:51.4351185Z [==========] Running 11 tests from 1 test suite.
2022-05-18T05:29:51.4351575Z [----------] Global test environment set-up.
2022-05-18T05:29:51.4352020Z [----------] 11 tests from ProcessGroupNCCLTest
2022-05-18T05:29:51.4352464Z [ RUN      ] ProcessGroupNCCLTest.testAllreduce
2022-05-18T05:29:59.1106090Z [       OK ] ProcessGroupNCCLTest.testAllreduce (7675 ms)
2022-05-18T05:29:59.1106595Z [ RUN      ] ProcessGroupNCCLTest.testBroadcast
2022-05-18T05:29:59.1932928Z [       OK ] ProcessGroupNCCLTest.testBroadcast (82 ms)
2022-05-18T05:29:59.1933434Z [ RUN      ] ProcessGroupNCCLTest.testReduce
2022-05-18T05:29:59.2719567Z [       OK ] ProcessGroupNCCLTest.testReduce (78 ms)
2022-05-18T05:29:59.2720065Z [ RUN      ] ProcessGroupNCCLTest.testAllgather
2022-05-18T05:29:59.3541698Z [       OK ] ProcessGroupNCCLTest.testAllgather (82 ms)
2022-05-18T05:29:59.3542247Z [ RUN      ] ProcessGroupNCCLTest.testAllgatherBase
2022-05-18T05:29:59.4300210Z [       OK ] ProcessGroupNCCLTest.testAllgatherBase (75 ms)
2022-05-18T05:29:59.4300781Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatter
2022-05-18T05:29:59.5158540Z [       OK ] ProcessGroupNCCLTest.testReduceScatter (85 ms)
2022-05-18T05:29:59.5159133Z [ RUN      ] ProcessGroupNCCLTest.testSequenceNumInit
2022-05-18T05:29:59.5650014Z [       OK ] ProcessGroupNCCLTest.testSequenceNumInit (49 ms)
2022-05-18T05:29:59.5650665Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout
2022-05-18T05:30:02.5694021Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout (3004 ms)
2022-05-18T05:30:02.5694792Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException
2022-05-18T05:30:05.5707604Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException (3001 ms)
2022-05-18T05:30:05.5708295Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatterBase
2022-05-18T05:30:05.6463594Z [       OK ] ProcessGroupNCCLTest.testReduceScatterBase (75 ms)
2022-05-18T05:30:05.6464188Z [ RUN      ] ProcessGroupNCCLTest.testBackendName
2022-05-18T05:30:05.6834171Z [       OK ] ProcessGroupNCCLTest.testBackendName (37 ms)
2022-05-18T05:30:05.6834784Z [----------] 11 tests from ProcessGroupNCCLTest (14248 ms total)
2022-05-18T05:30:05.6835034Z 
2022-05-18T05:30:05.6835272Z [----------] Global test environment tear-down
2022-05-18T05:30:05.6836491Z [==========] 11 tests from 1 test suite ran. (14248 ms total)
2022-05-18T05:30:05.6836838Z [  PASSED  ] 11 tests.
2022-05-18T05:30:06.4178643Z + /opt/conda/lib/python3.9/site-packages/torch/bin/ProcessGroupNCCLErrorsTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml
2022-05-18T05:30:06.6713503Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-05-18T05:30:06.6714147Z [==========] Running 3 tests from 1 test suite.
2022-05-18T05:30:06.6714564Z [----------] Global test environment set-up.
2022-05-18T05:30:06.6715247Z [----------] 3 tests from ProcessGroupNCCLErrorsTest
2022-05-18T05:30:06.6715807Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking
2022-05-18T05:30:12.5890817Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking (5917 ms)
2022-05-18T05:30:12.5891540Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking
2022-05-18T05:30:15.6392446Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking (3050 ms)
2022-05-18T05:30:15.6393167Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking
2022-05-18T05:30:15.6881323Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking (48 ms)
2022-05-18T05:30:15.6881997Z [----------] 3 tests from ProcessGroupNCCLErrorsTest (9016 ms total)
2022-05-18T05:30:15.6882272Z 
2022-05-18T05:30:15.6882491Z [----------] Global test environment tear-down
2022-05-18T05:30:15.6882947Z [==========] 3 tests from 1 test suite ran. (9016 ms total)
2022-05-18T05:30:15.6883298Z [  PASSED  ] 3 tests.
2022-05-18T05:30:16.4217190Z + [[ 1 == 1 ]]
2022-05-18T05:30:16.4217498Z + test_rpc
2022-05-18T05:30:16.4218072Z + [[ linux-bionic-cuda10.2-py3.9-gcc7-distributed != *rocm* ]]
2022-05-18T05:30:16.4218486Z + echo 'Testing RPC C++ tests'
2022-05-18T05:30:16.4218766Z Testing RPC C++ tests
2022-05-18T05:30:16.4220086Z + ln -sf /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.9/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.9/site-packages/torch/bin
2022-05-18T05:30:16.4235391Z + ln -sf /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.9/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.9/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.9/site-packages/torch/bin
2022-05-18T05:30:16.4247495Z + ln -sf '/opt/conda/lib/python3.9/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.9/site-packages/torch/bin
2022-05-18T05:30:16.4257831Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc
2022-05-18T05:30:16.4258268Z + mkdir -p test/test-reports/cpp-rpc/test_rpc
2022-05-18T05:30:16.4271223Z + /opt/conda/lib/python3.9/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml
2022-05-18T05:30:20.4442685Z [==========] Running 8 tests from 3 test suites.
2022-05-18T05:30:20.4443283Z [----------] Global test environment set-up.
2022-05-18T05:30:20.4443713Z [----------] 4 tests from WireSerialize
2022-05-18T05:30:20.4444084Z [ RUN      ] WireSerialize.Base
2022-05-18T05:30:20.4656958Z [       OK ] WireSerialize.Base (21 ms)
2022-05-18T05:30:20.4657402Z [ RUN      ] WireSerialize.RecopySparseTensors
2022-05-18T05:30:20.4761941Z [       OK ] WireSerialize.RecopySparseTensors (10 ms)
2022-05-18T05:30:20.4762426Z [ RUN      ] WireSerialize.CloneSparseTensors
2022-05-18T05:30:20.4856615Z [       OK ] WireSerialize.CloneSparseTensors (9 ms)
2022-05-18T05:30:20.4857031Z [ RUN      ] WireSerialize.Errors
2022-05-18T05:30:20.4882649Z [       OK ] WireSerialize.Errors (2 ms)
2022-05-18T05:30:20.4883100Z [----------] 4 tests from WireSerialize (44 ms total)
2022-05-18T05:30:20.4883344Z 
2022-05-18T05:30:20.4883573Z [----------] 1 test from TestE2ETensorPipe
2022-05-18T05:30:20.4883991Z [ RUN      ] TestE2ETensorPipe.TestTrainingLoop
2022-05-18T05:30:21.2309380Z [W tensorpipe_agent.cpp:728] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T05:30:21.2331560Z [       OK ] TestE2ETensorPipe.TestTrainingLoop (744 ms)
2022-05-18T05:30:21.2332266Z [----------] 1 test from TestE2ETensorPipe (744 ms total)
2022-05-18T05:30:21.2332521Z 
2022-05-18T05:30:21.2332770Z [----------] 3 tests from TensorpipeSerialize
2022-05-18T05:30:21.2333306Z [ RUN      ] TensorpipeSerialize.Base
2022-05-18T05:30:21.2333822Z [       OK ] TensorpipeSerialize.Base (0 ms)
2022-05-18T05:30:21.2334289Z [ RUN      ] TensorpipeSerialize.RecopySparseTensors
2022-05-18T05:30:21.2431332Z [       OK ] TensorpipeSerialize.RecopySparseTensors (9 ms)
2022-05-18T05:30:21.2432464Z [ RUN      ] TensorpipeSerialize.NoDeleterTensors
2022-05-18T05:30:21.2433171Z [       OK ] TensorpipeSerialize.NoDeleterTensors (0 ms)
2022-05-18T05:30:21.2433672Z [----------] 3 tests from TensorpipeSerialize (9 ms total)
2022-05-18T05:30:21.2433934Z 
2022-05-18T05:30:21.2434370Z [----------] Global test environment tear-down
2022-05-18T05:30:21.2437108Z [==========] 8 tests from 3 test suites ran. (799 ms total)
2022-05-18T05:30:21.2437759Z [  PASSED  ] 8 tests.
2022-05-18T05:30:21.2438565Z 
2022-05-18T05:30:21.2438908Z   YOU HAVE 1 DISABLED TEST
2022-05-18T05:30:21.2439206Z 
2022-05-18T05:30:21.7889027Z + cleanup
2022-05-18T05:30:21.7889296Z + retcode=0
2022-05-18T05:30:21.7889538Z + set +x
2022-05-18T05:30:21.7889774Z EXITED_USER_LAND
2022-05-18T05:30:21.7983933Z ##[group]Run pytorch/pytorch/.github/actions/get-workflow-job-id@master
2022-05-18T05:30:21.7984398Z with:
2022-05-18T05:30:21.7984944Z   github-token: ***
2022-05-18T05:30:21.7985190Z env:
2022-05-18T05:30:21.7985407Z   IN_CI: 1
2022-05-18T05:30:21.7985615Z   IS_GHA: 1
2022-05-18T05:30:21.7985864Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:30:21.7986131Z   GPU_FLAG: --gpus all
2022-05-18T05:30:21.7986361Z ##[endgroup]
2022-05-18T05:30:21.8018529Z ##[group]Run nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a
2022-05-18T05:30:21.8018847Z with:
2022-05-18T05:30:21.8019077Z   shell: bash
2022-05-18T05:30:21.8019303Z   timeout_minutes: 10
2022-05-18T05:30:21.8019554Z   max_attempts: 5
2022-05-18T05:30:21.8019808Z   retry_wait_seconds: 30
2022-05-18T05:30:21.8020338Z   command: set -x
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}"

2022-05-18T05:30:21.8020839Z   polling_interval_seconds: 1
2022-05-18T05:30:21.8021096Z   warning_on_retry: true
2022-05-18T05:30:21.8021365Z   continue_on_error: false
2022-05-18T05:30:21.8021610Z env:
2022-05-18T05:30:21.8021813Z   IN_CI: 1
2022-05-18T05:30:21.8022037Z   IS_GHA: 1
2022-05-18T05:30:21.8022289Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:30:21.8022539Z   GPU_FLAG: --gpus all
2022-05-18T05:30:21.8022930Z   GITHUB_TOKEN: ***
2022-05-18T05:30:21.8023182Z ##[endgroup]
2022-05-18T05:30:21.8464107Z 
2022-05-18T05:30:21.8538329Z + python3 -m pip install requests==2.26.0
2022-05-18T05:30:22.1382995Z Defaulting to user installation because normal site-packages is not writeable
2022-05-18T05:30:22.1595664Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0)
2022-05-18T05:30:22.1776696Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12)
2022-05-18T05:30:22.1803137Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2021.10.8)
2022-05-18T05:30:22.1813145Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.9)
2022-05-18T05:30:22.2021908Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.3)
2022-05-18T05:30:22.3107988Z ++ python3 .github/scripts/get_workflow_job_id.py 2342799949 i-08f720eed7f2b488f
2022-05-18T05:30:23.5120538Z + GHA_WORKFLOW_JOB_ID=6482671417
2022-05-18T05:30:23.5121796Z + echo '::set-output name=job-id::6482671417'
2022-05-18T05:30:23.8545253Z Command completed after 1 attempt(s).
2022-05-18T05:30:23.8545640Z 
2022-05-18T05:30:23.8696375Z Prepare all required actions
2022-05-18T05:30:23.8696795Z Getting action download info
2022-05-18T05:30:24.0256165Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2)
2022-05-18T05:30:24.1656627Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-05-18T05:30:24.1656939Z with:
2022-05-18T05:30:24.1657276Z   file-suffix: test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482671417
2022-05-18T05:30:24.1657620Z env:
2022-05-18T05:30:24.1657841Z   IN_CI: 1
2022-05-18T05:30:24.1658047Z   IS_GHA: 1
2022-05-18T05:30:24.1658292Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:30:24.1658561Z   GPU_FLAG: --gpus all
2022-05-18T05:30:24.1658793Z ##[endgroup]
2022-05-18T05:30:24.1686331Z ##[group]Run # Remove any previous test jsons if they exist
2022-05-18T05:30:24.1686941Z # Remove any previous test jsons if they exist
2022-05-18T05:30:24.1687272Z rm -f test-jsons-*.zip
2022-05-18T05:30:24.1687633Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-05-18T05:30:24.1699536Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T05:30:24.1699819Z env:
2022-05-18T05:30:24.1700039Z   IN_CI: 1
2022-05-18T05:30:24.1700262Z   IS_GHA: 1
2022-05-18T05:30:24.1700494Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:30:24.1700894Z   GPU_FLAG: --gpus all
2022-05-18T05:30:24.1701272Z   FILE_SUFFIX: test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482671417
2022-05-18T05:30:24.1701611Z ##[endgroup]
2022-05-18T05:30:24.1837241Z   adding: test/allowlist_for_publicAPI.json (deflated 82%)
2022-05-18T05:30:24.1871539Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-05-18T05:30:24.1872771Z   adding: test/.pytorch-slow-tests.json (deflated 71%)
2022-05-18T05:30:24.1876902Z   adding: test/.pytorch-disabled-tests.json (deflated 83%)
2022-05-18T05:30:24.1899902Z ##[group]Run # Remove any previous test reports if they exist
2022-05-18T05:30:24.1900329Z # Remove any previous test reports if they exist
2022-05-18T05:30:24.1900677Z rm -f test-reports-*.zip
2022-05-18T05:30:24.1901020Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml'
2022-05-18T05:30:24.1913368Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T05:30:24.1913687Z env:
2022-05-18T05:30:24.1913901Z   IN_CI: 1
2022-05-18T05:30:24.1914135Z   IS_GHA: 1
2022-05-18T05:30:24.1914398Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:30:24.1914662Z   GPU_FLAG: --gpus all
2022-05-18T05:30:24.1915057Z   FILE_SUFFIX: test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482671417
2022-05-18T05:30:24.1915426Z ##[endgroup]
2022-05-18T05:30:24.2091642Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040500.xml (deflated 41%)
2022-05-18T05:30:24.2092489Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040507.xml (deflated 41%)
2022-05-18T05:30:24.2093305Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040508.xml (deflated 42%)
2022-05-18T05:30:24.2094089Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040513.xml (deflated 42%)
2022-05-18T05:30:24.2094897Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040516.xml (deflated 42%)
2022-05-18T05:30:24.2095693Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040517.xml (deflated 42%)
2022-05-18T05:30:24.2096482Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040519.xml (deflated 41%)
2022-05-18T05:30:24.2097281Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040526.xml (deflated 40%)
2022-05-18T05:30:24.2098187Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040532.xml (deflated 40%)
2022-05-18T05:30:24.2099000Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040538.xml (deflated 39%)
2022-05-18T05:30:24.2099779Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040544.xml (deflated 39%)
2022-05-18T05:30:24.2100565Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040550.xml (deflated 40%)
2022-05-18T05:30:24.2101335Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040557.xml (deflated 40%)
2022-05-18T05:30:24.2102123Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040603.xml (deflated 42%)
2022-05-18T05:30:24.2102908Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040606.xml (deflated 41%)
2022-05-18T05:30:24.2103679Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040610.xml (deflated 42%)
2022-05-18T05:30:24.2104437Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040613.xml (deflated 42%)
2022-05-18T05:30:24.2105332Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040619.xml (deflated 42%)
2022-05-18T05:30:24.2106109Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040620.xml (deflated 46%)
2022-05-18T05:30:24.2106882Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040626.xml (deflated 47%)
2022-05-18T05:30:24.2107655Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040632.xml (deflated 48%)
2022-05-18T05:30:24.2108445Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040638.xml (deflated 45%)
2022-05-18T05:30:24.2109222Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040645.xml (deflated 41%)
2022-05-18T05:30:24.2110006Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040646.xml (deflated 43%)
2022-05-18T05:30:24.2110762Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040647.xml (deflated 43%)
2022-05-18T05:30:24.2111537Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040648.xml (deflated 43%)
2022-05-18T05:30:24.2112311Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040650.xml (deflated 43%)
2022-05-18T05:30:24.2113083Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040651.xml (deflated 43%)
2022-05-18T05:30:24.2113838Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040652.xml (deflated 41%)
2022-05-18T05:30:24.2114620Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040653.xml (deflated 40%)
2022-05-18T05:30:24.2115387Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040700.xml (deflated 40%)
2022-05-18T05:30:24.2116153Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040706.xml (deflated 41%)
2022-05-18T05:30:24.2116929Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040707.xml (deflated 41%)
2022-05-18T05:30:24.2117761Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040708.xml (deflated 40%)
2022-05-18T05:30:24.2118879Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040715.xml (deflated 40%)
2022-05-18T05:30:24.2119666Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040721.xml (deflated 40%)
2022-05-18T05:30:24.2120445Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040728.xml (deflated 40%)
2022-05-18T05:30:24.2121196Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040736.xml (deflated 41%)
2022-05-18T05:30:24.2121972Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040738.xml (deflated 41%)
2022-05-18T05:30:24.2122742Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040739.xml (deflated 41%)
2022-05-18T05:30:24.2123518Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040740.xml (deflated 41%)
2022-05-18T05:30:24.2124383Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040741.xml (deflated 40%)
2022-05-18T05:30:24.2125161Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040743.xml (deflated 41%)
2022-05-18T05:30:24.2125938Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040744.xml (deflated 41%)
2022-05-18T05:30:24.2126706Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040745.xml (deflated 40%)
2022-05-18T05:30:24.2127475Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040746.xml (deflated 41%)
2022-05-18T05:30:24.2128251Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040748.xml (deflated 41%)
2022-05-18T05:30:24.2129027Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040749.xml (deflated 41%)
2022-05-18T05:30:24.2129808Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040750.xml (deflated 41%)
2022-05-18T05:30:24.2130562Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040751.xml (deflated 41%)
2022-05-18T05:30:24.2131338Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040752.xml (deflated 41%)
2022-05-18T05:30:24.2132115Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040754.xml (deflated 41%)
2022-05-18T05:30:24.2132890Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040755.xml (deflated 41%)
2022-05-18T05:30:24.2133648Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040756.xml (deflated 41%)
2022-05-18T05:30:24.2134452Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040758.xml (deflated 41%)
2022-05-18T05:30:24.2135228Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040759.xml (deflated 41%)
2022-05-18T05:30:24.2136000Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040800.xml (deflated 41%)
2022-05-18T05:30:24.2136852Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040801.xml (deflated 41%)
2022-05-18T05:30:24.2137623Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040803.xml (deflated 42%)
2022-05-18T05:30:24.2138397Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040804.xml (deflated 41%)
2022-05-18T05:30:24.2139174Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040805.xml (deflated 41%)
2022-05-18T05:30:24.2139943Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040806.xml (deflated 43%)
2022-05-18T05:30:24.2140703Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040808.xml (deflated 43%)
2022-05-18T05:30:24.2141481Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040809.xml (deflated 41%)
2022-05-18T05:30:24.2142249Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040810.xml (deflated 41%)
2022-05-18T05:30:24.2143022Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040816.xml (deflated 41%)
2022-05-18T05:30:24.2143850Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040817.xml (deflated 41%)
2022-05-18T05:30:24.2144622Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040818.xml (deflated 41%)
2022-05-18T05:30:24.2145393Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040820.xml (deflated 41%)
2022-05-18T05:30:24.2146165Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040826.xml (deflated 40%)
2022-05-18T05:30:24.2146926Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040832.xml (deflated 41%)
2022-05-18T05:30:24.2147697Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040838.xml (deflated 42%)
2022-05-18T05:30:24.2148471Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040840.xml (deflated 42%)
2022-05-18T05:30:24.2149239Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040841.xml (deflated 41%)
2022-05-18T05:30:24.2149995Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040847.xml (deflated 41%)
2022-05-18T05:30:24.2150773Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040853.xml (deflated 42%)
2022-05-18T05:30:24.2151546Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040854.xml (deflated 41%)
2022-05-18T05:30:24.2152319Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040900.xml (deflated 42%)
2022-05-18T05:30:24.2153118Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040901.xml (deflated 42%)
2022-05-18T05:30:24.2153901Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040906.xml (deflated 42%)
2022-05-18T05:30:24.2154676Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040907.xml (deflated 42%)
2022-05-18T05:30:24.2155455Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040908.xml (deflated 41%)
2022-05-18T05:30:24.2156263Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040915.xml (deflated 41%)
2022-05-18T05:30:24.2157053Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040921.xml (deflated 42%)
2022-05-18T05:30:24.2157863Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040922.xml (deflated 40%)
2022-05-18T05:30:24.2158902Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040928.xml (deflated 42%)
2022-05-18T05:30:24.2159681Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040930.xml (deflated 42%)
2022-05-18T05:30:24.2160434Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040934.xml (deflated 42%)
2022-05-18T05:30:24.2161211Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040935.xml (deflated 42%)
2022-05-18T05:30:24.2161988Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040937.xml (deflated 41%)
2022-05-18T05:30:24.2162761Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040942.xml (deflated 40%)
2022-05-18T05:30:24.2163617Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040948.xml (deflated 42%)
2022-05-18T05:30:24.2164395Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040950.xml (deflated 40%)
2022-05-18T05:30:24.2165164Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040956.xml (deflated 42%)
2022-05-18T05:30:24.2165945Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518040957.xml (deflated 42%)
2022-05-18T05:30:24.2166704Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041001.xml (deflated 41%)
2022-05-18T05:30:24.2167480Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041008.xml (deflated 42%)
2022-05-18T05:30:24.2168252Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041013.xml (deflated 42%)
2022-05-18T05:30:24.2169030Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041014.xml (deflated 42%)
2022-05-18T05:30:24.2169785Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041015.xml (deflated 41%)
2022-05-18T05:30:24.2170559Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041022.xml (deflated 42%)
2022-05-18T05:30:24.2171332Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041023.xml (deflated 42%)
2022-05-18T05:30:24.2172104Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041027.xml (deflated 42%)
2022-05-18T05:30:24.2172861Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041029.xml (deflated 42%)
2022-05-18T05:30:24.2173636Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041033.xml (deflated 42%)
2022-05-18T05:30:24.2174411Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041034.xml (deflated 42%)
2022-05-18T05:30:24.2175178Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041036.xml (deflated 42%)
2022-05-18T05:30:24.2175998Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041037.xml (deflated 40%)
2022-05-18T05:30:24.2176792Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041038.xml (deflated 41%)
2022-05-18T05:30:24.2177573Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041039.xml (deflated 41%)
2022-05-18T05:30:24.2178344Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041044.xml (deflated 41%)
2022-05-18T05:30:24.2179097Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041050.xml (deflated 41%)
2022-05-18T05:30:24.2179863Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041054.xml (deflated 41%)
2022-05-18T05:30:24.2180633Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041100.xml (deflated 41%)
2022-05-18T05:30:24.2181402Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041105.xml (deflated 40%)
2022-05-18T05:30:24.2182240Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041111.xml (deflated 41%)
2022-05-18T05:30:24.2182995Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041117.xml (deflated 41%)
2022-05-18T05:30:24.2183762Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041121.xml (deflated 41%)
2022-05-18T05:30:24.2184532Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041123.xml (deflated 41%)
2022-05-18T05:30:24.2185306Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041129.xml (deflated 41%)
2022-05-18T05:30:24.2186060Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041130.xml (deflated 42%)
2022-05-18T05:30:24.2186828Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041132.xml (deflated 43%)
2022-05-18T05:30:24.2187599Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041133.xml (deflated 40%)
2022-05-18T05:30:24.2188365Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041140.xml (deflated 40%)
2022-05-18T05:30:24.2189122Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041146.xml (deflated 40%)
2022-05-18T05:30:24.2189895Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041152.xml (deflated 40%)
2022-05-18T05:30:24.2190666Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041158.xml (deflated 40%)
2022-05-18T05:30:24.2191439Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041204.xml (deflated 40%)
2022-05-18T05:30:24.2192200Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041210.xml (deflated 42%)
2022-05-18T05:30:24.2192970Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041213.xml (deflated 40%)
2022-05-18T05:30:24.2193746Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041219.xml (deflated 40%)
2022-05-18T05:30:24.2194624Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041225.xml (deflated 40%)
2022-05-18T05:30:24.2195416Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041231.xml (deflated 40%)
2022-05-18T05:30:24.2196168Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041237.xml (deflated 40%)
2022-05-18T05:30:24.2196956Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041243.xml (deflated 42%)
2022-05-18T05:30:24.2197724Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041245.xml (deflated 40%)
2022-05-18T05:30:24.2198701Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041251.xml (deflated 40%)
2022-05-18T05:30:24.2199461Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041257.xml (deflated 40%)
2022-05-18T05:30:24.2200240Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041303.xml (deflated 42%)
2022-05-18T05:30:24.2201012Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041306.xml (deflated 40%)
2022-05-18T05:30:24.2201909Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041312.xml (deflated 40%)
2022-05-18T05:30:24.2202661Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041319.xml (deflated 40%)
2022-05-18T05:30:24.2203434Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041326.xml (deflated 41%)
2022-05-18T05:30:24.2204205Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041327.xml (deflated 41%)
2022-05-18T05:30:24.2204981Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041328.xml (deflated 42%)
2022-05-18T05:30:24.2205732Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041330.xml (deflated 41%)
2022-05-18T05:30:24.2206509Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041331.xml (deflated 41%)
2022-05-18T05:30:24.2207281Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041332.xml (deflated 41%)
2022-05-18T05:30:24.2208047Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041333.xml (deflated 41%)
2022-05-18T05:30:24.2208801Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041335.xml (deflated 41%)
2022-05-18T05:30:24.2209581Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041336.xml (deflated 41%)
2022-05-18T05:30:24.2210348Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041337.xml (deflated 41%)
2022-05-18T05:30:24.2211119Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041338.xml (deflated 42%)
2022-05-18T05:30:24.2211872Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041340.xml (deflated 42%)
2022-05-18T05:30:24.2212641Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041341.xml (deflated 42%)
2022-05-18T05:30:24.2213405Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041344.xml (deflated 41%)
2022-05-18T05:30:24.2214247Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041351.xml (deflated 40%)
2022-05-18T05:30:24.2215038Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041357.xml (deflated 41%)
2022-05-18T05:30:24.2215794Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041358.xml (deflated 40%)
2022-05-18T05:30:24.2216577Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041404.xml (deflated 40%)
2022-05-18T05:30:24.2217347Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041410.xml (deflated 40%)
2022-05-18T05:30:24.2218120Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041427.xml (deflated 41%)
2022-05-18T05:30:24.2218880Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041434.xml (deflated 41%)
2022-05-18T05:30:24.2219647Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041440.xml (deflated 41%)
2022-05-18T05:30:24.2220416Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041447.xml (deflated 41%)
2022-05-18T05:30:24.2221263Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041453.xml (deflated 40%)
2022-05-18T05:30:24.2222017Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041459.xml (deflated 42%)
2022-05-18T05:30:24.2222784Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041502.xml (deflated 41%)
2022-05-18T05:30:24.2223563Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041511.xml (deflated 41%)
2022-05-18T05:30:24.2224334Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041517.xml (deflated 41%)
2022-05-18T05:30:24.2225087Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041523.xml (deflated 40%)
2022-05-18T05:30:24.2225866Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041529.xml (deflated 42%)
2022-05-18T05:30:24.2226632Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041532.xml (deflated 41%)
2022-05-18T05:30:24.2227404Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041539.xml (deflated 41%)
2022-05-18T05:30:24.2228156Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041545.xml (deflated 40%)
2022-05-18T05:30:24.2228926Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041551.xml (deflated 40%)
2022-05-18T05:30:24.2229693Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041557.xml (deflated 42%)
2022-05-18T05:30:24.2230472Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041600.xml (deflated 40%)
2022-05-18T05:30:24.2231222Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041606.xml (deflated 40%)
2022-05-18T05:30:24.2231994Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041613.xml (deflated 40%)
2022-05-18T05:30:24.2232767Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041617.xml (deflated 40%)
2022-05-18T05:30:24.2233590Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041622.xml (deflated 40%)
2022-05-18T05:30:24.2234361Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041628.xml (deflated 41%)
2022-05-18T05:30:24.2235147Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041634.xml (deflated 41%)
2022-05-18T05:30:24.2235914Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041638.xml (deflated 41%)
2022-05-18T05:30:24.2236686Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041640.xml (deflated 41%)
2022-05-18T05:30:24.2237441Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041641.xml (deflated 40%)
2022-05-18T05:30:24.2238416Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041647.xml (deflated 41%)
2022-05-18T05:30:24.2239205Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041648.xml (deflated 41%)
2022-05-18T05:30:24.2240078Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041650.xml (deflated 40%)
2022-05-18T05:30:24.2240840Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041657.xml (deflated 40%)
2022-05-18T05:30:24.2241617Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041704.xml (deflated 41%)
2022-05-18T05:30:24.2242389Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041708.xml (deflated 40%)
2022-05-18T05:30:24.2243165Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041714.xml (deflated 40%)
2022-05-18T05:30:24.2243919Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041719.xml (deflated 40%)
2022-05-18T05:30:24.2244693Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041724.xml (deflated 40%)
2022-05-18T05:30:24.2245466Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041728.xml (deflated 40%)
2022-05-18T05:30:24.2246238Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041734.xml (deflated 42%)
2022-05-18T05:30:24.2246988Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041735.xml (deflated 42%)
2022-05-18T05:30:24.2247766Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041737.xml (deflated 42%)
2022-05-18T05:30:24.2248536Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041738.xml (deflated 42%)
2022-05-18T05:30:24.2249307Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041739.xml (deflated 39%)
2022-05-18T05:30:24.2250068Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041740.xml (deflated 40%)
2022-05-18T05:30:24.2250916Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041742.xml (deflated 42%)
2022-05-18T05:30:24.2251669Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041743.xml (deflated 41%)
2022-05-18T05:30:24.2252512Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041744.xml (deflated 41%)
2022-05-18T05:30:24.2253348Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041745.xml (deflated 41%)
2022-05-18T05:30:24.2254126Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041747.xml (deflated 41%)
2022-05-18T05:30:24.2254889Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041748.xml (deflated 41%)
2022-05-18T05:30:24.2255661Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041754.xml (deflated 41%)
2022-05-18T05:30:24.2256431Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041800.xml (deflated 41%)
2022-05-18T05:30:24.2257198Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041806.xml (deflated 41%)
2022-05-18T05:30:24.2257956Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041811.xml (deflated 40%)
2022-05-18T05:30:24.2258728Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041819.xml (deflated 42%)
2022-05-18T05:30:24.2259561Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041820.xml (deflated 42%)
2022-05-18T05:30:24.2260338Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041821.xml (deflated 41%)
2022-05-18T05:30:24.2261092Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041823.xml (deflated 40%)
2022-05-18T05:30:24.2261868Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041827.xml (deflated 40%)
2022-05-18T05:30:24.2262647Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041832.xml (deflated 41%)
2022-05-18T05:30:24.2263417Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041833.xml (deflated 42%)
2022-05-18T05:30:24.2264173Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041834.xml (deflated 40%)
2022-05-18T05:30:24.2264945Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041840.xml (deflated 40%)
2022-05-18T05:30:24.2265711Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041847.xml (deflated 41%)
2022-05-18T05:30:24.2266476Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041854.xml (deflated 40%)
2022-05-18T05:30:24.2267234Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041900.xml (deflated 42%)
2022-05-18T05:30:24.2268011Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041903.xml (deflated 41%)
2022-05-18T05:30:24.2268785Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041906.xml (deflated 40%)
2022-05-18T05:30:24.2269563Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041911.xml (deflated 40%)
2022-05-18T05:30:24.2270311Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041915.xml (deflated 41%)
2022-05-18T05:30:24.2271082Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041917.xml (deflated 41%)
2022-05-18T05:30:24.2271911Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041918.xml (deflated 41%)
2022-05-18T05:30:24.2272698Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041919.xml (deflated 41%)
2022-05-18T05:30:24.2273450Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041920.xml (deflated 41%)
2022-05-18T05:30:24.2274240Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041922.xml (deflated 41%)
2022-05-18T05:30:24.2275012Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041923.xml (deflated 41%)
2022-05-18T05:30:24.2275785Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041924.xml (deflated 41%)
2022-05-18T05:30:24.2276536Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041925.xml (deflated 41%)
2022-05-18T05:30:24.2277314Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041927.xml (deflated 41%)
2022-05-18T05:30:24.2278370Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041928.xml (deflated 41%)
2022-05-18T05:30:24.2279273Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041934.xml (deflated 41%)
2022-05-18T05:30:24.2280026Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041936.xml (deflated 41%)
2022-05-18T05:30:24.2280800Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041937.xml (deflated 40%)
2022-05-18T05:30:24.2281579Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041943.xml (deflated 40%)
2022-05-18T05:30:24.2282356Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041949.xml (deflated 41%)
2022-05-18T05:30:24.2283131Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041951.xml (deflated 41%)
2022-05-18T05:30:24.2283893Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041952.xml (deflated 41%)
2022-05-18T05:30:24.2284666Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041953.xml (deflated 41%)
2022-05-18T05:30:24.2285437Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041954.xml (deflated 41%)
2022-05-18T05:30:24.2286208Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042001.xml (deflated 41%)
2022-05-18T05:30:24.2286968Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042007.xml (deflated 41%)
2022-05-18T05:30:24.2287737Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042008.xml (deflated 42%)
2022-05-18T05:30:24.2288513Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042010.xml (deflated 41%)
2022-05-18T05:30:24.2289287Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042011.xml (deflated 43%)
2022-05-18T05:30:24.2290038Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042012.xml (deflated 42%)
2022-05-18T05:30:24.2290807Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042013.xml (deflated 42%)
2022-05-18T05:30:24.2291651Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042015.xml (deflated 42%)
2022-05-18T05:30:24.2292442Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042016.xml (deflated 43%)
2022-05-18T05:30:24.2293200Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042017.xml (deflated 41%)
2022-05-18T05:30:24.2293974Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042023.xml (deflated 40%)
2022-05-18T05:30:24.2294744Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042029.xml (deflated 40%)
2022-05-18T05:30:24.2295513Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042037.xml (deflated 43%)
2022-05-18T05:30:24.2296269Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042038.xml (deflated 43%)
2022-05-18T05:30:24.2297043Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042039.xml (deflated 42%)
2022-05-18T05:30:24.2297885Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042040.xml (deflated 43%)
2022-05-18T05:30:24.2298661Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042042.xml (deflated 42%)
2022-05-18T05:30:24.2299408Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042043.xml (deflated 42%)
2022-05-18T05:30:24.2300173Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042044.xml (deflated 41%)
2022-05-18T05:30:24.2300953Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042050.xml (deflated 41%)
2022-05-18T05:30:24.2301722Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042052.xml (deflated 40%)
2022-05-18T05:30:24.2302473Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042058.xml (deflated 41%)
2022-05-18T05:30:24.2303248Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042104.xml (deflated 40%)
2022-05-18T05:30:24.2304021Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042115.xml (deflated 40%)
2022-05-18T05:30:24.2304787Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042128.xml (deflated 41%)
2022-05-18T05:30:24.2305539Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042135.xml (deflated 41%)
2022-05-18T05:30:24.2306319Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042136.xml (deflated 42%)
2022-05-18T05:30:24.2307082Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042140.xml (deflated 42%)
2022-05-18T05:30:24.2307852Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042143.xml (deflated 42%)
2022-05-18T05:30:24.2308606Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042144.xml (deflated 42%)
2022-05-18T05:30:24.2309379Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042146.xml (deflated 41%)
2022-05-18T05:30:24.2310209Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042153.xml (deflated 40%)
2022-05-18T05:30:24.2311000Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042159.xml (deflated 40%)
2022-05-18T05:30:24.2311752Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042205.xml (deflated 40%)
2022-05-18T05:30:24.2312532Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042211.xml (deflated 40%)
2022-05-18T05:30:24.2313302Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042217.xml (deflated 40%)
2022-05-18T05:30:24.2314074Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042223.xml (deflated 40%)
2022-05-18T05:30:24.2314820Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042229.xml (deflated 42%)
2022-05-18T05:30:24.2315591Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042232.xml (deflated 41%)
2022-05-18T05:30:24.2316359Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042237.xml (deflated 42%)
2022-05-18T05:30:24.2317202Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042240.xml (deflated 42%)
2022-05-18T05:30:24.2317966Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042246.xml (deflated 42%)
2022-05-18T05:30:24.2318920Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042247.xml (deflated 46%)
2022-05-18T05:30:24.2319694Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042253.xml (deflated 47%)
2022-05-18T05:30:24.2320465Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042259.xml (deflated 48%)
2022-05-18T05:30:24.2321233Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042305.xml (deflated 45%)
2022-05-18T05:30:24.2321999Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042311.xml (deflated 41%)
2022-05-18T05:30:24.2322765Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042313.xml (deflated 44%)
2022-05-18T05:30:24.2323532Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042314.xml (deflated 43%)
2022-05-18T05:30:24.2324305Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042315.xml (deflated 43%)
2022-05-18T05:30:24.2325061Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042317.xml (deflated 43%)
2022-05-18T05:30:24.2325831Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042318.xml (deflated 43%)
2022-05-18T05:30:24.2326603Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042319.xml (deflated 41%)
2022-05-18T05:30:24.2327370Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042320.xml (deflated 40%)
2022-05-18T05:30:24.2328126Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042327.xml (deflated 40%)
2022-05-18T05:30:24.2328895Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042333.xml (deflated 41%)
2022-05-18T05:30:24.2329737Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042334.xml (deflated 41%)
2022-05-18T05:30:24.2330526Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042335.xml (deflated 40%)
2022-05-18T05:30:24.2331280Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042342.xml (deflated 40%)
2022-05-18T05:30:24.2332053Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042348.xml (deflated 40%)
2022-05-18T05:30:24.2332822Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042355.xml (deflated 39%)
2022-05-18T05:30:24.2333594Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042403.xml (deflated 41%)
2022-05-18T05:30:24.2334347Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042404.xml (deflated 41%)
2022-05-18T05:30:24.2335112Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042406.xml (deflated 41%)
2022-05-18T05:30:24.2335882Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042407.xml (deflated 41%)
2022-05-18T05:30:24.2336832Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042408.xml (deflated 41%)
2022-05-18T05:30:24.2337603Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042409.xml (deflated 41%)
2022-05-18T05:30:24.2338378Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042411.xml (deflated 41%)
2022-05-18T05:30:24.2339126Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042412.xml (deflated 41%)
2022-05-18T05:30:24.2339901Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042413.xml (deflated 41%)
2022-05-18T05:30:24.2340670Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042414.xml (deflated 42%)
2022-05-18T05:30:24.2341440Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042416.xml (deflated 41%)
2022-05-18T05:30:24.2342191Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042417.xml (deflated 41%)
2022-05-18T05:30:24.2342973Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042418.xml (deflated 41%)
2022-05-18T05:30:24.2343741Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042419.xml (deflated 41%)
2022-05-18T05:30:24.2344514Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042421.xml (deflated 41%)
2022-05-18T05:30:24.2345267Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042422.xml (deflated 41%)
2022-05-18T05:30:24.2346048Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042423.xml (deflated 41%)
2022-05-18T05:30:24.2346818Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042424.xml (deflated 42%)
2022-05-18T05:30:24.2347592Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042426.xml (deflated 41%)
2022-05-18T05:30:24.2348341Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042427.xml (deflated 41%)
2022-05-18T05:30:24.2349173Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042428.xml (deflated 41%)
2022-05-18T05:30:24.2349958Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042429.xml (deflated 41%)
2022-05-18T05:30:24.2350737Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042431.xml (deflated 41%)
2022-05-18T05:30:24.2351512Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042432.xml (deflated 41%)
2022-05-18T05:30:24.2352262Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042433.xml (deflated 43%)
2022-05-18T05:30:24.2353033Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042434.xml (deflated 42%)
2022-05-18T05:30:24.2353857Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042436.xml (deflated 41%)
2022-05-18T05:30:24.2354634Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042437.xml (deflated 41%)
2022-05-18T05:30:24.2355459Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042443.xml (deflated 41%)
2022-05-18T05:30:24.2356228Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042444.xml (deflated 41%)
2022-05-18T05:30:24.2356998Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042445.xml (deflated 41%)
2022-05-18T05:30:24.2357780Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042447.xml (deflated 40%)
2022-05-18T05:30:24.2358738Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042453.xml (deflated 40%)
2022-05-18T05:30:24.2359517Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042459.xml (deflated 41%)
2022-05-18T05:30:24.2360288Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042505.xml (deflated 42%)
2022-05-18T05:30:24.2361063Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042507.xml (deflated 42%)
2022-05-18T05:30:24.2361812Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042508.xml (deflated 41%)
2022-05-18T05:30:24.2362588Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042514.xml (deflated 41%)
2022-05-18T05:30:24.2363361Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042520.xml (deflated 43%)
2022-05-18T05:30:24.2364131Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042521.xml (deflated 41%)
2022-05-18T05:30:24.2364885Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042527.xml (deflated 43%)
2022-05-18T05:30:24.2365657Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042528.xml (deflated 42%)
2022-05-18T05:30:24.2366425Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042533.xml (deflated 42%)
2022-05-18T05:30:24.2367193Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042534.xml (deflated 42%)
2022-05-18T05:30:24.2368027Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042535.xml (deflated 41%)
2022-05-18T05:30:24.2368815Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042541.xml (deflated 41%)
2022-05-18T05:30:24.2369593Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042548.xml (deflated 42%)
2022-05-18T05:30:24.2370374Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042549.xml (deflated 40%)
2022-05-18T05:30:24.2371125Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042555.xml (deflated 42%)
2022-05-18T05:30:24.2371890Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042557.xml (deflated 42%)
2022-05-18T05:30:24.2372662Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042601.xml (deflated 42%)
2022-05-18T05:30:24.2373430Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042602.xml (deflated 42%)
2022-05-18T05:30:24.2374186Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042604.xml (deflated 41%)
2022-05-18T05:30:24.2375056Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042610.xml (deflated 41%)
2022-05-18T05:30:24.2375827Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042616.xml (deflated 42%)
2022-05-18T05:30:24.2376602Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042617.xml (deflated 40%)
2022-05-18T05:30:24.2377360Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042623.xml (deflated 42%)
2022-05-18T05:30:24.2378144Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042624.xml (deflated 42%)
2022-05-18T05:30:24.2378916Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042629.xml (deflated 41%)
2022-05-18T05:30:24.2379690Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042635.xml (deflated 42%)
2022-05-18T05:30:24.2380439Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042640.xml (deflated 42%)
2022-05-18T05:30:24.2381214Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042641.xml (deflated 42%)
2022-05-18T05:30:24.2381981Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042642.xml (deflated 40%)
2022-05-18T05:30:24.2382763Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042649.xml (deflated 42%)
2022-05-18T05:30:24.2383516Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042650.xml (deflated 41%)
2022-05-18T05:30:24.2384292Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042655.xml (deflated 42%)
2022-05-18T05:30:24.2385058Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042656.xml (deflated 42%)
2022-05-18T05:30:24.2385830Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042700.xml (deflated 42%)
2022-05-18T05:30:24.2386582Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042702.xml (deflated 42%)
2022-05-18T05:30:24.2387416Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042703.xml (deflated 42%)
2022-05-18T05:30:24.2388203Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042704.xml (deflated 40%)
2022-05-18T05:30:24.2388976Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042705.xml (deflated 40%)
2022-05-18T05:30:24.2389756Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042707.xml (deflated 41%)
2022-05-18T05:30:24.2390515Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042711.xml (deflated 40%)
2022-05-18T05:30:24.2391346Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042717.xml (deflated 41%)
2022-05-18T05:30:24.2392122Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042722.xml (deflated 41%)
2022-05-18T05:30:24.2392894Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042727.xml (deflated 41%)
2022-05-18T05:30:24.2393644Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042732.xml (deflated 40%)
2022-05-18T05:30:24.2394494Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042738.xml (deflated 40%)
2022-05-18T05:30:24.2395259Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042744.xml (deflated 41%)
2022-05-18T05:30:24.2396028Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042749.xml (deflated 41%)
2022-05-18T05:30:24.2396787Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042750.xml (deflated 40%)
2022-05-18T05:30:24.2397556Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042756.xml (deflated 41%)
2022-05-18T05:30:24.2398505Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042758.xml (deflated 41%)
2022-05-18T05:30:24.2399292Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042759.xml (deflated 43%)
2022-05-18T05:30:24.2400047Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042800.xml (deflated 41%)
2022-05-18T05:30:24.2400817Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042807.xml (deflated 40%)
2022-05-18T05:30:24.2401586Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042813.xml (deflated 40%)
2022-05-18T05:30:24.2402357Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042819.xml (deflated 40%)
2022-05-18T05:30:24.2403107Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042825.xml (deflated 40%)
2022-05-18T05:30:24.2403882Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042831.xml (deflated 40%)
2022-05-18T05:30:24.2404649Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042838.xml (deflated 41%)
2022-05-18T05:30:24.2405416Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042841.xml (deflated 40%)
2022-05-18T05:30:24.2406174Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042846.xml (deflated 40%)
2022-05-18T05:30:24.2407039Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042852.xml (deflated 40%)
2022-05-18T05:30:24.2407830Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042858.xml (deflated 40%)
2022-05-18T05:30:24.2408610Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042905.xml (deflated 40%)
2022-05-18T05:30:24.2409365Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042911.xml (deflated 42%)
2022-05-18T05:30:24.2410133Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042912.xml (deflated 40%)
2022-05-18T05:30:24.2410905Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042918.xml (deflated 41%)
2022-05-18T05:30:24.2411679Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042924.xml (deflated 40%)
2022-05-18T05:30:24.2412434Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042931.xml (deflated 42%)
2022-05-18T05:30:24.2413208Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042934.xml (deflated 40%)
2022-05-18T05:30:24.2414070Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042940.xml (deflated 40%)
2022-05-18T05:30:24.2414840Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042947.xml (deflated 40%)
2022-05-18T05:30:24.2415590Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042953.xml (deflated 42%)
2022-05-18T05:30:24.2416373Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042954.xml (deflated 41%)
2022-05-18T05:30:24.2417142Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042956.xml (deflated 41%)
2022-05-18T05:30:24.2417916Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042957.xml (deflated 41%)
2022-05-18T05:30:24.2418675Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042958.xml (deflated 41%)
2022-05-18T05:30:24.2419450Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042959.xml (deflated 41%)
2022-05-18T05:30:24.2420226Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043001.xml (deflated 41%)
2022-05-18T05:30:24.2421001Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043002.xml (deflated 41%)
2022-05-18T05:30:24.2421755Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043003.xml (deflated 41%)
2022-05-18T05:30:24.2422524Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043004.xml (deflated 42%)
2022-05-18T05:30:24.2423297Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043006.xml (deflated 42%)
2022-05-18T05:30:24.2424062Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043007.xml (deflated 42%)
2022-05-18T05:30:24.2424810Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043008.xml (deflated 42%)
2022-05-18T05:30:24.2425640Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043011.xml (deflated 41%)
2022-05-18T05:30:24.2426426Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043018.xml (deflated 40%)
2022-05-18T05:30:24.2427200Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043024.xml (deflated 41%)
2022-05-18T05:30:24.2427980Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043025.xml (deflated 40%)
2022-05-18T05:30:24.2428735Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043031.xml (deflated 40%)
2022-05-18T05:30:24.2429505Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043037.xml (deflated 40%)
2022-05-18T05:30:24.2430279Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043054.xml (deflated 41%)
2022-05-18T05:30:24.2431051Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043101.xml (deflated 41%)
2022-05-18T05:30:24.2431804Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043108.xml (deflated 41%)
2022-05-18T05:30:24.2432647Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043114.xml (deflated 40%)
2022-05-18T05:30:24.2433415Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043120.xml (deflated 40%)
2022-05-18T05:30:24.2434186Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043126.xml (deflated 42%)
2022-05-18T05:30:24.2434939Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043129.xml (deflated 41%)
2022-05-18T05:30:24.2435717Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043138.xml (deflated 40%)
2022-05-18T05:30:24.2436488Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043144.xml (deflated 41%)
2022-05-18T05:30:24.2437266Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043150.xml (deflated 40%)
2022-05-18T05:30:24.2438026Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043156.xml (deflated 42%)
2022-05-18T05:30:24.2438987Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043159.xml (deflated 41%)
2022-05-18T05:30:24.2439761Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043206.xml (deflated 41%)
2022-05-18T05:30:24.2440542Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043212.xml (deflated 40%)
2022-05-18T05:30:24.2441299Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043218.xml (deflated 40%)
2022-05-18T05:30:24.2442073Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043224.xml (deflated 42%)
2022-05-18T05:30:24.2442846Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043227.xml (deflated 40%)
2022-05-18T05:30:24.2443615Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043233.xml (deflated 40%)
2022-05-18T05:30:24.2444368Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043239.xml (deflated 40%)
2022-05-18T05:30:24.2445208Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043244.xml (deflated 40%)
2022-05-18T05:30:24.2445994Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043249.xml (deflated 40%)
2022-05-18T05:30:24.2446763Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043255.xml (deflated 41%)
2022-05-18T05:30:24.2447526Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043301.xml (deflated 40%)
2022-05-18T05:30:24.2448295Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043305.xml (deflated 41%)
2022-05-18T05:30:24.2449065Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043306.xml (deflated 41%)
2022-05-18T05:30:24.2449835Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043308.xml (deflated 40%)
2022-05-18T05:30:24.2450585Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043314.xml (deflated 41%)
2022-05-18T05:30:24.2451363Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043315.xml (deflated 41%)
2022-05-18T05:30:24.2452227Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043317.xml (deflated 40%)
2022-05-18T05:30:24.2453000Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043324.xml (deflated 40%)
2022-05-18T05:30:24.2453800Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043331.xml (deflated 41%)
2022-05-18T05:30:24.2454578Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043335.xml (deflated 40%)
2022-05-18T05:30:24.2455355Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043341.xml (deflated 41%)
2022-05-18T05:30:24.2456125Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043346.xml (deflated 40%)
2022-05-18T05:30:24.2456886Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043351.xml (deflated 40%)
2022-05-18T05:30:24.2457662Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043355.xml (deflated 40%)
2022-05-18T05:30:24.2458431Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043401.xml (deflated 42%)
2022-05-18T05:30:24.2459207Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043403.xml (deflated 42%)
2022-05-18T05:30:24.2459962Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043404.xml (deflated 42%)
2022-05-18T05:30:24.2460730Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043405.xml (deflated 42%)
2022-05-18T05:30:24.2461505Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043406.xml (deflated 39%)
2022-05-18T05:30:24.2462273Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043408.xml (deflated 40%)
2022-05-18T05:30:24.2463022Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043409.xml (deflated 42%)
2022-05-18T05:30:24.2463854Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043410.xml (deflated 41%)
2022-05-18T05:30:24.2464680Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043411.xml (deflated 41%)
2022-05-18T05:30:24.2465468Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043413.xml (deflated 41%)
2022-05-18T05:30:24.2466250Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043414.xml (deflated 41%)
2022-05-18T05:30:24.2467007Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043415.xml (deflated 41%)
2022-05-18T05:30:24.2467905Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043421.xml (deflated 41%)
2022-05-18T05:30:24.2468680Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043427.xml (deflated 41%)
2022-05-18T05:30:24.2469460Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043433.xml (deflated 41%)
2022-05-18T05:30:24.2470216Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043439.xml (deflated 40%)
2022-05-18T05:30:24.2470994Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043445.xml (deflated 42%)
2022-05-18T05:30:24.2471855Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043446.xml (deflated 41%)
2022-05-18T05:30:24.2472631Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043448.xml (deflated 41%)
2022-05-18T05:30:24.2473383Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043449.xml (deflated 40%)
2022-05-18T05:30:24.2474162Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043454.xml (deflated 40%)
2022-05-18T05:30:24.2474933Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043458.xml (deflated 41%)
2022-05-18T05:30:24.2475703Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043459.xml (deflated 42%)
2022-05-18T05:30:24.2476466Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043501.xml (deflated 40%)
2022-05-18T05:30:24.2477243Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043507.xml (deflated 40%)
2022-05-18T05:30:24.2478016Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043513.xml (deflated 41%)
2022-05-18T05:30:24.2478995Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043520.xml (deflated 41%)
2022-05-18T05:30:24.2479750Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043527.xml (deflated 42%)
2022-05-18T05:30:24.2480524Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043530.xml (deflated 41%)
2022-05-18T05:30:24.2481302Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043533.xml (deflated 40%)
2022-05-18T05:30:24.2482074Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043537.xml (deflated 40%)
2022-05-18T05:30:24.2482840Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043541.xml (deflated 41%)
2022-05-18T05:30:24.2483595Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043543.xml (deflated 41%)
2022-05-18T05:30:24.2484438Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043544.xml (deflated 41%)
2022-05-18T05:30:24.2485228Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043545.xml (deflated 41%)
2022-05-18T05:30:24.2486008Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043546.xml (deflated 41%)
2022-05-18T05:30:24.2486768Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043548.xml (deflated 41%)
2022-05-18T05:30:24.2487536Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043549.xml (deflated 41%)
2022-05-18T05:30:24.2488300Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043550.xml (deflated 41%)
2022-05-18T05:30:24.2489074Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043551.xml (deflated 41%)
2022-05-18T05:30:24.2489831Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043553.xml (deflated 41%)
2022-05-18T05:30:24.2490698Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043554.xml (deflated 40%)
2022-05-18T05:30:24.2491475Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043600.xml (deflated 41%)
2022-05-18T05:30:24.2492252Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043601.xml (deflated 41%)
2022-05-18T05:30:24.2493001Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043603.xml (deflated 40%)
2022-05-18T05:30:24.2493779Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043609.xml (deflated 41%)
2022-05-18T05:30:24.2494554Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043615.xml (deflated 41%)
2022-05-18T05:30:24.2495327Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043616.xml (deflated 41%)
2022-05-18T05:30:24.2496078Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043618.xml (deflated 41%)
2022-05-18T05:30:24.2496849Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043619.xml (deflated 41%)
2022-05-18T05:30:24.2497616Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043620.xml (deflated 41%)
2022-05-18T05:30:24.2498388Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043627.xml (deflated 40%)
2022-05-18T05:30:24.2499145Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043633.xml (deflated 41%)
2022-05-18T05:30:24.2499914Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043634.xml (deflated 41%)
2022-05-18T05:30:24.2500692Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043636.xml (deflated 41%)
2022-05-18T05:30:24.2501466Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043637.xml (deflated 43%)
2022-05-18T05:30:24.2502218Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043638.xml (deflated 42%)
2022-05-18T05:30:24.2503043Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043639.xml (deflated 42%)
2022-05-18T05:30:24.2503831Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043641.xml (deflated 42%)
2022-05-18T05:30:24.2504603Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043642.xml (deflated 43%)
2022-05-18T05:30:24.2505365Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043643.xml (deflated 41%)
2022-05-18T05:30:24.2506141Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043649.xml (deflated 40%)
2022-05-18T05:30:24.2506910Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043655.xml (deflated 40%)
2022-05-18T05:30:24.2507687Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043703.xml (deflated 42%)
2022-05-18T05:30:24.2508443Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043704.xml (deflated 42%)
2022-05-18T05:30:24.2509219Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043705.xml (deflated 42%)
2022-05-18T05:30:24.2510059Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043707.xml (deflated 43%)
2022-05-18T05:30:24.2510832Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043708.xml (deflated 42%)
2022-05-18T05:30:24.2511583Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043709.xml (deflated 42%)
2022-05-18T05:30:24.2512353Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043710.xml (deflated 41%)
2022-05-18T05:30:24.2513128Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043716.xml (deflated 41%)
2022-05-18T05:30:24.2513897Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043718.xml (deflated 40%)
2022-05-18T05:30:24.2514649Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043724.xml (deflated 41%)
2022-05-18T05:30:24.2515424Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043730.xml (deflated 40%)
2022-05-18T05:30:24.2516193Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043741.xml (deflated 40%)
2022-05-18T05:30:24.2516956Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043754.xml (deflated 41%)
2022-05-18T05:30:24.2517706Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043800.xml (deflated 41%)
2022-05-18T05:30:24.2518672Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043802.xml (deflated 42%)
2022-05-18T05:30:24.2519449Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043806.xml (deflated 42%)
2022-05-18T05:30:24.2520220Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043809.xml (deflated 41%)
2022-05-18T05:30:24.2520971Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043814.xml (deflated 41%)
2022-05-18T05:30:24.2521740Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043818.xml (deflated 41%)
2022-05-18T05:30:24.2522585Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043825.xml (deflated 40%)
2022-05-18T05:30:24.2523372Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043831.xml (deflated 40%)
2022-05-18T05:30:24.2524151Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043837.xml (deflated 39%)
2022-05-18T05:30:24.2524909Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043843.xml (deflated 40%)
2022-05-18T05:30:24.2525679Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043849.xml (deflated 40%)
2022-05-18T05:30:24.2526454Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043855.xml (deflated 39%)
2022-05-18T05:30:24.2527222Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043901.xml (deflated 42%)
2022-05-18T05:30:24.2527979Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043904.xml (deflated 41%)
2022-05-18T05:30:24.2528753Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043909.xml (deflated 42%)
2022-05-18T05:30:24.2529610Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043912.xml (deflated 41%)
2022-05-18T05:30:24.2530381Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043918.xml (deflated 42%)
2022-05-18T05:30:24.2531137Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043919.xml (deflated 45%)
2022-05-18T05:30:24.2531913Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043923.xml (deflated 47%)
2022-05-18T05:30:24.2532684Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043928.xml (deflated 48%)
2022-05-18T05:30:24.2533455Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043932.xml (deflated 45%)
2022-05-18T05:30:24.2534215Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043937.xml (deflated 40%)
2022-05-18T05:30:24.2534982Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043941.xml (deflated 41%)
2022-05-18T05:30:24.2535753Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043946.xml (deflated 40%)
2022-05-18T05:30:24.2536524Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043950.xml (deflated 41%)
2022-05-18T05:30:24.2537277Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043955.xml (deflated 40%)
2022-05-18T05:30:24.2538046Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043959.xml (deflated 40%)
2022-05-18T05:30:24.2538818Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044004.xml (deflated 41%)
2022-05-18T05:30:24.2539593Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044008.xml (deflated 41%)
2022-05-18T05:30:24.2540338Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044009.xml (deflated 41%)
2022-05-18T05:30:24.2541111Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044011.xml (deflated 41%)
2022-05-18T05:30:24.2541936Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044015.xml (deflated 42%)
2022-05-18T05:30:24.2542722Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044020.xml (deflated 42%)
2022-05-18T05:30:24.2543479Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044021.xml (deflated 42%)
2022-05-18T05:30:24.2544254Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044022.xml (deflated 40%)
2022-05-18T05:30:24.2545025Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044027.xml (deflated 40%)
2022-05-18T05:30:24.2545791Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044031.xml (deflated 40%)
2022-05-18T05:30:24.2546548Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044036.xml (deflated 40%)
2022-05-18T05:30:24.2547319Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044040.xml (deflated 40%)
2022-05-18T05:30:24.2548155Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044045.xml (deflated 40%)
2022-05-18T05:30:24.2548927Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044049.xml (deflated 41%)
2022-05-18T05:30:24.2549683Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044054.xml (deflated 42%)
2022-05-18T05:30:24.2550450Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044058.xml (deflated 42%)
2022-05-18T05:30:24.2551227Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044102.xml (deflated 41%)
2022-05-18T05:30:24.2551995Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044107.xml (deflated 41%)
2022-05-18T05:30:24.2552754Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044111.xml (deflated 40%)
2022-05-18T05:30:24.2553584Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044116.xml (deflated 40%)
2022-05-18T05:30:24.2554356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044120.xml (deflated 41%)
2022-05-18T05:30:24.2555123Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044125.xml (deflated 40%)
2022-05-18T05:30:24.2555877Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044129.xml (deflated 40%)
2022-05-18T05:30:24.2556651Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044134.xml (deflated 40%)
2022-05-18T05:30:24.2557417Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044138.xml (deflated 40%)
2022-05-18T05:30:24.2558383Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044143.xml (deflated 40%)
2022-05-18T05:30:24.2559150Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044147.xml (deflated 40%)
2022-05-18T05:30:24.2559924Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044152.xml (deflated 42%)
2022-05-18T05:30:24.2560766Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044156.xml (deflated 42%)
2022-05-18T05:30:24.2561559Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044201.xml (deflated 42%)
2022-05-18T05:30:24.2562326Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044205.xml (deflated 42%)
2022-05-18T05:30:24.2563089Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044210.xml (deflated 41%)
2022-05-18T05:30:24.2563857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044214.xml (deflated 41%)
2022-05-18T05:30:24.2564630Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044218.xml (deflated 40%)
2022-05-18T05:30:24.2565402Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044225.xml (deflated 41%)
2022-05-18T05:30:24.2566155Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044231.xml (deflated 40%)
2022-05-18T05:30:24.2566920Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044235.xml (deflated 41%)
2022-05-18T05:30:24.2567778Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044241.xml (deflated 41%)
2022-05-18T05:30:24.2568550Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044246.xml (deflated 40%)
2022-05-18T05:30:24.2569309Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044250.xml (deflated 40%)
2022-05-18T05:30:24.2570079Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044255.xml (deflated 41%)
2022-05-18T05:30:24.2570854Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044303.xml (deflated 41%)
2022-05-18T05:30:24.2571629Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044311.xml (deflated 41%)
2022-05-18T05:30:24.2572390Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044319.xml (deflated 42%)
2022-05-18T05:30:24.2573160Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044321.xml (deflated 42%)
2022-05-18T05:30:24.2573935Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044322.xml (deflated 42%)
2022-05-18T05:30:24.2574707Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044323.xml (deflated 41%)
2022-05-18T05:30:24.2575462Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044325.xml (deflated 43%)
2022-05-18T05:30:24.2576235Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044326.xml (deflated 41%)
2022-05-18T05:30:24.2577016Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044327.xml (deflated 42%)
2022-05-18T05:30:24.2577786Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044328.xml (deflated 42%)
2022-05-18T05:30:24.2578540Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044330.xml (deflated 42%)
2022-05-18T05:30:24.2579306Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044331.xml (deflated 42%)
2022-05-18T05:30:24.2580131Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044332.xml (deflated 42%)
2022-05-18T05:30:24.2580917Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044333.xml (deflated 42%)
2022-05-18T05:30:24.2581679Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044335.xml (deflated 42%)
2022-05-18T05:30:24.2582449Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044336.xml (deflated 42%)
2022-05-18T05:30:24.2583224Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044337.xml (deflated 42%)
2022-05-18T05:30:24.2583989Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044338.xml (deflated 42%)
2022-05-18T05:30:24.2584751Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044340.xml (deflated 42%)
2022-05-18T05:30:24.2585529Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044341.xml (deflated 42%)
2022-05-18T05:30:24.2586299Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044342.xml (deflated 42%)
2022-05-18T05:30:24.2587141Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044343.xml (deflated 42%)
2022-05-18T05:30:24.2587894Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044345.xml (deflated 42%)
2022-05-18T05:30:24.2588674Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044346.xml (deflated 42%)
2022-05-18T05:30:24.2589446Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044347.xml (deflated 42%)
2022-05-18T05:30:24.2590217Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044349.xml (deflated 42%)
2022-05-18T05:30:24.2590964Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044350.xml (deflated 41%)
2022-05-18T05:30:24.2591743Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044357.xml (deflated 41%)
2022-05-18T05:30:24.2592520Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044401.xml (deflated 42%)
2022-05-18T05:30:24.2593287Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044402.xml (deflated 41%)
2022-05-18T05:30:24.2594043Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044408.xml (deflated 40%)
2022-05-18T05:30:24.2594814Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044414.xml (deflated 40%)
2022-05-18T05:30:24.2595579Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044420.xml (deflated 42%)
2022-05-18T05:30:24.2596353Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044424.xml (deflated 42%)
2022-05-18T05:30:24.2597105Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044429.xml (deflated 42%)
2022-05-18T05:30:24.2597872Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044433.xml (deflated 40%)
2022-05-18T05:30:24.2598885Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044439.xml (deflated 41%)
2022-05-18T05:30:24.2599730Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044440.xml (deflated 42%)
2022-05-18T05:30:24.2600519Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044444.xml (deflated 41%)
2022-05-18T05:30:24.2601286Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044449.xml (deflated 40%)
2022-05-18T05:30:24.2602058Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044453.xml (deflated 41%)
2022-05-18T05:30:24.2602830Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044455.xml (deflated 41%)
2022-05-18T05:30:24.2603597Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044456.xml (deflated 40%)
2022-05-18T05:30:24.2604353Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044457.xml (deflated 41%)
2022-05-18T05:30:24.2605122Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044458.xml (deflated 41%)
2022-05-18T05:30:24.2605995Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044500.xml (deflated 40%)
2022-05-18T05:30:24.2606765Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044501.xml (deflated 41%)
2022-05-18T05:30:24.2607516Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044502.xml (deflated 41%)
2022-05-18T05:30:24.2608283Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044503.xml (deflated 41%)
2022-05-18T05:30:24.2609063Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044508.xml (deflated 41%)
2022-05-18T05:30:24.2609837Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044514.xml (deflated 40%)
2022-05-18T05:30:24.2610593Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044518.xml (deflated 42%)
2022-05-18T05:30:24.2611365Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044523.xml (deflated 40%)
2022-05-18T05:30:24.2612136Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044529.xml (deflated 40%)
2022-05-18T05:30:24.2612902Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044533.xml (deflated 40%)
2022-05-18T05:30:24.2613658Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044539.xml (deflated 40%)
2022-05-18T05:30:24.2614425Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044545.xml (deflated 40%)
2022-05-18T05:30:24.2615192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044552.xml (deflated 40%)
2022-05-18T05:30:24.2615964Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044558.xml (deflated 40%)
2022-05-18T05:30:24.2639673Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044604.xml (deflated 42%)
2022-05-18T05:30:24.2641067Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044607.xml (deflated 40%)
2022-05-18T05:30:24.2642046Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044612.xml (deflated 40%)
2022-05-18T05:30:24.2643406Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044618.xml (deflated 40%)
2022-05-18T05:30:24.2645035Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044624.xml (deflated 40%)
2022-05-18T05:30:24.2646367Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044631.xml (deflated 40%)
2022-05-18T05:30:24.2647186Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044637.xml (deflated 40%)
2022-05-18T05:30:24.2647967Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044641.xml (deflated 41%)
2022-05-18T05:30:24.2648744Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044647.xml (deflated 41%)
2022-05-18T05:30:24.2649528Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044653.xml (deflated 40%)
2022-05-18T05:30:24.2650307Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044659.xml (deflated 41%)
2022-05-18T05:30:24.2651306Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044702.xml (deflated 40%)
2022-05-18T05:30:24.2652132Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044709.xml (deflated 40%)
2022-05-18T05:30:24.2652971Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044715.xml (deflated 40%)
2022-05-18T05:30:24.2653880Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044722.xml (deflated 41%)
2022-05-18T05:30:24.2654725Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044728.xml (deflated 41%)
2022-05-18T05:30:24.2655545Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044735.xml (deflated 41%)
2022-05-18T05:30:24.2656394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044741.xml (deflated 41%)
2022-05-18T05:30:24.2657234Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044747.xml (deflated 41%)
2022-05-18T05:30:24.2658069Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044753.xml (deflated 40%)
2022-05-18T05:30:24.2658890Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044800.xml (deflated 41%)
2022-05-18T05:30:24.2659730Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044806.xml (deflated 41%)
2022-05-18T05:30:24.2660571Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044812.xml (deflated 41%)
2022-05-18T05:30:24.2661412Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044818.xml (deflated 41%)
2022-05-18T05:30:24.2662232Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044825.xml (deflated 41%)
2022-05-18T05:30:24.2663064Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044831.xml (deflated 41%)
2022-05-18T05:30:24.2663899Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044837.xml (deflated 42%)
2022-05-18T05:30:24.2664797Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044840.xml (deflated 41%)
2022-05-18T05:30:24.2665645Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044847.xml (deflated 40%)
2022-05-18T05:30:24.2666473Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044853.xml (deflated 40%)
2022-05-18T05:30:24.2667311Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044857.xml (deflated 40%)
2022-05-18T05:30:24.2668146Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044904.xml (deflated 40%)
2022-05-18T05:30:24.2668982Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044910.xml (deflated 40%)
2022-05-18T05:30:24.2669803Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044925.xml (deflated 41%)
2022-05-18T05:30:24.2670646Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044932.xml (deflated 41%)
2022-05-18T05:30:24.2671483Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044939.xml (deflated 41%)
2022-05-18T05:30:24.2672412Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044945.xml (deflated 41%)
2022-05-18T05:30:24.2673232Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044951.xml (deflated 40%)
2022-05-18T05:30:24.2674074Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518044957.xml (deflated 42%)
2022-05-18T05:30:24.2674913Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045000.xml (deflated 42%)
2022-05-18T05:30:24.2675753Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045009.xml (deflated 40%)
2022-05-18T05:30:24.2676573Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045015.xml (deflated 40%)
2022-05-18T05:30:24.2677414Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045021.xml (deflated 40%)
2022-05-18T05:30:24.2678553Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045027.xml (deflated 42%)
2022-05-18T05:30:24.2679412Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045030.xml (deflated 41%)
2022-05-18T05:30:24.2680236Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045037.xml (deflated 41%)
2022-05-18T05:30:24.2681081Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045043.xml (deflated 41%)
2022-05-18T05:30:24.2681917Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045048.xml (deflated 40%)
2022-05-18T05:30:24.2682757Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045054.xml (deflated 42%)
2022-05-18T05:30:24.2683578Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045057.xml (deflated 40%)
2022-05-18T05:30:24.2684415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045104.xml (deflated 41%)
2022-05-18T05:30:24.2685244Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045110.xml (deflated 40%)
2022-05-18T05:30:24.2686167Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045114.xml (deflated 40%)
2022-05-18T05:30:24.2687005Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045119.xml (deflated 41%)
2022-05-18T05:30:24.2687847Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045125.xml (deflated 41%)
2022-05-18T05:30:24.2688687Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045131.xml (deflated 40%)
2022-05-18T05:30:24.2689527Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045135.xml (deflated 40%)
2022-05-18T05:30:24.2690346Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045140.xml (deflated 40%)
2022-05-18T05:30:24.2691259Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045144.xml (deflated 41%)
2022-05-18T05:30:24.2692103Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045145.xml (deflated 40%)
2022-05-18T05:30:24.2693040Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045150.xml (deflated 42%)
2022-05-18T05:30:24.2693860Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045154.xml (deflated 40%)
2022-05-18T05:30:24.2694701Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045159.xml (deflated 40%)
2022-05-18T05:30:24.2695537Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045204.xml (deflated 41%)
2022-05-18T05:30:24.2696373Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045208.xml (deflated 40%)
2022-05-18T05:30:24.2697192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045212.xml (deflated 40%)
2022-05-18T05:30:24.2698032Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045217.xml (deflated 41%)
2022-05-18T05:30:24.2698867Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045221.xml (deflated 40%)
2022-05-18T05:30:24.2699879Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045226.xml (deflated 40%)
2022-05-18T05:30:24.2700719Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045232.xml (deflated 40%)
2022-05-18T05:30:24.2701547Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045236.xml (deflated 40%)
2022-05-18T05:30:24.2702393Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045241.xml (deflated 40%)
2022-05-18T05:30:24.2703232Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045245.xml (deflated 40%)
2022-05-18T05:30:24.2704375Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045250.xml (deflated 40%)
2022-05-18T05:30:24.2705196Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045251.xml (deflated 40%)
2022-05-18T05:30:24.2706034Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045253.xml (deflated 42%)
2022-05-18T05:30:24.2706932Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045257.xml (deflated 40%)
2022-05-18T05:30:24.2707791Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045304.xml (deflated 40%)
2022-05-18T05:30:24.2708610Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045308.xml (deflated 40%)
2022-05-18T05:30:24.2709443Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045313.xml (deflated 41%)
2022-05-18T05:30:24.2710275Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045317.xml (deflated 42%)
2022-05-18T05:30:24.2711107Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045319.xml (deflated 42%)
2022-05-18T05:30:24.2711920Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045320.xml (deflated 42%)
2022-05-18T05:30:24.2712755Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045321.xml (deflated 42%)
2022-05-18T05:30:24.2713577Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045322.xml (deflated 42%)
2022-05-18T05:30:24.2714489Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045324.xml (deflated 42%)
2022-05-18T05:30:24.2715309Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045325.xml (deflated 41%)
2022-05-18T05:30:24.2716149Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045326.xml (deflated 41%)
2022-05-18T05:30:24.2716979Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045327.xml (deflated 40%)
2022-05-18T05:30:24.2717820Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045332.xml (deflated 40%)
2022-05-18T05:30:24.2718844Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045336.xml (deflated 41%)
2022-05-18T05:30:24.2719641Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045338.xml (deflated 42%)
2022-05-18T05:30:24.2720424Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045339.xml (deflated 40%)
2022-05-18T05:30:24.2721210Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045345.xml (deflated 40%)
2022-05-18T05:30:24.2721979Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045351.xml (deflated 40%)
2022-05-18T05:30:24.2722768Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045358.xml (deflated 40%)
2022-05-18T05:30:24.2723551Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045405.xml (deflated 41%)
2022-05-18T05:30:24.2724336Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045408.xml (deflated 41%)
2022-05-18T05:30:24.2725107Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045411.xml (deflated 40%)
2022-05-18T05:30:24.2725885Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045415.xml (deflated 40%)
2022-05-18T05:30:24.2726663Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045419.xml (deflated 40%)
2022-05-18T05:30:24.2727526Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045424.xml (deflated 41%)
2022-05-18T05:30:24.2728309Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045428.xml (deflated 40%)
2022-05-18T05:30:24.2729098Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045433.xml (deflated 40%)
2022-05-18T05:30:24.2729877Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045437.xml (deflated 42%)
2022-05-18T05:30:24.2730652Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045442.xml (deflated 42%)
2022-05-18T05:30:24.2731413Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045446.xml (deflated 42%)
2022-05-18T05:30:24.2732199Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045451.xml (deflated 43%)
2022-05-18T05:30:24.2732982Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045455.xml (deflated 41%)
2022-05-18T05:30:24.2733753Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045500.xml (deflated 41%)
2022-05-18T05:30:24.2734612Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045504.xml (deflated 42%)
2022-05-18T05:30:24.2735394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045506.xml (deflated 41%)
2022-05-18T05:30:24.2736174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045510.xml (deflated 40%)
2022-05-18T05:30:24.2736951Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045515.xml (deflated 42%)
2022-05-18T05:30:24.2737712Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045516.xml (deflated 42%)
2022-05-18T05:30:24.2738487Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045517.xml (deflated 40%)
2022-05-18T05:30:24.2739261Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045522.xml (deflated 41%)
2022-05-18T05:30:24.2740037Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045526.xml (deflated 40%)
2022-05-18T05:30:24.2740799Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045531.xml (deflated 40%)
2022-05-18T05:30:24.2741583Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045535.xml (deflated 41%)
2022-05-18T05:30:24.2742361Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045536.xml (deflated 41%)
2022-05-18T05:30:24.2743136Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045538.xml (deflated 40%)
2022-05-18T05:30:24.2743897Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045542.xml (deflated 42%)
2022-05-18T05:30:24.2744671Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045547.xml (deflated 40%)
2022-05-18T05:30:24.2745444Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045551.xml (deflated 41%)
2022-05-18T05:30:24.2746221Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045555.xml (deflated 41%)
2022-05-18T05:30:24.2747050Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045600.xml (deflated 40%)
2022-05-18T05:30:24.2747823Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045604.xml (deflated 40%)
2022-05-18T05:30:24.2748604Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045609.xml (deflated 41%)
2022-05-18T05:30:24.2749381Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045613.xml (deflated 41%)
2022-05-18T05:30:24.2750154Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045614.xml (deflated 40%)
2022-05-18T05:30:24.2750911Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045616.xml (deflated 41%)
2022-05-18T05:30:24.2751694Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045617.xml (deflated 41%)
2022-05-18T05:30:24.2752465Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045622.xml (deflated 41%)
2022-05-18T05:30:24.2753356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045626.xml (deflated 40%)
2022-05-18T05:30:24.2754121Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045630.xml (deflated 40%)
2022-05-18T05:30:24.2754899Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045635.xml (deflated 40%)
2022-05-18T05:30:24.2755668Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045640.xml (deflated 40%)
2022-05-18T05:30:24.2756453Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045645.xml (deflated 41%)
2022-05-18T05:30:24.2757215Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045652.xml (deflated 41%)
2022-05-18T05:30:24.2757997Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045656.xml (deflated 40%)
2022-05-18T05:30:24.2758992Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045702.xml (deflated 41%)
2022-05-18T05:30:24.2759769Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045708.xml (deflated 40%)
2022-05-18T05:30:24.2760527Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045719.xml (deflated 40%)
2022-05-18T05:30:24.2761308Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045731.xml (deflated 41%)
2022-05-18T05:30:24.2762084Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045738.xml (deflated 41%)
2022-05-18T05:30:24.2762862Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045739.xml (deflated 41%)
2022-05-18T05:30:24.2763627Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045744.xml (deflated 42%)
2022-05-18T05:30:24.2764395Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045747.xml (deflated 41%)
2022-05-18T05:30:24.2765170Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045751.xml (deflated 41%)
2022-05-18T05:30:24.2766020Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045756.xml (deflated 41%)
2022-05-18T05:30:24.2766795Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045803.xml (deflated 40%)
2022-05-18T05:30:24.2767569Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045809.xml (deflated 40%)
2022-05-18T05:30:24.2768346Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045814.xml (deflated 39%)
2022-05-18T05:30:24.2769122Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045820.xml (deflated 40%)
2022-05-18T05:30:24.2769884Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045827.xml (deflated 40%)
2022-05-18T05:30:24.2770663Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045833.xml (deflated 40%)
2022-05-18T05:30:24.2771441Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045839.xml (deflated 42%)
2022-05-18T05:30:24.2772215Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045842.xml (deflated 41%)
2022-05-18T05:30:24.2773082Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045846.xml (deflated 42%)
2022-05-18T05:30:24.2773860Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045849.xml (deflated 42%)
2022-05-18T05:30:24.2774636Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045855.xml (deflated 42%)
2022-05-18T05:30:24.2775410Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045857.xml (deflated 45%)
2022-05-18T05:30:24.2776169Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045901.xml (deflated 47%)
2022-05-18T05:30:24.2776948Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045906.xml (deflated 48%)
2022-05-18T05:30:24.2777723Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045910.xml (deflated 45%)
2022-05-18T05:30:24.2778496Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045915.xml (deflated 41%)
2022-05-18T05:30:24.2779253Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045919.xml (deflated 41%)
2022-05-18T05:30:24.2780024Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045924.xml (deflated 40%)
2022-05-18T05:30:24.2780794Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045928.xml (deflated 42%)
2022-05-18T05:30:24.2781565Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045933.xml (deflated 40%)
2022-05-18T05:30:24.2782319Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045937.xml (deflated 40%)
2022-05-18T05:30:24.2783094Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045942.xml (deflated 40%)
2022-05-18T05:30:24.2783869Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045946.xml (deflated 41%)
2022-05-18T05:30:24.2784641Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045948.xml (deflated 41%)
2022-05-18T05:30:24.2785457Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045949.xml (deflated 40%)
2022-05-18T05:30:24.2786250Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045953.xml (deflated 42%)
2022-05-18T05:30:24.2787028Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045958.xml (deflated 42%)
2022-05-18T05:30:24.2787808Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518045959.xml (deflated 42%)
2022-05-18T05:30:24.2788564Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050000.xml (deflated 40%)
2022-05-18T05:30:24.2789340Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050005.xml (deflated 40%)
2022-05-18T05:30:24.2790123Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050010.xml (deflated 41%)
2022-05-18T05:30:24.2790890Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050014.xml (deflated 40%)
2022-05-18T05:30:24.2791667Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050018.xml (deflated 41%)
2022-05-18T05:30:24.2792499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050023.xml (deflated 41%)
2022-05-18T05:30:24.2793277Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050027.xml (deflated 41%)
2022-05-18T05:30:24.2794051Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050032.xml (deflated 41%)
2022-05-18T05:30:24.2794832Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050036.xml (deflated 42%)
2022-05-18T05:30:24.2795594Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050041.xml (deflated 41%)
2022-05-18T05:30:24.2796371Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050045.xml (deflated 40%)
2022-05-18T05:30:24.2797148Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050050.xml (deflated 41%)
2022-05-18T05:30:24.2797924Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050054.xml (deflated 41%)
2022-05-18T05:30:24.2798897Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050059.xml (deflated 41%)
2022-05-18T05:30:24.2799680Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050103.xml (deflated 41%)
2022-05-18T05:30:24.2800455Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050108.xml (deflated 41%)
2022-05-18T05:30:24.2801236Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050112.xml (deflated 40%)
2022-05-18T05:30:24.2802004Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050117.xml (deflated 40%)
2022-05-18T05:30:24.2802777Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050121.xml (deflated 41%)
2022-05-18T05:30:24.2803546Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050126.xml (deflated 41%)
2022-05-18T05:30:24.2804320Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050130.xml (deflated 42%)
2022-05-18T05:30:24.2805156Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050135.xml (deflated 42%)
2022-05-18T05:30:24.2805951Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050139.xml (deflated 42%)
2022-05-18T05:30:24.2806737Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050144.xml (deflated 42%)
2022-05-18T05:30:24.2807508Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050148.xml (deflated 41%)
2022-05-18T05:30:24.2808267Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050153.xml (deflated 40%)
2022-05-18T05:30:24.2809038Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050157.xml (deflated 40%)
2022-05-18T05:30:24.2809813Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050203.xml (deflated 41%)
2022-05-18T05:30:24.2810588Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050209.xml (deflated 41%)
2022-05-18T05:30:24.2811432Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050214.xml (deflated 41%)
2022-05-18T05:30:24.2812187Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050220.xml (deflated 41%)
2022-05-18T05:30:24.2812934Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050224.xml (deflated 41%)
2022-05-18T05:30:24.2813682Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050228.xml (deflated 41%)
2022-05-18T05:30:24.2814435Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050233.xml (deflated 40%)
2022-05-18T05:30:24.2815211Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050241.xml (deflated 40%)
2022-05-18T05:30:24.2815987Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050249.xml (deflated 41%)
2022-05-18T05:30:24.2816764Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050257.xml (deflated 42%)
2022-05-18T05:30:24.2817523Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050259.xml (deflated 42%)
2022-05-18T05:30:24.2818288Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050300.xml (deflated 42%)
2022-05-18T05:30:24.2819066Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050301.xml (deflated 41%)
2022-05-18T05:30:24.2819839Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050302.xml (deflated 42%)
2022-05-18T05:30:24.2820594Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050304.xml (deflated 41%)
2022-05-18T05:30:24.2821370Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050305.xml (deflated 43%)
2022-05-18T05:30:24.2822143Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050306.xml (deflated 42%)
2022-05-18T05:30:24.2822920Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050307.xml (deflated 42%)
2022-05-18T05:30:24.2823729Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050309.xml (deflated 42%)
2022-05-18T05:30:24.2824524Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050310.xml (deflated 43%)
2022-05-18T05:30:24.2825305Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050311.xml (deflated 42%)
2022-05-18T05:30:24.2826084Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050312.xml (deflated 42%)
2022-05-18T05:30:24.2826836Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050314.xml (deflated 42%)
2022-05-18T05:30:24.2827592Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050315.xml (deflated 42%)
2022-05-18T05:30:24.2828347Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050316.xml (deflated 42%)
2022-05-18T05:30:24.2829114Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050317.xml (deflated 42%)
2022-05-18T05:30:24.2829870Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050319.xml (deflated 42%)
2022-05-18T05:30:24.2830720Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050320.xml (deflated 43%)
2022-05-18T05:30:24.2831491Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050321.xml (deflated 42%)
2022-05-18T05:30:24.2832267Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050322.xml (deflated 42%)
2022-05-18T05:30:24.2833028Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050324.xml (deflated 42%)
2022-05-18T05:30:24.2833799Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050325.xml (deflated 42%)
2022-05-18T05:30:24.2834554Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050326.xml (deflated 42%)
2022-05-18T05:30:24.2835312Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050327.xml (deflated 41%)
2022-05-18T05:30:24.2836062Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050334.xml (deflated 41%)
2022-05-18T05:30:24.2836819Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050339.xml (deflated 42%)
2022-05-18T05:30:24.2837574Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050340.xml (deflated 41%)
2022-05-18T05:30:24.2838505Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050345.xml (deflated 40%)
2022-05-18T05:30:24.2839271Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050352.xml (deflated 40%)
2022-05-18T05:30:24.2840036Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050357.xml (deflated 42%)
2022-05-18T05:30:24.2840795Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050402.xml (deflated 42%)
2022-05-18T05:30:24.2841544Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050406.xml (deflated 42%)
2022-05-18T05:30:24.2842297Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050411.xml (deflated 40%)
2022-05-18T05:30:24.2843123Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050416.xml (deflated 40%)
2022-05-18T05:30:24.2843895Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050422.xml (deflated 42%)
2022-05-18T05:30:24.2844659Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050426.xml (deflated 40%)
2022-05-18T05:30:24.2845431Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050431.xml (deflated 41%)
2022-05-18T05:30:24.2846188Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050435.xml (deflated 41%)
2022-05-18T05:30:24.2846965Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050436.xml (deflated 41%)
2022-05-18T05:30:24.2847742Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050438.xml (deflated 40%)
2022-05-18T05:30:24.2848519Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050439.xml (deflated 41%)
2022-05-18T05:30:24.2849276Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050440.xml (deflated 41%)
2022-05-18T05:30:24.2850146Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050441.xml (deflated 40%)
2022-05-18T05:30:24.2850926Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050443.xml (deflated 41%)
2022-05-18T05:30:24.2851695Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050444.xml (deflated 41%)
2022-05-18T05:30:24.2852444Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050445.xml (deflated 41%)
2022-05-18T05:30:24.2853197Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050450.xml (deflated 41%)
2022-05-18T05:30:24.2854011Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050456.xml (deflated 40%)
2022-05-18T05:30:24.2854794Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050500.xml (deflated 43%)
2022-05-18T05:30:24.2855553Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050505.xml (deflated 41%)
2022-05-18T05:30:24.2856323Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050511.xml (deflated 41%)
2022-05-18T05:30:24.2857098Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050515.xml (deflated 40%)
2022-05-18T05:30:24.2857870Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050521.xml (deflated 40%)
2022-05-18T05:30:24.2858626Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050527.xml (deflated 40%)
2022-05-18T05:30:24.2859404Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050533.xml (deflated 40%)
2022-05-18T05:30:24.2860176Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050539.xml (deflated 40%)
2022-05-18T05:30:24.2860945Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050545.xml (deflated 42%)
2022-05-18T05:30:24.2861702Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050548.xml (deflated 41%)
2022-05-18T05:30:24.2862518Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050554.xml (deflated 40%)
2022-05-18T05:30:24.2863288Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050600.xml (deflated 41%)
2022-05-18T05:30:24.2864069Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050606.xml (deflated 41%)
2022-05-18T05:30:24.2864827Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050612.xml (deflated 40%)
2022-05-18T05:30:24.2865595Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050618.xml (deflated 41%)
2022-05-18T05:30:24.2866359Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050623.xml (deflated 40%)
2022-05-18T05:30:24.2867117Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050629.xml (deflated 41%)
2022-05-18T05:30:24.2867875Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050635.xml (deflated 40%)
2022-05-18T05:30:24.2868721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050641.xml (deflated 41%)
2022-05-18T05:30:24.2869495Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050644.xml (deflated 40%)
2022-05-18T05:30:24.2870259Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050650.xml (deflated 40%)
2022-05-18T05:30:24.2871011Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050657.xml (deflated 40%)
2022-05-18T05:30:24.2871788Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050704.xml (deflated 41%)
2022-05-18T05:30:24.2872564Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050710.xml (deflated 41%)
2022-05-18T05:30:24.2873334Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050716.xml (deflated 41%)
2022-05-18T05:30:24.2874093Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050722.xml (deflated 41%)
2022-05-18T05:30:24.2874867Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050729.xml (deflated 40%)
2022-05-18T05:30:24.2875639Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050735.xml (deflated 41%)
2022-05-18T05:30:24.2876420Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050741.xml (deflated 41%)
2022-05-18T05:30:24.2877173Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050748.xml (deflated 41%)
2022-05-18T05:30:24.2877938Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050754.xml (deflated 41%)
2022-05-18T05:30:24.2878906Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050800.xml (deflated 41%)
2022-05-18T05:30:24.2879682Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050806.xml (deflated 41%)
2022-05-18T05:30:24.2880440Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050813.xml (deflated 41%)
2022-05-18T05:30:24.2881291Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050819.xml (deflated 42%)
2022-05-18T05:30:24.2882086Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050822.xml (deflated 41%)
2022-05-18T05:30:24.2882858Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050829.xml (deflated 40%)
2022-05-18T05:30:24.2883624Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050835.xml (deflated 40%)
2022-05-18T05:30:24.2884404Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050839.xml (deflated 40%)
2022-05-18T05:30:24.2885173Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050845.xml (deflated 40%)
2022-05-18T05:30:24.2885943Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050851.xml (deflated 40%)
2022-05-18T05:30:24.2886723Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050907.xml (deflated 41%)
2022-05-18T05:30:24.2887483Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050914.xml (deflated 41%)
2022-05-18T05:30:24.2888346Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050921.xml (deflated 41%)
2022-05-18T05:30:24.2889124Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050927.xml (deflated 41%)
2022-05-18T05:30:24.2889904Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050933.xml (deflated 40%)
2022-05-18T05:30:24.2890658Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050939.xml (deflated 42%)
2022-05-18T05:30:24.2891442Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050942.xml (deflated 41%)
2022-05-18T05:30:24.2892212Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050951.xml (deflated 41%)
2022-05-18T05:30:24.2892985Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518050957.xml (deflated 41%)
2022-05-18T05:30:24.2893745Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051003.xml (deflated 40%)
2022-05-18T05:30:24.2894522Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051009.xml (deflated 42%)
2022-05-18T05:30:24.2894965Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051012.xml (deflated 41%)
2022-05-18T05:30:24.2895407Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051019.xml (deflated 41%)
2022-05-18T05:30:24.2895847Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051025.xml (deflated 40%)
2022-05-18T05:30:24.2896292Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051030.xml (deflated 40%)
2022-05-18T05:30:24.2896734Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051037.xml (deflated 42%)
2022-05-18T05:30:24.2897157Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051040.xml (deflated 40%)
2022-05-18T05:30:24.2897598Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051046.xml (deflated 40%)
2022-05-18T05:30:24.2898092Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051052.xml (deflated 41%)
2022-05-18T05:30:24.2898539Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051056.xml (deflated 41%)
2022-05-18T05:30:24.2898979Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051101.xml (deflated 41%)
2022-05-18T05:30:24.2899427Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051107.xml (deflated 40%)
2022-05-18T05:30:24.2899867Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051113.xml (deflated 40%)
2022-05-18T05:30:24.2900303Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051117.xml (deflated 41%)
2022-05-18T05:30:24.2900745Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051122.xml (deflated 41%)
2022-05-18T05:30:24.2901184Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051126.xml (deflated 41%)
2022-05-18T05:30:24.2901620Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051128.xml (deflated 40%)
2022-05-18T05:30:24.2902159Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051132.xml (deflated 42%)
2022-05-18T05:30:24.2902597Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051137.xml (deflated 40%)
2022-05-18T05:30:24.2903033Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051141.xml (deflated 40%)
2022-05-18T05:30:24.2903477Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051146.xml (deflated 41%)
2022-05-18T05:30:24.2903914Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051150.xml (deflated 40%)
2022-05-18T05:30:24.2904354Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051155.xml (deflated 41%)
2022-05-18T05:30:24.2904796Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051159.xml (deflated 40%)
2022-05-18T05:30:24.2905234Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051204.xml (deflated 40%)
2022-05-18T05:30:24.2905673Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051208.xml (deflated 40%)
2022-05-18T05:30:24.2906108Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051214.xml (deflated 41%)
2022-05-18T05:30:24.2906531Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051218.xml (deflated 41%)
2022-05-18T05:30:24.2906971Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051223.xml (deflated 41%)
2022-05-18T05:30:24.2907407Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051227.xml (deflated 41%)
2022-05-18T05:30:24.2907846Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051232.xml (deflated 40%)
2022-05-18T05:30:24.2908284Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051233.xml (deflated 40%)
2022-05-18T05:30:24.2908723Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051235.xml (deflated 42%)
2022-05-18T05:30:24.2909207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051239.xml (deflated 40%)
2022-05-18T05:30:24.2909659Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051246.xml (deflated 40%)
2022-05-18T05:30:24.2910102Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051250.xml (deflated 40%)
2022-05-18T05:30:24.2910535Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051255.xml (deflated 41%)
2022-05-18T05:30:24.2910949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051259.xml (deflated 42%)
2022-05-18T05:30:24.2911392Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051301.xml (deflated 42%)
2022-05-18T05:30:24.2911834Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051302.xml (deflated 42%)
2022-05-18T05:30:24.2912270Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051303.xml (deflated 42%)
2022-05-18T05:30:24.2912764Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051304.xml (deflated 42%)
2022-05-18T05:30:24.2913196Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051306.xml (deflated 42%)
2022-05-18T05:30:24.2913635Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051307.xml (deflated 41%)
2022-05-18T05:30:24.2914071Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051308.xml (deflated 41%)
2022-05-18T05:30:24.2914514Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051309.xml (deflated 40%)
2022-05-18T05:30:24.2914950Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051314.xml (deflated 40%)
2022-05-18T05:30:24.2915380Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051318.xml (deflated 41%)
2022-05-18T05:30:24.2915802Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051320.xml (deflated 42%)
2022-05-18T05:30:24.2916234Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051321.xml (deflated 40%)
2022-05-18T05:30:24.2916665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051327.xml (deflated 40%)
2022-05-18T05:30:24.2917101Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051333.xml (deflated 40%)
2022-05-18T05:30:24.2917534Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051340.xml (deflated 40%)
2022-05-18T05:30:24.2917957Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051347.xml (deflated 41%)
2022-05-18T05:30:24.2918561Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051350.xml (deflated 41%)
2022-05-18T05:30:24.2918989Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051353.xml (deflated 40%)
2022-05-18T05:30:24.2919419Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051357.xml (deflated 40%)
2022-05-18T05:30:24.2919916Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051402.xml (deflated 40%)
2022-05-18T05:30:24.2920348Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051406.xml (deflated 40%)
2022-05-18T05:30:24.2920777Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051411.xml (deflated 41%)
2022-05-18T05:30:24.2921217Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051415.xml (deflated 40%)
2022-05-18T05:30:24.2921652Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051420.xml (deflated 42%)
2022-05-18T05:30:24.2922086Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051424.xml (deflated 42%)
2022-05-18T05:30:24.2922520Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051429.xml (deflated 42%)
2022-05-18T05:30:24.2922957Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051433.xml (deflated 42%)
2022-05-18T05:30:24.2923394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051438.xml (deflated 41%)
2022-05-18T05:30:24.2923912Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051442.xml (deflated 41%)
2022-05-18T05:30:24.2924354Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051447.xml (deflated 42%)
2022-05-18T05:30:24.2924789Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051448.xml (deflated 41%)
2022-05-18T05:30:24.2925207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051452.xml (deflated 40%)
2022-05-18T05:30:24.2925650Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051457.xml (deflated 42%)
2022-05-18T05:30:24.2926088Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051458.xml (deflated 42%)
2022-05-18T05:30:24.2926526Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051459.xml (deflated 40%)
2022-05-18T05:30:24.2926960Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051504.xml (deflated 41%)
2022-05-18T05:30:24.2927394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051508.xml (deflated 41%)
2022-05-18T05:30:24.2927826Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051513.xml (deflated 41%)
2022-05-18T05:30:24.2928262Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051517.xml (deflated 41%)
2022-05-18T05:30:24.2928693Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051519.xml (deflated 41%)
2022-05-18T05:30:24.2929133Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051520.xml (deflated 40%)
2022-05-18T05:30:24.2929549Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051524.xml (deflated 42%)
2022-05-18T05:30:24.2929982Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051529.xml (deflated 40%)
2022-05-18T05:30:24.2930415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051533.xml (deflated 41%)
2022-05-18T05:30:24.2930896Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051538.xml (deflated 41%)
2022-05-18T05:30:24.2931340Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051542.xml (deflated 40%)
2022-05-18T05:30:24.2931773Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051547.xml (deflated 40%)
2022-05-18T05:30:24.2932209Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051551.xml (deflated 41%)
2022-05-18T05:30:24.2932642Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051556.xml (deflated 41%)
2022-05-18T05:30:24.2933073Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051557.xml (deflated 41%)
2022-05-18T05:30:24.2933507Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051558.xml (deflated 41%)
2022-05-18T05:30:24.2933940Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051600.xml (deflated 41%)
2022-05-18T05:30:24.2934354Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051604.xml (deflated 40%)
2022-05-18T05:30:24.2934855Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051609.xml (deflated 40%)
2022-05-18T05:30:24.2935286Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051613.xml (deflated 40%)
2022-05-18T05:30:24.2935719Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051617.xml (deflated 41%)
2022-05-18T05:30:24.2936158Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051622.xml (deflated 40%)
2022-05-18T05:30:24.2936590Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051628.xml (deflated 41%)
2022-05-18T05:30:24.2937021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051634.xml (deflated 40%)
2022-05-18T05:30:24.2937458Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051639.xml (deflated 40%)
2022-05-18T05:30:24.2937890Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051645.xml (deflated 41%)
2022-05-18T05:30:24.2938324Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051651.xml (deflated 40%)
2022-05-18T05:30:24.2938739Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518051702.xml (deflated 40%)
2022-05-18T05:30:24.2939189Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20220518051712.xml (deflated 94%)
2022-05-18T05:30:24.2939644Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20220518051949.xml (deflated 90%)
2022-05-18T05:30:24.2940045Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518052106.xml (deflated 40%)
2022-05-18T05:30:24.2940433Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518052109.xml (deflated 39%)
2022-05-18T05:30:24.2940826Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518052111.xml (deflated 40%)
2022-05-18T05:30:24.2941216Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518052114.xml (deflated 39%)
2022-05-18T05:30:24.2941684Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518052117.xml (deflated 40%)
2022-05-18T05:30:24.2942109Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518052120.xml (deflated 40%)
2022-05-18T05:30:24.2942521Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518052122.xml (deflated 39%)
2022-05-18T05:30:24.2942918Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518052125.xml (deflated 40%)
2022-05-18T05:30:24.2943323Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220518052128.xml (deflated 39%)
2022-05-18T05:30:24.2943733Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220518052130.xml (deflated 39%)
2022-05-18T05:30:24.2944133Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518052133.xml (deflated 40%)
2022-05-18T05:30:24.2944535Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518052136.xml (deflated 39%)
2022-05-18T05:30:24.2944940Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518052139.xml (deflated 39%)
2022-05-18T05:30:24.2945406Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518052141.xml (deflated 39%)
2022-05-18T05:30:24.2945816Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518052144.xml (deflated 39%)
2022-05-18T05:30:24.2946222Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518052147.xml (deflated 39%)
2022-05-18T05:30:24.2946621Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220518052159.xml (deflated 39%)
2022-05-18T05:30:24.2946993Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052202.xml (deflated 39%)
2022-05-18T05:30:24.2947389Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052205.xml (deflated 39%)
2022-05-18T05:30:24.2947780Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052208.xml (deflated 38%)
2022-05-18T05:30:24.2948175Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052210.xml (deflated 38%)
2022-05-18T05:30:24.2948556Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052213.xml (deflated 38%)
2022-05-18T05:30:24.2948938Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052216.xml (deflated 39%)
2022-05-18T05:30:24.2949321Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052219.xml (deflated 38%)
2022-05-18T05:30:24.2949711Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518052223.xml (deflated 39%)
2022-05-18T05:30:24.2950165Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220518052226.xml (deflated 80%)
2022-05-18T05:30:24.2950614Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220518052226.xml (deflated 85%)
2022-05-18T05:30:24.2951007Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20220518052310.xml (deflated 82%)
2022-05-18T05:30:24.2951412Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20220518052310.xml (deflated 85%)
2022-05-18T05:30:24.2951881Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20220518052354.xml (deflated 84%)
2022-05-18T05:30:24.2952367Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052428.xml (deflated 42%)
2022-05-18T05:30:24.2952891Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052435.xml (deflated 42%)
2022-05-18T05:30:24.2953426Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052442.xml (deflated 43%)
2022-05-18T05:30:24.2953922Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052448.xml (deflated 42%)
2022-05-18T05:30:24.2954397Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052455.xml (deflated 42%)
2022-05-18T05:30:24.2954871Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052502.xml (deflated 42%)
2022-05-18T05:30:24.2955354Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518052509.xml (deflated 42%)
2022-05-18T05:30:24.2955724Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220518052515.xml (deflated 38%)
2022-05-18T05:30:24.2956193Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518052519.xml (deflated 41%)
2022-05-18T05:30:24.2956724Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518052522.xml (deflated 41%)
2022-05-18T05:30:24.2957187Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518052524.xml (deflated 40%)
2022-05-18T05:30:24.2957644Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518052527.xml (deflated 41%)
2022-05-18T05:30:24.2958285Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518052530.xml (deflated 42%)
2022-05-18T05:30:24.2958792Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518052534.xml (deflated 41%)
2022-05-18T05:30:24.2959278Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518052540.xml (deflated 41%)
2022-05-18T05:30:24.2959764Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518052544.xml (deflated 41%)
2022-05-18T05:30:24.2960210Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20220518052550.xml (deflated 83%)
2022-05-18T05:30:24.2960658Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20220518052615.xml (deflated 86%)
2022-05-18T05:30:24.2961126Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220518052638.xml (deflated 55%)
2022-05-18T05:30:24.2961630Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220518052638.xml (deflated 81%)
2022-05-18T05:30:24.2962110Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220518052638.xml (deflated 79%)
2022-05-18T05:30:24.2962554Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220518052638.xml (deflated 63%)
2022-05-18T05:30:24.2963003Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20220518052656.xml (deflated 83%)
2022-05-18T05:30:24.2963497Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220518052710.xml (deflated 43%)
2022-05-18T05:30:24.2964052Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220518052710.xml (deflated 43%)
2022-05-18T05:30:24.2964531Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20220518052724.xml (deflated 64%)
2022-05-18T05:30:24.2964940Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20220518052736.xml (deflated 61%)
2022-05-18T05:30:24.2965441Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20220518052747.xml (deflated 59%)
2022-05-18T05:30:24.2965957Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20220518052755.xml (deflated 73%)
2022-05-18T05:30:24.2966431Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220518052803.xml (deflated 66%)
2022-05-18T05:30:24.2966899Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220518052803.xml (deflated 78%)
2022-05-18T05:30:24.2967448Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220518052810.xml (deflated 69%)
2022-05-18T05:30:24.2968016Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20220518052816.xml (deflated 68%)
2022-05-18T05:30:24.2968494Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220518052823.xml (deflated 71%)
2022-05-18T05:30:24.2968965Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220518052828.xml (deflated 47%)
2022-05-18T05:30:24.2969415Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20220518052834.xml (deflated 41%)
2022-05-18T05:30:24.2969918Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220518052840.xml (deflated 60%)
2022-05-18T05:30:24.2970384Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20220518052845.xml (deflated 61%)
2022-05-18T05:30:24.2970870Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20220518052850.xml (deflated 59%)
2022-05-18T05:30:24.2971408Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20220518052855.xml (deflated 43%)
2022-05-18T05:30:24.2971839Z   adding: test/test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20220518052859.xml (deflated 45%)
2022-05-18T05:30:24.2972259Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220518052903.xml (deflated 63%)
2022-05-18T05:30:24.2972662Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220518052903.xml (deflated 69%)
2022-05-18T05:30:24.2973050Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220518052906.xml (deflated 69%)
2022-05-18T05:30:24.2973490Z   adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220518052909.xml (deflated 63%)
2022-05-18T05:30:24.2973819Z   adding: test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml (deflated 72%)
2022-05-18T05:30:24.2974147Z   adding: test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml (deflated 71%)
2022-05-18T05:30:24.2974468Z   adding: test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml (deflated 80%)
2022-05-18T05:30:24.2974820Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml (deflated 81%)
2022-05-18T05:30:24.2975214Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml (deflated 80%)
2022-05-18T05:30:24.2975584Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml (deflated 67%)
2022-05-18T05:30:24.2975866Z   adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%)
2022-05-18T05:30:24.3014494Z ##[group]Run seemethere/upload-artifact-s3@v4
2022-05-18T05:30:24.3014595Z with:
2022-05-18T05:30:24.3014710Z   retention-days: 14
2022-05-18T05:30:24.3014830Z   if-no-files-found: warn
2022-05-18T05:30:24.3014954Z   path: test-jsons-*.zip
2022-05-18T05:30:24.3015041Z   name: artifact
2022-05-18T05:30:24.3015157Z   s3-bucket: gha-artifacts
2022-05-18T05:30:24.3015267Z   region: us-east-1
2022-05-18T05:30:24.3015354Z env:
2022-05-18T05:30:24.3015452Z   IN_CI: 1
2022-05-18T05:30:24.3015548Z   IS_GHA: 1
2022-05-18T05:30:24.3015671Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:30:24.3015768Z   GPU_FLAG: --gpus all
2022-05-18T05:30:24.3015876Z ##[endgroup]
2022-05-18T05:30:24.7445965Z With the provided path, there will be 1 file uploaded
2022-05-18T05:30:24.7446599Z Uploading to s3 prefix: pytorch/pytorch/2342799949/1/artifact
2022-05-18T05:30:24.7457201Z Starting upload of test-jsons-test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482671417.zip
2022-05-18T05:30:24.9251599Z Finished upload of test-jsons-test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482671417.zip
2022-05-18T05:30:24.9380622Z ##[group]Run seemethere/upload-artifact-s3@v4
2022-05-18T05:30:24.9380918Z with:
2022-05-18T05:30:24.9381157Z   retention-days: 14
2022-05-18T05:30:24.9381411Z   if-no-files-found: error
2022-05-18T05:30:24.9381692Z   path: test-reports-*.zip
2022-05-18T05:30:24.9381946Z   name: artifact
2022-05-18T05:30:24.9382196Z   s3-bucket: gha-artifacts
2022-05-18T05:30:24.9382442Z   region: us-east-1
2022-05-18T05:30:24.9382675Z env:
2022-05-18T05:30:24.9382890Z   IN_CI: 1
2022-05-18T05:30:24.9383095Z   IS_GHA: 1
2022-05-18T05:30:24.9383342Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:30:24.9383615Z   GPU_FLAG: --gpus all
2022-05-18T05:30:24.9383848Z ##[endgroup]
2022-05-18T05:30:25.3626615Z With the provided path, there will be 1 file uploaded
2022-05-18T05:30:25.3627012Z Uploading to s3 prefix: pytorch/pytorch/2342799949/1/artifact
2022-05-18T05:30:25.3638421Z Starting upload of test-reports-test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482671417.zip
2022-05-18T05:30:25.5803294Z Finished upload of test-reports-test-distributed-1-2-linux.8xlarge.nvidia.gpu_6482671417.zip
2022-05-18T05:30:25.5940600Z ##[group]Run set -x
2022-05-18T05:30:25.5941005Z set -x
2022-05-18T05:30:25.5941314Z python3 -m pip install -r requirements.txt
2022-05-18T05:30:25.5941639Z python3 -m pip install boto3==1.19.12
2022-05-18T05:30:25.5942037Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-05-18T05:30:25.5955299Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T05:30:25.5955583Z env:
2022-05-18T05:30:25.5955804Z   IN_CI: 1
2022-05-18T05:30:25.5956034Z   IS_GHA: 1
2022-05-18T05:30:25.5956269Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:30:25.5956537Z   GPU_FLAG: --gpus all
2022-05-18T05:30:25.5956807Z   AWS_DEFAULT_REGION: us-east-1
2022-05-18T05:30:25.5957052Z   BRANCH: master
2022-05-18T05:30:25.5957378Z   JOB_BASE_NAME: linux-bionic-cuda10.2-py3.9-gcc7-test
2022-05-18T05:30:25.5957703Z   TEST_CONFIG: distributed
2022-05-18T05:30:25.5957944Z   SHARD_NUMBER: 1
2022-05-18T05:30:25.5958589Z   BUILD_ENVIRONMENT: linux-bionic-cuda10.2-py3.9-gcc7
2022-05-18T05:30:25.5958906Z   PR_NUMBER: 
2022-05-18T05:30:25.5959173Z   SHA1: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T05:30:25.5959444Z   TAG: 
2022-05-18T05:30:25.5959675Z   WORKFLOW_ID: 2342799949
2022-05-18T05:30:25.5960079Z   GITHUB_TOKEN: ***
2022-05-18T05:30:25.5960347Z   GHA_WORKFLOW_JOB_ID: 6482671417
2022-05-18T05:30:25.5960614Z ##[endgroup]
2022-05-18T05:30:25.5990020Z + python3 -m pip install -r requirements.txt
2022-05-18T05:30:25.8986305Z Defaulting to user installation because normal site-packages is not writeable
2022-05-18T05:30:25.9305689Z Ignoring dataclasses: markers 'python_version < "3.7"' don't match your environment
2022-05-18T05:30:25.9309263Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3)
2022-05-18T05:30:25.9346312Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.3)
2022-05-18T05:30:25.9357405Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2)
2022-05-18T05:30:25.9369188Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (1.21.6)
2022-05-18T05:30:25.9380865Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (5.9.0)
2022-05-18T05:30:25.9519723Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (6.0)
2022-05-18T05:30:25.9530574Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (2.26.0)
2022-05-18T05:30:25.9700199Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (49.1.3)
2022-05-18T05:30:25.9952093Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (1.16.0)
2022-05-18T05:30:25.9964455Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (0.6.5)
2022-05-18T05:30:25.9972216Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (4.2.0)
2022-05-18T05:30:25.9986371Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.37.1)
2022-05-18T05:30:26.0020625Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (2021.10.8)
2022-05-18T05:30:26.0033675Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (3.3)
2022-05-18T05:30:26.0049490Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (1.26.9)
2022-05-18T05:30:26.0347220Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (2.0.12)
2022-05-18T05:30:26.0993250Z + python3 -m pip install boto3==1.19.12
2022-05-18T05:30:26.3950854Z Defaulting to user installation because normal site-packages is not writeable
2022-05-18T05:30:26.4170695Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12)
2022-05-18T05:30:26.4243984Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0)
2022-05-18T05:30:26.4261483Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12)
2022-05-18T05:30:26.4326567Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2)
2022-05-18T05:30:26.4363717Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.9)
2022-05-18T05:30:26.4590141Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2022-05-18T05:30:26.4619300Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-05-18T05:30:26.5798717Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-05-18T05:30:31.7968263Z [scribe] Scribe access token not provided, sending report via boto3...
2022-05-18T05:30:31.7968901Z 
2022-05-18T05:30:31.7969496Z ----- Historic stats comparison result ------
2022-05-18T05:30:31.7969925Z 
2022-05-18T05:30:31.7970461Z     job: linux-bionic-cuda10.2-py3.9-gcc7-test
2022-05-18T05:30:31.7971199Z     commit: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T05:30:31.7971619Z 
2022-05-18T05:30:31.7972048Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-05-18T05:30:31.7972563Z 
2022-05-18T05:30:31.7972763Z     : (master)
2022-05-18T05:30:31.7973188Z     |
2022-05-18T05:30:31.7973720Z     * 3b2375291a (HEAD)              total time  1276.49s
2022-05-18T05:30:31.7974791Z     * 6e3391a7c3 (base)   6 reports, total time  2175.20s ± 1392.92s
2022-05-18T05:30:31.7975676Z     * 48581d74ad          6 reports, total time  2145.63s ± 1382.99s
2022-05-18T05:30:31.7977078Z     * c35bd8d423          7 reports, total time  2037.99s ± 1309.37s
2022-05-18T05:30:31.7977990Z     * f6beda89c6          8 reports, total time  1525.68s ± 1510.80s
2022-05-18T05:30:31.7978892Z     * ee080918df          9 reports, total time  2710.43s ± 2712.08s
2022-05-18T05:30:31.7979488Z     * bbaefdf6b5          0 reports
2022-05-18T05:30:31.7980054Z     * 7c52f204e0          0 reports
2022-05-18T05:30:31.7980565Z     * e0451d8022          0 reports
2022-05-18T05:30:31.7981354Z     * 4e2f5507d0          9 reports, total time  2696.76s ± 2644.11s
2022-05-18T05:30:31.7982242Z     * b64845eb18          9 reports, total time  2712.94s ± 2654.45s
2022-05-18T05:30:31.7982794Z     |
2022-05-18T05:30:31.7983205Z     :
2022-05-18T05:30:31.7983487Z 
2022-05-18T05:30:31.7983835Z Removed  (across  670 suites)     0 tests, totaling     0.00s
2022-05-18T05:30:31.7984575Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-05-18T05:30:31.7985324Z Added    (across   57 suites)   641 tests, totaling +3507.62s
2022-05-18T05:30:31.8509993Z Prepare all required actions
2022-05-18T05:30:31.8533426Z ##[group]Run ./.github/actions/teardown-linux
2022-05-18T05:30:31.8533707Z with:
2022-05-18T05:30:31.8533902Z env:
2022-05-18T05:30:31.8534120Z   IN_CI: 1
2022-05-18T05:30:31.8534344Z   IS_GHA: 1
2022-05-18T05:30:31.8534578Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:30:31.8534845Z   GPU_FLAG: --gpus all
2022-05-18T05:30:31.8535095Z ##[endgroup]
2022-05-18T05:30:31.8552031Z ##[group]Run .github/scripts/wait_for_ssh_to_drain.sh
2022-05-18T05:30:31.8552384Z .github/scripts/wait_for_ssh_to_drain.sh
2022-05-18T05:30:31.8565802Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T05:30:31.8566090Z env:
2022-05-18T05:30:31.8566311Z   IN_CI: 1
2022-05-18T05:30:31.8566533Z   IS_GHA: 1
2022-05-18T05:30:31.8566765Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:30:31.8567038Z   GPU_FLAG: --gpus all
2022-05-18T05:30:31.8567285Z ##[endgroup]
2022-05-18T05:30:31.8612140Z Holding runner for 2 hours until all ssh sessions have logged out
2022-05-18T05:30:31.8660359Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-05-18T05:30:31.8660782Z # ignore expansion of "docker ps -q" since it could be empty
2022-05-18T05:30:31.8661125Z # shellcheck disable=SC2046
2022-05-18T05:30:31.8661416Z docker stop $(docker ps -q) || true
2022-05-18T05:30:31.8661728Z # Prune all of the docker images
2022-05-18T05:30:31.8662025Z docker system prune -af
2022-05-18T05:30:31.8673972Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T05:30:31.8674275Z env:
2022-05-18T05:30:31.8674491Z   IN_CI: 1
2022-05-18T05:30:31.8674700Z   IS_GHA: 1
2022-05-18T05:30:31.8674948Z   GIT_DEFAULT_BRANCH: master
2022-05-18T05:30:31.8675216Z   GPU_FLAG: --gpus all
2022-05-18T05:30:31.8675448Z ##[endgroup]
2022-05-18T05:30:32.2771560Z 4e1873759d18
2022-05-18T05:30:32.9679118Z Deleted Containers:
2022-05-18T05:30:32.9679538Z 4e1873759d184a295db3cfb7e4bbb284ef9fc37ca3243b456bdcb3bd87dc97d9
2022-05-18T05:30:32.9679784Z 
2022-05-18T05:30:36.9038384Z Deleted Images:
2022-05-18T05:30:36.9039543Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T05:30:36.9040520Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7@sha256:9737b662edb86afcd12a9367db6178a57889543632c0b710c5058abe14dc048f
2022-05-18T05:30:36.9041123Z deleted: sha256:914b650c5e1ee0f842697bbae2306dd6d831a4fa7fb861ca07bf056998b8539a
2022-05-18T05:30:36.9041573Z deleted: sha256:1034dda927c8a98e2c5d65a336554b89dbbe1e12c28d4d48b88e54f147a2e4e0
2022-05-18T05:30:36.9042034Z deleted: sha256:9daaaebd2559405012ffcc55a915a07af3fd8dfffccf3a4095f52a8d3a2a0808
2022-05-18T05:30:36.9042441Z deleted: sha256:62633a2457311070c286784502f87ac7817442880550ef46ef31086f62f63bd8
2022-05-18T05:30:36.9043087Z deleted: sha256:b40174881876c17fba8e4416c64d2b2065ba27f412e978c62446c6bd9975f43d
2022-05-18T05:30:36.9043497Z deleted: sha256:2b9f3cf2c41f5277698e8a3507d610f08552eb7289a4388f78a18f4934288b8c
2022-05-18T05:30:36.9043921Z deleted: sha256:bd03b60328b2f30ca7a665b612f2cc06f82974a2523f37e690f2eb32b20e23b1
2022-05-18T05:30:36.9044336Z deleted: sha256:9ead2207e8271970850e6a2fd7eacfc78f81c37d45b383107a12ba34b33a0068
2022-05-18T05:30:36.9044772Z deleted: sha256:03fe25e910ef9c726eef212a600805ba6fdd2cba133eec3a76ae6a62e71c50a8
2022-05-18T05:30:36.9045207Z deleted: sha256:42e9502eca4ade58460a090e6049a4c886d6667dc476a43c122110e9970e0504
2022-05-18T05:30:36.9045603Z deleted: sha256:3e18692fe2820772fe2b383c23571e3871b1e76e6ed758ca077a24e1fdae6a28
2022-05-18T05:30:36.9046032Z deleted: sha256:a9c1ea768838d14bfbdde1eb39006e75c504ef0e289e20b1cf1a0960ad20d993
2022-05-18T05:30:36.9046491Z deleted: sha256:653ed47cee104744163b9185cfc53ab6e751d141965b21a2f8bff4fb24acfd37
2022-05-18T05:30:36.9046913Z deleted: sha256:2ff0727ba124b0079c011424c629c2a5e27c5d7afb7b950b5513d4ab4f5e958d
2022-05-18T05:30:36.9047425Z deleted: sha256:4c3c43891ad25595b7374a30159f60ec584375dbc3820ecb30f5ad0374e5e86e
2022-05-18T05:30:36.9047874Z deleted: sha256:fa7d613a19e64cdd36a0c27fc6a2a50dd27c841da90bfae85e542064284ab2fd
2022-05-18T05:30:36.9048328Z deleted: sha256:22ec6f7d0cdf47c266dd9f601a0c98bd88bbd7e4ce3d21c9f7e00349cf7a0f8d
2022-05-18T05:30:36.9049024Z deleted: sha256:3ddedefb6de6867b92dc64bef9ed3206b098bcec87336ba702a4eec81de23bdf
2022-05-18T05:30:36.9049773Z deleted: sha256:6d2243fa3601d3ad6f7187388ef2f63d2eb318689d897e70fafdf33f22667537
2022-05-18T05:30:36.9050428Z deleted: sha256:8d2732c0f78444380cf8b5381c7b649a2e38315a0c11b8f03c7aab8f436d5390
2022-05-18T05:30:36.9051079Z deleted: sha256:85365c4faa86a33743f2107ccd2057705ec1aba1968cfeafbd737362b5499158
2022-05-18T05:30:36.9051710Z deleted: sha256:1aa2e018ba9609d32285b9d5ae5d41d884801742d27f0cbfcd249ab14b4bd4dc
2022-05-18T05:30:36.9052329Z deleted: sha256:3e096c567269719a45cda64f50eb9814c8bb7049822811461314641c8eb96c61
2022-05-18T05:30:36.9052997Z deleted: sha256:6c5ba201ed4d2056c53645f53d30efb8e4ba80fbea2c45042319090bd48d473c
2022-05-18T05:30:36.9054057Z deleted: sha256:64928ee816f9ae39d46f7dd36a5e45302562fd147967f6ab287a487c354b6b6c
2022-05-18T05:30:36.9054756Z deleted: sha256:a57b906a61d609815d662f4f4b65996a46514b07aea462793fd4143718ffc840
2022-05-18T05:30:36.9055379Z deleted: sha256:494612c761757956fbc4227e61a4a1e63e0f9b3372cf2430e2ee002ab523cfde
2022-05-18T05:30:36.9055995Z deleted: sha256:ff2c733048c22f423a6b20c35ff08bcbb6fe1bc76306464e654ef1ee28c3d861
2022-05-18T05:30:36.9056634Z deleted: sha256:4661c25de76163d8c2e45ca688f5b819c61c6c9f8e49ed83df44db353263f033
2022-05-18T05:30:36.9057289Z deleted: sha256:e23913a427ab6a1d96fc5ac9b9916776209427c2ef8eb9f44a4d16735f8c8494
2022-05-18T05:30:36.9057970Z deleted: sha256:832b3ad6407fa37ec6d8fd8f9d28172fe3bd5f6280fad98472d09eb0bc252ae0
2022-05-18T05:30:36.9058751Z deleted: sha256:a2b9dd02872fa4e35324d54aba02a6f1f21cb993714948cb709f94a2d85029f9
2022-05-18T05:30:36.9059340Z deleted: sha256:cb96bd5b78d181c6c2779f27e47036a8e9c3e1bcf09da94039148abd1c7d05ee
2022-05-18T05:30:36.9059792Z deleted: sha256:ea52cef0d0fe0c5edd5d235153b16fb0ce71bd0120ad33ed45f75bbfa3d9eadf
2022-05-18T05:30:36.9060245Z deleted: sha256:4fb97c7eb8955725be2bae74694a3af51e36e515a6c92a1aa75965cc09864f99
2022-05-18T05:30:36.9060655Z deleted: sha256:b2537994f751dde0a341c1f0d09a833be0150eb5a1cd60c7e65874442f6475a3
2022-05-18T05:30:36.9061077Z deleted: sha256:412f35baea526807361ea20e8f0e18576bdf2c6c40bdec402e94d86222a2b56e
2022-05-18T05:30:36.9061494Z deleted: sha256:cf621551bc4ed287124425a3d232f6c751dff14e9986bf7b7a697634d2f599bc
2022-05-18T05:30:36.9061917Z deleted: sha256:8003ff14feede16807731ad20c8151882bb62d724eb628e4c99ceaa2eea2a479
2022-05-18T05:30:36.9062325Z deleted: sha256:a1270a733ee0912cf66cd39d15f2ceace3789554b56647c5a5638b6ba73e3dab
2022-05-18T05:30:36.9062772Z deleted: sha256:a2811bdab35ec13d2eb84fdf4de75cbd29c5f6e227e4f11e9e8a9de714b7e132
2022-05-18T05:30:36.9063213Z deleted: sha256:f80e00922ecb54c1458a8c92d41e262173286ff550ed7468674de42de539714b
2022-05-18T05:30:36.9063728Z deleted: sha256:eb265251ed90e139bb4bfd41d9fa6a2cc6275eab106538fead323171069af9c9
2022-05-18T05:30:36.9064182Z deleted: sha256:fbee4dd8d443dcf0791e3965ee624b8ecc7b15d503ffbf8f2912d4d1d0a0cb47
2022-05-18T05:30:36.9064851Z deleted: sha256:e2f7d8e2982218fbc16adfe64b71e1839795e7a3ea82f5ff65336d58ae4cea0b
2022-05-18T05:30:36.9065295Z deleted: sha256:275df7d7943e762bf0a85fc2a9cd297c01ecb5d87ae4d86466c3f7f704d1c778
2022-05-18T05:30:36.9065718Z deleted: sha256:c2c5293df593b2d991852fe08e5db0f8c5d3c06b64247dc508084e747e64a42e
2022-05-18T05:30:36.9066142Z deleted: sha256:986cd2e7c143559516bc8388d5dd603eec6a1be4855c777c7e7f16bf22b9fa23
2022-05-18T05:30:36.9066575Z deleted: sha256:9d6787a516e72b7ed9422c8df1a4b298d82982bdf80ee1e198eedf1e1a010d76
2022-05-18T05:30:36.9066824Z 
2022-05-18T05:30:36.9067622Z Total reclaimed space: 12.14GB
2022-05-18T05:30:36.9131805Z Post job cleanup.
2022-05-18T05:30:36.9167061Z Post job cleanup.
2022-05-18T05:30:37.0491529Z [command]/usr/bin/git version
2022-05-18T05:30:37.0540079Z git version 2.32.0
2022-05-18T05:30:37.0604270Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/e4578bcd-0aa1-420f-bc0d-f44e847070b7' before making global git config changes
2022-05-18T05:30:37.0604848Z Adding repository directory to the temporary git global config as a safe directory
2022-05-18T05:30:37.0613507Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T05:30:37.0661782Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-05-18T05:30:37.0701067Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-05-18T05:30:37.1028408Z Entering 'android/libs/fbjni'
2022-05-18T05:30:37.1070431Z Entering 'third_party/FP16'
2022-05-18T05:30:37.1113606Z Entering 'third_party/FXdiv'
2022-05-18T05:30:37.1153686Z Entering 'third_party/NNPACK'
2022-05-18T05:30:37.1195355Z Entering 'third_party/QNNPACK'
2022-05-18T05:30:37.1237097Z Entering 'third_party/XNNPACK'
2022-05-18T05:30:37.1290088Z Entering 'third_party/benchmark'
2022-05-18T05:30:37.1331740Z Entering 'third_party/cpuinfo'
2022-05-18T05:30:37.1374179Z Entering 'third_party/cub'
2022-05-18T05:30:37.1414361Z Entering 'third_party/cudnn_frontend'
2022-05-18T05:30:37.1462091Z Entering 'third_party/eigen'
2022-05-18T05:30:37.1506164Z Entering 'third_party/fbgemm'
2022-05-18T05:30:37.1547566Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T05:30:37.1589513Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T05:30:37.1631471Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T05:30:37.1674416Z Entering 'third_party/flatbuffers'
2022-05-18T05:30:37.1719920Z Entering 'third_party/fmt'
2022-05-18T05:30:37.1760579Z Entering 'third_party/foxi'
2022-05-18T05:30:37.1801581Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T05:30:37.1843127Z Entering 'third_party/gloo'
2022-05-18T05:30:37.1883586Z Entering 'third_party/googletest'
2022-05-18T05:30:37.1925413Z Entering 'third_party/ideep'
2022-05-18T05:30:37.1966089Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T05:30:37.2008817Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T05:30:37.2055680Z Entering 'third_party/ios-cmake'
2022-05-18T05:30:37.2098652Z Entering 'third_party/kineto'
2022-05-18T05:30:37.2140425Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T05:30:37.2182470Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T05:30:37.2225909Z Entering 'third_party/nccl/nccl'
2022-05-18T05:30:37.2266368Z Entering 'third_party/neon2sse'
2022-05-18T05:30:37.2307758Z Entering 'third_party/onnx'
2022-05-18T05:30:37.2362640Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T05:30:37.2404717Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T05:30:37.2447917Z Entering 'third_party/onnx-tensorrt'
2022-05-18T05:30:37.2488129Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T05:30:37.2533685Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T05:30:37.2576104Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T05:30:37.2618042Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T05:30:37.2664713Z Entering 'third_party/pocketfft'
2022-05-18T05:30:37.2705923Z Entering 'third_party/protobuf'
2022-05-18T05:30:37.2750951Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T05:30:37.2792681Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T05:30:37.2836027Z Entering 'third_party/psimd'
2022-05-18T05:30:37.2880629Z Entering 'third_party/pthreadpool'
2022-05-18T05:30:37.2921734Z Entering 'third_party/pybind11'
2022-05-18T05:30:37.2964979Z Entering 'third_party/python-enum'
2022-05-18T05:30:37.3007967Z Entering 'third_party/python-peachpy'
2022-05-18T05:30:37.3049423Z Entering 'third_party/python-six'
2022-05-18T05:30:37.3091448Z Entering 'third_party/sleef'
2022-05-18T05:30:37.3133424Z Entering 'third_party/tbb'
2022-05-18T05:30:37.3176630Z Entering 'third_party/tensorpipe'
2022-05-18T05:30:37.3217960Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T05:30:37.3258913Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T05:30:37.3299680Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T05:30:37.3341022Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T05:30:37.3380850Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T05:30:37.3424711Z Entering 'third_party/zstd'
2022-05-18T05:30:37.3486585Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-05-18T05:30:37.3516030Z http.https://github.com/.extraheader
2022-05-18T05:30:37.3527808Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-05-18T05:30:37.3568487Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-05-18T05:30:37.3889404Z Entering 'android/libs/fbjni'
2022-05-18T05:30:37.3913220Z http.https://github.com/.extraheader
2022-05-18T05:30:37.3945935Z Entering 'third_party/FP16'
2022-05-18T05:30:37.3970765Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4002317Z Entering 'third_party/FXdiv'
2022-05-18T05:30:37.4026696Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4059028Z Entering 'third_party/NNPACK'
2022-05-18T05:30:37.4084474Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4116301Z Entering 'third_party/QNNPACK'
2022-05-18T05:30:37.4140798Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4173306Z Entering 'third_party/XNNPACK'
2022-05-18T05:30:37.4196837Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4240531Z Entering 'third_party/benchmark'
2022-05-18T05:30:37.4265029Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4297016Z Entering 'third_party/cpuinfo'
2022-05-18T05:30:37.4323505Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4355655Z Entering 'third_party/cub'
2022-05-18T05:30:37.4380054Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4411223Z Entering 'third_party/cudnn_frontend'
2022-05-18T05:30:37.4435272Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4472987Z Entering 'third_party/eigen'
2022-05-18T05:30:37.4497116Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4531603Z Entering 'third_party/fbgemm'
2022-05-18T05:30:37.4556080Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4587514Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T05:30:37.4611764Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4643286Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T05:30:37.4667998Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4700947Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T05:30:37.4725400Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4758208Z Entering 'third_party/flatbuffers'
2022-05-18T05:30:37.4782839Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4817250Z Entering 'third_party/fmt'
2022-05-18T05:30:37.4842320Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4873978Z Entering 'third_party/foxi'
2022-05-18T05:30:37.4899391Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4931001Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T05:30:37.4955252Z http.https://github.com/.extraheader
2022-05-18T05:30:37.4987537Z Entering 'third_party/gloo'
2022-05-18T05:30:37.5012124Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5044292Z Entering 'third_party/googletest'
2022-05-18T05:30:37.5067909Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5099805Z Entering 'third_party/ideep'
2022-05-18T05:30:37.5124507Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5155239Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T05:30:37.5179491Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5212735Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T05:30:37.5236874Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5275506Z Entering 'third_party/ios-cmake'
2022-05-18T05:30:37.5299972Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5331885Z Entering 'third_party/kineto'
2022-05-18T05:30:37.5356682Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5388626Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T05:30:37.5414175Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5446776Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T05:30:37.5471114Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5505355Z Entering 'third_party/nccl/nccl'
2022-05-18T05:30:37.5530024Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5561816Z Entering 'third_party/neon2sse'
2022-05-18T05:30:37.5585510Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5617126Z Entering 'third_party/onnx'
2022-05-18T05:30:37.5642362Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5686686Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T05:30:37.5710917Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5742669Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T05:30:37.5768068Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5802040Z Entering 'third_party/onnx-tensorrt'
2022-05-18T05:30:37.5826042Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5856832Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T05:30:37.5881700Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5917742Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T05:30:37.5942992Z http.https://github.com/.extraheader
2022-05-18T05:30:37.5976525Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T05:30:37.6001233Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6032952Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T05:30:37.6057514Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6094963Z Entering 'third_party/pocketfft'
2022-05-18T05:30:37.6119119Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6151200Z Entering 'third_party/protobuf'
2022-05-18T05:30:37.6176070Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6212173Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T05:30:37.6236159Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6267708Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T05:30:37.6291658Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6326069Z Entering 'third_party/psimd'
2022-05-18T05:30:37.6350103Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6381930Z Entering 'third_party/pthreadpool'
2022-05-18T05:30:37.6405948Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6437426Z Entering 'third_party/pybind11'
2022-05-18T05:30:37.6462014Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6493901Z Entering 'third_party/python-enum'
2022-05-18T05:30:37.6517631Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6549007Z Entering 'third_party/python-peachpy'
2022-05-18T05:30:37.6573701Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6604976Z Entering 'third_party/python-six'
2022-05-18T05:30:37.6628712Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6660979Z Entering 'third_party/sleef'
2022-05-18T05:30:37.6685429Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6716564Z Entering 'third_party/tbb'
2022-05-18T05:30:37.6740834Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6774071Z Entering 'third_party/tensorpipe'
2022-05-18T05:30:37.6798323Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6831554Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T05:30:37.6855922Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6887999Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T05:30:37.6911728Z http.https://github.com/.extraheader
2022-05-18T05:30:37.6943708Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T05:30:37.6969150Z http.https://github.com/.extraheader
2022-05-18T05:30:37.7001747Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T05:30:37.7025150Z http.https://github.com/.extraheader
2022-05-18T05:30:37.7056838Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T05:30:37.7081495Z http.https://github.com/.extraheader
2022-05-18T05:30:37.7115898Z Entering 'third_party/zstd'
2022-05-18T05:30:37.7140285Z http.https://github.com/.extraheader
2022-05-18T05:30:37.7437854Z Cleaning up orphan processes