2022-05-18T03:57:55.3149813Z Requested labels: linux.16xlarge.nvidia.gpu
2022-05-18T03:57:55.3149903Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master
2022-05-18T03:57:55.3149929Z Waiting for a runner to pick up this job...
2022-05-18T04:02:06.6052903Z Job is about to start running on the runner: i-0d4a316768328dd7a (repository)
2022-05-18T04:02:11.8823326Z Current runner version: '2.291.1'
2022-05-18T04:02:11.8832246Z Runner name: 'i-0d4a316768328dd7a'
2022-05-18T04:02:11.8832908Z Runner group name: 'Default'
2022-05-18T04:02:11.8833617Z Machine name: 'ip-10-0-4-136'
2022-05-18T04:02:11.8836406Z ##[group]GITHUB_TOKEN Permissions
2022-05-18T04:02:11.8837130Z Actions: write
2022-05-18T04:02:11.8837532Z Checks: write
2022-05-18T04:02:11.8837892Z Contents: write
2022-05-18T04:02:11.8838686Z Deployments: write
2022-05-18T04:02:11.8839190Z Discussions: write
2022-05-18T04:02:11.8839540Z Issues: write
2022-05-18T04:02:11.8839930Z Metadata: read
2022-05-18T04:02:11.8840362Z Packages: write
2022-05-18T04:02:11.8840716Z Pages: write
2022-05-18T04:02:11.8841127Z PullRequests: write
2022-05-18T04:02:11.8841588Z RepositoryProjects: write
2022-05-18T04:02:11.8842140Z SecurityEvents: write
2022-05-18T04:02:11.8842573Z Statuses: write
2022-05-18T04:02:11.8842995Z ##[endgroup]
2022-05-18T04:02:11.8847718Z Secret source: Actions
2022-05-18T04:02:11.8848527Z Prepare workflow directory
2022-05-18T04:02:12.0221064Z Prepare all required actions
2022-05-18T04:02:12.0456275Z Getting action download info
2022-05-18T04:02:12.2338176Z Download action repository 'pytorch/pytorch@master' (SHA:7b8cf1f7366bff95e9954037a58a8bb0edaaebd3)
2022-05-18T04:02:15.3128343Z Download action repository 'nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a' (SHA:71062288b76e2b6214ebde0e673ce0de1755740a)
2022-05-18T04:02:15.4204144Z Download action repository 'seemethere/upload-artifact-s3@v4' (SHA:c1c31f57581a11fe6d4d052da6276adb2df71f1e)
2022-05-18T04:02:15.7077647Z Getting action download info
2022-05-18T04:02:15.8371894Z Download action repository 'malfet/checkout@silent-checkout' (SHA:f63e9e15406be6060f159846cd2e098f759c5246)
2022-05-18T04:02:16.0403709Z Getting action download info
2022-05-18T04:02:16.3479427Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-05-18T04:02:16.3479815Z with:
2022-05-18T04:02:16.3480069Z   submodules: recursive
2022-05-18T04:02:16.3480329Z   fetch-depth: 0
2022-05-18T04:02:16.3480568Z env:
2022-05-18T04:02:16.3480790Z   IN_CI: 1
2022-05-18T04:02:16.3481011Z   IS_GHA: 1
2022-05-18T04:02:16.3481233Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:02:16.3481534Z ##[endgroup]
2022-05-18T04:02:16.3778082Z ##[group]Run echo "${GITHUB_WORKSPACE}"
2022-05-18T04:02:16.3778450Z echo "${GITHUB_WORKSPACE}"
2022-05-18T04:02:16.3778756Z if [ -z "${NO_SUDO}" ]; then
2022-05-18T04:02:16.3779059Z   sudo rm -rf "${GITHUB_WORKSPACE}"
2022-05-18T04:02:16.3779318Z else
2022-05-18T04:02:16.3779586Z   rm -rf "${GITHUB_WORKSPACE}"
2022-05-18T04:02:16.3779849Z fi
2022-05-18T04:02:16.3780110Z mkdir "${GITHUB_WORKSPACE}"
2022-05-18T04:02:16.3799618Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:02:16.3799968Z env:
2022-05-18T04:02:16.3800186Z   IN_CI: 1
2022-05-18T04:02:16.3800421Z   IS_GHA: 1
2022-05-18T04:02:16.3800680Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:02:16.3800927Z   NO_SUDO: 
2022-05-18T04:02:16.3801173Z ##[endgroup]
2022-05-18T04:02:16.4032686Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T04:02:16.4510267Z ##[group]Run malfet/checkout@silent-checkout
2022-05-18T04:02:16.4510572Z with:
2022-05-18T04:02:16.4510838Z   ref: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T04:02:16.4511097Z   fetch-depth: 0
2022-05-18T04:02:16.4511347Z   submodules: recursive
2022-05-18T04:02:16.4511602Z   quiet-checkout: true
2022-05-18T04:02:16.4511852Z   repository: pytorch/pytorch
2022-05-18T04:02:16.4512261Z   token: ***
2022-05-18T04:02:16.4512494Z   ssh-strict: true
2022-05-18T04:02:16.4512756Z   persist-credentials: true
2022-05-18T04:02:16.4512993Z   clean: true
2022-05-18T04:02:16.4513214Z   lfs: false
2022-05-18T04:02:16.4513653Z   set-safe-directory: true
2022-05-18T04:02:16.4513887Z env:
2022-05-18T04:02:16.4514096Z   IN_CI: 1
2022-05-18T04:02:16.4514317Z   IS_GHA: 1
2022-05-18T04:02:16.4514548Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:02:16.4514982Z ##[endgroup]
2022-05-18T04:02:16.6101100Z Syncing repository: pytorch/pytorch
2022-05-18T04:02:16.6103299Z ##[group]Getting Git version info
2022-05-18T04:02:16.6103868Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-05-18T04:02:16.6104476Z [command]/usr/bin/git version
2022-05-18T04:02:16.6104764Z git version 2.32.0
2022-05-18T04:02:16.6120063Z ##[endgroup]
2022-05-18T04:02:16.6144573Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/7fa1dd48-33b3-4997-85d6-31df21a1286a' before making global git config changes
2022-05-18T04:02:16.6145183Z Adding repository directory to the temporary git global config as a safe directory
2022-05-18T04:02:16.6154267Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T04:02:16.6202026Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-05-18T04:02:16.6207397Z ##[group]Initializing the repository
2022-05-18T04:02:16.6214876Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T04:02:16.6257766Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-05-18T04:02:16.6258286Z hint: is subject to change. To configure the initial branch name to use in all
2022-05-18T04:02:16.6258715Z hint: of your new repositories, which will suppress this warning, call:
2022-05-18T04:02:16.6259048Z hint: 
2022-05-18T04:02:16.6259430Z hint: 	git config --global init.defaultBranch <name>
2022-05-18T04:02:16.6259716Z hint: 
2022-05-18T04:02:16.6260117Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-05-18T04:02:16.6260642Z hint: 'development'. The just-created branch can be renamed via this command:
2022-05-18T04:02:16.6260977Z hint: 
2022-05-18T04:02:16.6261433Z hint: 	git branch -m <name>
2022-05-18T04:02:16.6262263Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-05-18T04:02:16.6274263Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-05-18T04:02:16.6317329Z ##[endgroup]
2022-05-18T04:02:16.6317879Z ##[group]Disabling automatic garbage collection
2022-05-18T04:02:16.6325012Z [command]/usr/bin/git config --local gc.auto 0
2022-05-18T04:02:16.6366858Z ##[endgroup]
2022-05-18T04:02:16.6367370Z ##[group]Setting up auth
2022-05-18T04:02:16.6378364Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-05-18T04:02:16.6422304Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-05-18T04:02:16.6877989Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-05-18T04:02:16.6917680Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-05-18T04:02:16.7286110Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-05-18T04:02:16.7344510Z ##[endgroup]
2022-05-18T04:02:16.7345018Z ##[group]Fetching the repository
2022-05-18T04:02:16.7355307Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-05-18T04:03:07.8049669Z [command]/usr/bin/git rev-parse --verify --quiet 3b2375291aab7b48442f2e6fb1ef66cebc761e24^{object}
2022-05-18T04:03:07.8085013Z 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T04:03:07.8092482Z ##[endgroup]
2022-05-18T04:03:07.8095097Z ##[group]Determining the checkout info
2022-05-18T04:03:07.8095603Z ##[endgroup]
2022-05-18T04:03:07.8096108Z ##[group]Checking out the ref
2022-05-18T04:03:07.8100546Z [command]/usr/bin/git checkout --quiet --force 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T04:03:09.4390996Z ##[endgroup]
2022-05-18T04:03:09.4391826Z ##[group]Setting up auth for fetching submodules
2022-05-18T04:03:09.4400374Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-05-18T04:03:09.4462922Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-05-18T04:03:09.4499261Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-05-18T04:03:09.4534603Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-05-18T04:03:09.4567820Z ##[endgroup]
2022-05-18T04:03:09.4568318Z ##[group]Fetching submodules
2022-05-18T04:03:09.4575515Z [command]/usr/bin/git submodule sync --recursive
2022-05-18T04:03:09.4936885Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-05-18T04:03:09.5298955Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-05-18T04:03:09.5300916Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-05-18T04:03:09.5303689Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-05-18T04:03:09.5307175Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-05-18T04:03:09.5309965Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-05-18T04:03:09.5313952Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-05-18T04:03:09.5318492Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-05-18T04:03:09.5321444Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-05-18T04:03:09.5325057Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-05-18T04:03:09.5330439Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-05-18T04:03:09.5333041Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-05-18T04:03:09.5339978Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-05-18T04:03:09.5341442Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-05-18T04:03:09.5348188Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-05-18T04:03:09.5351092Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-05-18T04:03:09.5356981Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:03:09.5361621Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-05-18T04:03:09.5366974Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-05-18T04:03:09.5371257Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-05-18T04:03:09.5377405Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-05-18T04:03:09.5383232Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-05-18T04:03:09.5389840Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-05-18T04:03:09.5394858Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-05-18T04:03:09.5400685Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-05-18T04:03:09.5406350Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-05-18T04:03:09.5412729Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-05-18T04:03:09.5418448Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-05-18T04:03:09.5424806Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-05-18T04:03:09.5432578Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-05-18T04:03:09.5438968Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-05-18T04:03:09.5445904Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-05-18T04:03:09.5452731Z Submodule 'third_party/python-peachpy' (https://github.com/Maratyszcza/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-05-18T04:03:09.5460550Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-05-18T04:03:09.5468065Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-05-18T04:03:09.5475160Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-05-18T04:03:09.5483340Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-05-18T04:03:09.5490641Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-05-18T04:03:09.5566936Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-05-18T04:03:09.7956009Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-05-18T04:03:09.9881136Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-05-18T04:03:10.1841492Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-05-18T04:03:10.4323622Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-05-18T04:03:10.6758855Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-05-18T04:03:14.9780877Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-05-18T04:03:15.3277851Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-05-18T04:03:15.7755415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-05-18T04:03:17.1055135Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-05-18T04:03:18.7840441Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-05-18T04:03:24.9788956Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-05-18T04:03:25.5269595Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-05-18T04:03:26.6648213Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-05-18T04:03:27.7292243Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-05-18T04:03:27.9389100Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-05-18T04:03:28.4364150Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-05-18T04:03:28.7050624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-05-18T04:03:29.6426359Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-05-18T04:03:29.9751526Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-05-18T04:03:30.1480839Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-05-18T04:03:32.2589668Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-05-18T04:03:32.5991429Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-05-18T04:03:32.9611215Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-05-18T04:03:34.3074813Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-05-18T04:03:34.6757792Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-05-18T04:03:34.8943524Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-05-18T04:03:42.3003152Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-05-18T04:03:42.5002523Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-05-18T04:03:42.6963376Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-05-18T04:03:43.3228844Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-05-18T04:03:43.5296907Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-05-18T04:03:43.7740538Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-05-18T04:03:44.0414293Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-05-18T04:03:44.5345587Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-05-18T04:03:46.6208746Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-05-18T04:03:47.0709430Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-05-18T04:03:48.7345785Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-05-18T04:03:48.7801356Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-05-18T04:03:48.8220067Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-05-18T04:03:48.8820570Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-05-18T04:03:48.9407913Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-05-18T04:03:49.7901391Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-05-18T04:03:49.8503321Z Submodule path 'third_party/benchmark': checked out 'e991355c02b93fe17713efe04cbc2e278e00fdbd'
2022-05-18T04:03:50.0080028Z Submodule path 'third_party/cpuinfo': checked out '5916273f79a21551890fd3d56fc5375a78d1598d'
2022-05-18T04:03:50.0814621Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-05-18T04:03:50.4903275Z Submodule path 'third_party/cudnn_frontend': checked out '43709ab96c47e26eebcdac72f93f946d44ceffa8'
2022-05-18T04:03:50.8269505Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-05-18T04:03:50.9149543Z Submodule path 'third_party/fbgemm': checked out '2e9be65810107a9595da717f95d21924b73be833'
2022-05-18T04:03:50.9211198Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:03:50.9212694Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:03:50.9216453Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:03:50.9270681Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-05-18T04:03:51.5886464Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-05-18T04:03:52.1083739Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-05-18T04:03:52.9811075Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out '8b35b4cffb62ecb58a903bf91cb7537d7a672211'
2022-05-18T04:03:53.1418217Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-05-18T04:03:53.2468133Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-05-18T04:03:53.3922067Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-05-18T04:03:53.4640866Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2022-05-18T04:03:53.5058405Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-05-18T04:03:53.5867228Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-05-18T04:03:53.6487919Z Submodule path 'third_party/gloo': checked out 'c22a5cfba94edf8ea4f53a174d38aa0c629d070f'
2022-05-18T04:03:53.7376768Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-05-18T04:03:53.7815093Z Submodule path 'third_party/ideep': checked out '02b17c5748c9349dcc586c359af800c684d9b1ab'
2022-05-18T04:03:53.7869860Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-05-18T04:03:53.7919217Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-05-18T04:03:58.8975519Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46'
2022-05-18T04:03:58.9047976Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:03:58.9106371Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-05-18T04:04:04.7234824Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815'
2022-05-18T04:04:04.7696398Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-05-18T04:04:04.9163785Z Submodule path 'third_party/kineto': checked out 'b2b48c00c6e5bd8e807e2231adb229db6a1d1c22'
2022-05-18T04:04:04.9225680Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:04:04.9227671Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:04:04.9282363Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-05-18T04:04:05.8512624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-05-18T04:04:06.7250933Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-05-18T04:04:06.8237151Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-05-18T04:04:06.8811926Z Submodule path 'third_party/nccl/nccl': checked out '7e515921295adaab72adf56ea71a0fafb0ecb5f3'
2022-05-18T04:04:06.9305618Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-05-18T04:04:07.2848651Z Submodule path 'third_party/onnx': checked out '96046b8ccfb8e6fa82f6b2b34b3d56add2e8849c'
2022-05-18T04:04:07.2919293Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-05-18T04:04:07.2921742Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-05-18T04:04:07.2984925Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-05-18T04:04:07.6410860Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-05-18T04:04:08.3660238Z Submodule path 'third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-05-18T04:04:08.4378445Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '59a2ac2745d8a57ac94c6accced73620d59fb844'
2022-05-18T04:04:08.4881662Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-05-18T04:04:08.4940838Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:04:08.4993729Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-05-18T04:04:09.8379881Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-05-18T04:04:09.8461788Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:04:09.8463767Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:04:09.8524161Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-05-18T04:04:10.1878051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-05-18T04:04:10.8940914Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-05-18T04:04:11.0053430Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-05-18T04:04:11.0122373Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:04:11.0175553Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-05-18T04:04:11.2498607Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-05-18T04:04:11.2941032Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-05-18T04:04:11.6609389Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-05-18T04:04:11.6668441Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:04:11.6669548Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-05-18T04:04:11.6731533Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-05-18T04:04:12.0336640Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-05-18T04:04:12.8924874Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-05-18T04:04:13.0094646Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-05-18T04:04:13.0541158Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-05-18T04:04:13.0997753Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-05-18T04:04:13.1684453Z Submodule path 'third_party/pybind11': checked out '8de7772cc72daca8e947b79b83fea46214931604'
2022-05-18T04:04:13.2099712Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-05-18T04:04:13.2765730Z Submodule path 'third_party/python-peachpy': checked out '07d8fde8ac45d7705129475c0f94ed8925b93473'
2022-05-18T04:04:13.3205083Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-05-18T04:04:13.4066893Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-05-18T04:04:13.5823197Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-05-18T04:04:13.6477410Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-05-18T04:04:13.6530968Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:04:13.6532671Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:04:13.6536173Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:04:13.6539154Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:04:13.6593328Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-05-18T04:04:14.4579247Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-05-18T04:04:14.6989798Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-05-18T04:04:15.6952605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-05-18T04:04:16.4350383Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-05-18T04:04:16.4856382Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-05-18T04:04:16.6009072Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-05-18T04:04:16.6679260Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-05-18T04:04:16.6745376Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:04:16.6798995Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-05-18T04:04:16.9197256Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-05-18T04:04:17.1116328Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-05-18T04:04:17.1222536Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-05-18T04:04:17.1597116Z Entering 'android/libs/fbjni'
2022-05-18T04:04:17.1645533Z Entering 'third_party/FP16'
2022-05-18T04:04:17.1696083Z Entering 'third_party/FXdiv'
2022-05-18T04:04:17.1745617Z Entering 'third_party/NNPACK'
2022-05-18T04:04:17.1794123Z Entering 'third_party/QNNPACK'
2022-05-18T04:04:17.1840719Z Entering 'third_party/XNNPACK'
2022-05-18T04:04:17.1901730Z Entering 'third_party/benchmark'
2022-05-18T04:04:17.1952892Z Entering 'third_party/cpuinfo'
2022-05-18T04:04:17.2002089Z Entering 'third_party/cub'
2022-05-18T04:04:17.2054244Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:04:17.2108402Z Entering 'third_party/eigen'
2022-05-18T04:04:17.2159811Z Entering 'third_party/fbgemm'
2022-05-18T04:04:17.2209099Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:04:17.2257983Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:04:17.2307195Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:04:17.2359120Z Entering 'third_party/flatbuffers'
2022-05-18T04:04:17.2411065Z Entering 'third_party/fmt'
2022-05-18T04:04:17.2460662Z Entering 'third_party/foxi'
2022-05-18T04:04:17.2510983Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:04:17.2557652Z Entering 'third_party/gloo'
2022-05-18T04:04:17.2609126Z Entering 'third_party/googletest'
2022-05-18T04:04:17.2657818Z Entering 'third_party/ideep'
2022-05-18T04:04:17.2704262Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:04:17.2753802Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:04:17.2814206Z Entering 'third_party/ios-cmake'
2022-05-18T04:04:17.2860745Z Entering 'third_party/kineto'
2022-05-18T04:04:17.2911969Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:04:17.2963556Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:04:17.3015350Z Entering 'third_party/nccl/nccl'
2022-05-18T04:04:17.3061968Z Entering 'third_party/neon2sse'
2022-05-18T04:04:17.3110376Z Entering 'third_party/onnx'
2022-05-18T04:04:17.3171507Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:04:17.3220426Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:04:17.3272364Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:04:17.3318768Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:04:17.3374340Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:04:17.3423815Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:04:17.3472473Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:04:17.3527499Z Entering 'third_party/pocketfft'
2022-05-18T04:04:17.3574985Z Entering 'third_party/protobuf'
2022-05-18T04:04:17.3628467Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:04:17.3677337Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:04:17.3729844Z Entering 'third_party/psimd'
2022-05-18T04:04:17.3777486Z Entering 'third_party/pthreadpool'
2022-05-18T04:04:17.3827295Z Entering 'third_party/pybind11'
2022-05-18T04:04:17.3875207Z Entering 'third_party/python-enum'
2022-05-18T04:04:17.3925385Z Entering 'third_party/python-peachpy'
2022-05-18T04:04:17.3972240Z Entering 'third_party/python-six'
2022-05-18T04:04:17.4022886Z Entering 'third_party/sleef'
2022-05-18T04:04:17.4073101Z Entering 'third_party/tbb'
2022-05-18T04:04:17.4123978Z Entering 'third_party/tensorpipe'
2022-05-18T04:04:17.4172633Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:04:17.4223358Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:04:17.4270961Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:04:17.4320696Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:04:17.4367286Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:04:17.4420959Z Entering 'third_party/zstd'
2022-05-18T04:04:17.4481513Z ##[endgroup]
2022-05-18T04:04:17.4482339Z ##[group]Persisting credentials for submodules
2022-05-18T04:04:17.4491150Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-05-18T04:04:17.4872378Z Entering 'android/libs/fbjni'
2022-05-18T04:04:17.4923755Z Entering 'third_party/FP16'
2022-05-18T04:04:17.4971907Z Entering 'third_party/FXdiv'
2022-05-18T04:04:17.5019812Z Entering 'third_party/NNPACK'
2022-05-18T04:04:17.5069192Z Entering 'third_party/QNNPACK'
2022-05-18T04:04:17.5113734Z Entering 'third_party/XNNPACK'
2022-05-18T04:04:17.5175571Z Entering 'third_party/benchmark'
2022-05-18T04:04:17.5225432Z Entering 'third_party/cpuinfo'
2022-05-18T04:04:17.5274238Z Entering 'third_party/cub'
2022-05-18T04:04:17.5323044Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:04:17.5376106Z Entering 'third_party/eigen'
2022-05-18T04:04:17.5428202Z Entering 'third_party/fbgemm'
2022-05-18T04:04:17.5474702Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:04:17.5522375Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:04:17.5570440Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:04:17.5619422Z Entering 'third_party/flatbuffers'
2022-05-18T04:04:17.5672566Z Entering 'third_party/fmt'
2022-05-18T04:04:17.5721557Z Entering 'third_party/foxi'
2022-05-18T04:04:17.5771192Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:04:17.5819028Z Entering 'third_party/gloo'
2022-05-18T04:04:17.5869823Z Entering 'third_party/googletest'
2022-05-18T04:04:17.5916622Z Entering 'third_party/ideep'
2022-05-18T04:04:17.5963573Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:04:17.6012485Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:04:17.6071508Z Entering 'third_party/ios-cmake'
2022-05-18T04:04:17.6117328Z Entering 'third_party/kineto'
2022-05-18T04:04:17.6164837Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:04:17.6211928Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:04:17.6262944Z Entering 'third_party/nccl/nccl'
2022-05-18T04:04:17.6311500Z Entering 'third_party/neon2sse'
2022-05-18T04:04:17.6356896Z Entering 'third_party/onnx'
2022-05-18T04:04:17.6418033Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:04:17.6468269Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:04:17.6520395Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:04:17.6568393Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:04:17.6619765Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:04:17.6667829Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:04:17.6717013Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:04:17.6769821Z Entering 'third_party/pocketfft'
2022-05-18T04:04:17.6815322Z Entering 'third_party/protobuf'
2022-05-18T04:04:17.6869839Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:04:17.6917347Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:04:17.6968059Z Entering 'third_party/psimd'
2022-05-18T04:04:17.7014088Z Entering 'third_party/pthreadpool'
2022-05-18T04:04:17.7065129Z Entering 'third_party/pybind11'
2022-05-18T04:04:17.7115381Z Entering 'third_party/python-enum'
2022-05-18T04:04:17.7163538Z Entering 'third_party/python-peachpy'
2022-05-18T04:04:17.7212279Z Entering 'third_party/python-six'
2022-05-18T04:04:17.7259438Z Entering 'third_party/sleef'
2022-05-18T04:04:17.7306103Z Entering 'third_party/tbb'
2022-05-18T04:04:17.7357308Z Entering 'third_party/tensorpipe'
2022-05-18T04:04:17.7404109Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:04:17.7449774Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:04:17.7496195Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:04:17.7545150Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:04:17.7593802Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:04:17.7644205Z Entering 'third_party/zstd'
2022-05-18T04:04:17.7712444Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-05-18T04:04:17.8078802Z Entering 'android/libs/fbjni'
2022-05-18T04:04:17.8121621Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-05-18T04:04:17.8141393Z Entering 'third_party/FP16'
2022-05-18T04:04:17.8184928Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-05-18T04:04:17.8204364Z Entering 'third_party/FXdiv'
2022-05-18T04:04:17.8247480Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-05-18T04:04:17.8267527Z Entering 'third_party/NNPACK'
2022-05-18T04:04:17.8313280Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-05-18T04:04:17.8336173Z Entering 'third_party/QNNPACK'
2022-05-18T04:04:17.8379643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-05-18T04:04:17.8400904Z Entering 'third_party/XNNPACK'
2022-05-18T04:04:17.8444549Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-05-18T04:04:17.8476458Z Entering 'third_party/benchmark'
2022-05-18T04:04:17.8521287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T04:04:17.8540399Z Entering 'third_party/cpuinfo'
2022-05-18T04:04:17.8583303Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-05-18T04:04:17.8603403Z Entering 'third_party/cub'
2022-05-18T04:04:17.8649158Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-05-18T04:04:17.8669200Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:04:17.8712858Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-05-18T04:04:17.8739695Z Entering 'third_party/eigen'
2022-05-18T04:04:17.8784531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-05-18T04:04:17.8805229Z Entering 'third_party/fbgemm'
2022-05-18T04:04:17.8849454Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-05-18T04:04:17.8869758Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:04:17.8913318Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-05-18T04:04:17.8936382Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:04:17.8980061Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-05-18T04:04:17.9002057Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:04:17.9044609Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-05-18T04:04:17.9066261Z Entering 'third_party/flatbuffers'
2022-05-18T04:04:17.9109265Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-05-18T04:04:17.9130507Z Entering 'third_party/fmt'
2022-05-18T04:04:17.9173214Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-05-18T04:04:17.9194121Z Entering 'third_party/foxi'
2022-05-18T04:04:17.9236344Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-05-18T04:04:17.9256624Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:04:17.9301465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-05-18T04:04:17.9320944Z Entering 'third_party/gloo'
2022-05-18T04:04:17.9366325Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-05-18T04:04:17.9386367Z Entering 'third_party/googletest'
2022-05-18T04:04:17.9431828Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-05-18T04:04:17.9455266Z Entering 'third_party/ideep'
2022-05-18T04:04:17.9501055Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-05-18T04:04:17.9521075Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:04:17.9565135Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-05-18T04:04:17.9586741Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:04:17.9631571Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-05-18T04:04:17.9660689Z Entering 'third_party/ios-cmake'
2022-05-18T04:04:17.9704811Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-05-18T04:04:17.9724139Z Entering 'third_party/kineto'
2022-05-18T04:04:17.9767776Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-05-18T04:04:17.9787887Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:04:17.9831343Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-05-18T04:04:17.9850184Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:04:17.9895954Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-05-18T04:04:17.9917851Z Entering 'third_party/nccl/nccl'
2022-05-18T04:04:17.9962657Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-05-18T04:04:17.9981506Z Entering 'third_party/neon2sse'
2022-05-18T04:04:18.0028408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-05-18T04:04:18.0049062Z Entering 'third_party/onnx'
2022-05-18T04:04:18.0092842Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-05-18T04:04:18.0125743Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:04:18.0169805Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T04:04:18.0188922Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:04:18.0235028Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T04:04:18.0256992Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:04:18.0302152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-05-18T04:04:18.0321919Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:04:18.0368068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-05-18T04:04:18.0392533Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:04:18.0438970Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T04:04:18.0458465Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:04:18.0504358Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T04:04:18.0522785Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:04:18.0566294Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-05-18T04:04:18.0592095Z Entering 'third_party/pocketfft'
2022-05-18T04:04:18.0637743Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-05-18T04:04:18.0659020Z Entering 'third_party/protobuf'
2022-05-18T04:04:18.0704584Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-05-18T04:04:18.0727776Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:04:18.0770423Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T04:04:18.0791252Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:04:18.0835840Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-05-18T04:04:18.0858916Z Entering 'third_party/psimd'
2022-05-18T04:04:18.0903893Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-05-18T04:04:18.0922933Z Entering 'third_party/pthreadpool'
2022-05-18T04:04:18.0968401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-05-18T04:04:18.0988067Z Entering 'third_party/pybind11'
2022-05-18T04:04:18.1031242Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T04:04:18.1052079Z Entering 'third_party/python-enum'
2022-05-18T04:04:18.1097969Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-05-18T04:04:18.1117684Z Entering 'third_party/python-peachpy'
2022-05-18T04:04:18.1159473Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-05-18T04:04:18.1179215Z Entering 'third_party/python-six'
2022-05-18T04:04:18.1223403Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-05-18T04:04:18.1243462Z Entering 'third_party/sleef'
2022-05-18T04:04:18.1287207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-05-18T04:04:18.1306808Z Entering 'third_party/tbb'
2022-05-18T04:04:18.1351564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-05-18T04:04:18.1373989Z Entering 'third_party/tensorpipe'
2022-05-18T04:04:18.1419515Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-05-18T04:04:18.1439756Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:04:18.1480997Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-05-18T04:04:18.1501017Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:04:18.1546278Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-05-18T04:04:18.1566689Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:04:18.1610901Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-05-18T04:04:18.1631214Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:04:18.1677113Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T04:04:18.1696408Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:04:18.1742133Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-05-18T04:04:18.1764738Z Entering 'third_party/zstd'
2022-05-18T04:04:18.1809100Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-05-18T04:04:18.2879114Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-05-18T04:04:18.3252878Z Entering 'android/libs/fbjni'
2022-05-18T04:04:18.3305606Z Entering 'third_party/FP16'
2022-05-18T04:04:18.3355710Z Entering 'third_party/FXdiv'
2022-05-18T04:04:18.3405867Z Entering 'third_party/NNPACK'
2022-05-18T04:04:18.3456714Z Entering 'third_party/QNNPACK'
2022-05-18T04:04:18.3507854Z Entering 'third_party/XNNPACK'
2022-05-18T04:04:18.3568574Z Entering 'third_party/benchmark'
2022-05-18T04:04:18.3620935Z Entering 'third_party/cpuinfo'
2022-05-18T04:04:18.3671717Z Entering 'third_party/cub'
2022-05-18T04:04:18.3721834Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:04:18.3776522Z Entering 'third_party/eigen'
2022-05-18T04:04:18.3833045Z Entering 'third_party/fbgemm'
2022-05-18T04:04:18.3882304Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:04:18.3931081Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:04:18.3980586Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:04:18.4030123Z Entering 'third_party/flatbuffers'
2022-05-18T04:04:18.4080200Z Entering 'third_party/fmt'
2022-05-18T04:04:18.4128978Z Entering 'third_party/foxi'
2022-05-18T04:04:18.4174957Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:04:18.4226246Z Entering 'third_party/gloo'
2022-05-18T04:04:18.4275257Z Entering 'third_party/googletest'
2022-05-18T04:04:18.4325925Z Entering 'third_party/ideep'
2022-05-18T04:04:18.4372409Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:04:18.4423737Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:04:18.4480883Z Entering 'third_party/ios-cmake'
2022-05-18T04:04:18.4530609Z Entering 'third_party/kineto'
2022-05-18T04:04:18.4580456Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:04:18.4632362Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:04:18.4684037Z Entering 'third_party/nccl/nccl'
2022-05-18T04:04:18.4732648Z Entering 'third_party/neon2sse'
2022-05-18T04:04:18.4781460Z Entering 'third_party/onnx'
2022-05-18T04:04:18.4843864Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:04:18.4893731Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:04:18.4943785Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:04:18.4992797Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:04:18.5049461Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:04:18.5097233Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:04:18.5147640Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:04:18.5207147Z Entering 'third_party/pocketfft'
2022-05-18T04:04:18.5256136Z Entering 'third_party/protobuf'
2022-05-18T04:04:18.5312806Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:04:18.5365020Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:04:18.5415315Z Entering 'third_party/psimd'
2022-05-18T04:04:18.5464518Z Entering 'third_party/pthreadpool'
2022-05-18T04:04:18.5516666Z Entering 'third_party/pybind11'
2022-05-18T04:04:18.5564309Z Entering 'third_party/python-enum'
2022-05-18T04:04:18.5612230Z Entering 'third_party/python-peachpy'
2022-05-18T04:04:18.5662205Z Entering 'third_party/python-six'
2022-05-18T04:04:18.5713325Z Entering 'third_party/sleef'
2022-05-18T04:04:18.5759194Z Entering 'third_party/tbb'
2022-05-18T04:04:18.5813423Z Entering 'third_party/tensorpipe'
2022-05-18T04:04:18.5865626Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:04:18.5915457Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:04:18.5963089Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:04:18.6011500Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:04:18.6058495Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:04:18.6111638Z Entering 'third_party/zstd'
2022-05-18T04:04:18.6177226Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-05-18T04:04:18.6547274Z Entering 'android/libs/fbjni'
2022-05-18T04:04:18.6594171Z Entering 'third_party/FP16'
2022-05-18T04:04:18.6642342Z Entering 'third_party/FXdiv'
2022-05-18T04:04:18.6693199Z Entering 'third_party/NNPACK'
2022-05-18T04:04:18.6745699Z Entering 'third_party/QNNPACK'
2022-05-18T04:04:18.6795661Z Entering 'third_party/XNNPACK'
2022-05-18T04:04:18.6856857Z Entering 'third_party/benchmark'
2022-05-18T04:04:18.6910793Z Entering 'third_party/cpuinfo'
2022-05-18T04:04:18.6961140Z Entering 'third_party/cub'
2022-05-18T04:04:18.7011646Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:04:18.7070498Z Entering 'third_party/eigen'
2022-05-18T04:04:18.7120047Z Entering 'third_party/fbgemm'
2022-05-18T04:04:18.7169141Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:04:18.7213789Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:04:18.7263631Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:04:18.7315189Z Entering 'third_party/flatbuffers'
2022-05-18T04:04:18.7367039Z Entering 'third_party/fmt'
2022-05-18T04:04:18.7414761Z Entering 'third_party/foxi'
2022-05-18T04:04:18.7464313Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:04:18.7514432Z Entering 'third_party/gloo'
2022-05-18T04:04:18.7562988Z Entering 'third_party/googletest'
2022-05-18T04:04:18.7613989Z Entering 'third_party/ideep'
2022-05-18T04:04:18.7660412Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:04:18.7710504Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:04:18.7769998Z Entering 'third_party/ios-cmake'
2022-05-18T04:04:18.7820570Z Entering 'third_party/kineto'
2022-05-18T04:04:18.7874241Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:04:18.7921721Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:04:18.7970861Z Entering 'third_party/nccl/nccl'
2022-05-18T04:04:18.8020762Z Entering 'third_party/neon2sse'
2022-05-18T04:04:18.8071946Z Entering 'third_party/onnx'
2022-05-18T04:04:18.8132756Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:04:18.8184029Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:04:18.8236192Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:04:18.8282717Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:04:18.8335626Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:04:18.8386309Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:04:18.8438731Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:04:18.8497217Z Entering 'third_party/pocketfft'
2022-05-18T04:04:18.8548388Z Entering 'third_party/protobuf'
2022-05-18T04:04:18.8601059Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:04:18.8648097Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:04:18.8701744Z Entering 'third_party/psimd'
2022-05-18T04:04:18.8753833Z Entering 'third_party/pthreadpool'
2022-05-18T04:04:18.8801168Z Entering 'third_party/pybind11'
2022-05-18T04:04:18.8849862Z Entering 'third_party/python-enum'
2022-05-18T04:04:18.8894946Z Entering 'third_party/python-peachpy'
2022-05-18T04:04:18.8945642Z Entering 'third_party/python-six'
2022-05-18T04:04:18.8997776Z Entering 'third_party/sleef'
2022-05-18T04:04:18.9047653Z Entering 'third_party/tbb'
2022-05-18T04:04:18.9098070Z Entering 'third_party/tensorpipe'
2022-05-18T04:04:18.9148278Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:04:18.9197084Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:04:18.9245470Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:04:18.9294025Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:04:18.9343375Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:04:18.9396527Z Entering 'third_party/zstd'
2022-05-18T04:04:18.9460489Z ##[endgroup]
2022-05-18T04:04:18.9514075Z [command]/usr/bin/git log -1 --format='%H'
2022-05-18T04:04:18.9551661Z '3b2375291aab7b48442f2e6fb1ef66cebc761e24'
2022-05-18T04:04:18.9722932Z Prepare all required actions
2022-05-18T04:04:18.9753947Z ##[group]Run ./.github/actions/setup-linux
2022-05-18T04:04:18.9754220Z env:
2022-05-18T04:04:18.9754441Z   IN_CI: 1
2022-05-18T04:04:18.9754810Z   IS_GHA: 1
2022-05-18T04:04:18.9755057Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:04:18.9755316Z ##[endgroup]
2022-05-18T04:04:18.9773124Z ##[group]Run set -euo pipefail
2022-05-18T04:04:18.9773442Z set -euo pipefail
2022-05-18T04:04:18.9773734Z function get_ec2_metadata() {
2022-05-18T04:04:18.9774052Z   # Pulled from instance metadata endpoint for EC2
2022-05-18T04:04:18.9774533Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-05-18T04:04:18.9774950Z   category=$1
2022-05-18T04:04:18.9775285Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-05-18T04:04:18.9775733Z }
2022-05-18T04:04:18.9776037Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-05-18T04:04:18.9776361Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-05-18T04:04:18.9776724Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-05-18T04:04:18.9777055Z echo "system info $(uname -a)"
2022-05-18T04:04:18.9791427Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:04:18.9791693Z env:
2022-05-18T04:04:18.9791903Z   IN_CI: 1
2022-05-18T04:04:18.9792301Z   IS_GHA: 1
2022-05-18T04:04:18.9792533Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:04:18.9792791Z ##[endgroup]
2022-05-18T04:04:18.9910165Z ami-id: ami-096198a0bccc6bad4
2022-05-18T04:04:18.9982678Z instance-id: i-0d4a316768328dd7a
2022-05-18T04:04:19.0054627Z instance-type: g3.16xlarge
2022-05-18T04:04:19.0064785Z system info Linux ip-10-0-4-136.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-05-18T04:04:19.0085401Z ##[group]Run if systemctl is-active --quiet docker; then
2022-05-18T04:04:19.0085900Z if systemctl is-active --quiet docker; then
2022-05-18T04:04:19.0086330Z     echo "Docker daemon is running...";
2022-05-18T04:04:19.0086607Z else
2022-05-18T04:04:19.0086971Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-05-18T04:04:19.0087320Z fi
2022-05-18T04:04:19.0099108Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:04:19.0099450Z env:
2022-05-18T04:04:19.0099714Z   IN_CI: 1
2022-05-18T04:04:19.0099970Z   IS_GHA: 1
2022-05-18T04:04:19.0100330Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:04:19.0100622Z ##[endgroup]
2022-05-18T04:04:19.0160004Z Docker daemon is running...
2022-05-18T04:04:19.0184117Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-05-18T04:04:19.0184698Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-05-18T04:04:19.0185120Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-05-18T04:04:19.0185925Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-05-18T04:04:19.0186576Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-05-18T04:04:19.0199842Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:04:19.0200155Z env:
2022-05-18T04:04:19.0200509Z   IN_CI: 1
2022-05-18T04:04:19.0200797Z   IS_GHA: 1
2022-05-18T04:04:19.0201109Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:04:19.0201589Z   AWS_RETRY_MODE: standard
2022-05-18T04:04:19.0201939Z   AWS_MAX_ATTEMPTS: 5
2022-05-18T04:04:19.0202219Z   AWS_DEFAULT_REGION: us-east-1
2022-05-18T04:04:19.0202728Z ##[endgroup]
2022-05-18T04:04:20.0192042Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-05-18T04:04:20.0192541Z Configure a credential helper to remove this warning. See
2022-05-18T04:04:20.0193304Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-05-18T04:04:20.0193608Z 
2022-05-18T04:04:20.0195355Z Login Succeeded
2022-05-18T04:04:20.0284653Z ##[group]Run env | grep '^GITHUB' > "/tmp/github_env_${GITHUB_RUN_ID}"
2022-05-18T04:04:20.0285063Z env | grep '^GITHUB' > "/tmp/github_env_${GITHUB_RUN_ID}"
2022-05-18T04:04:20.0299436Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:04:20.0299721Z env:
2022-05-18T04:04:20.0299955Z   IN_CI: 1
2022-05-18T04:04:20.0300195Z   IS_GHA: 1
2022-05-18T04:04:20.0300431Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:04:20.0300703Z ##[endgroup]
2022-05-18T04:04:20.0370684Z Prepare all required actions
2022-05-18T04:04:20.0371059Z Getting action download info
2022-05-18T04:04:20.1697451Z Download action repository 'seemethere/add-github-ssh-key@v1' (SHA:1ecffedb1e192a50aa67dba2f0e048e5d3bfa144)
2022-05-18T04:04:20.2904923Z ##[group]Run ./.github/actions/setup-ssh
2022-05-18T04:04:20.2905217Z with:
2022-05-18T04:04:20.2905630Z   github-secret: ***
2022-05-18T04:04:20.2905882Z env:
2022-05-18T04:04:20.2906113Z   IN_CI: 1
2022-05-18T04:04:20.2906326Z   IS_GHA: 1
2022-05-18T04:04:20.2906586Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:04:20.2906860Z ##[endgroup]
2022-05-18T04:04:20.2931539Z ##[group]Run seemethere/add-github-ssh-key@v1
2022-05-18T04:04:20.2931836Z with:
2022-05-18T04:04:20.2932212Z   GITHUB_TOKEN: ***
2022-05-18T04:04:20.2932489Z   activate-with-label: false
2022-05-18T04:04:20.2932741Z   label: with-ssh
2022-05-18T04:04:20.2933012Z   remove-existing-keys: true
2022-05-18T04:04:20.2933269Z env:
2022-05-18T04:04:20.2933467Z   IN_CI: 1
2022-05-18T04:04:20.2933739Z   IS_GHA: 1
2022-05-18T04:04:20.2933973Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:04:20.2934245Z ##[endgroup]
2022-05-18T04:04:20.3663913Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2022-05-18T04:04:20.3714726Z Prepare all required actions
2022-05-18T04:04:20.3736582Z ##[group]Run ./.github/actions/pull-docker-image
2022-05-18T04:04:20.3736887Z with:
2022-05-18T04:04:20.3737390Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:04:20.3737857Z env:
2022-05-18T04:04:20.3738082Z   IN_CI: 1
2022-05-18T04:04:20.3738316Z   IS_GHA: 1
2022-05-18T04:04:20.3738553Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:04:20.3738978Z ##[endgroup]
2022-05-18T04:04:20.3755637Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-05-18T04:04:20.3756056Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-05-18T04:04:20.3756396Z retry docker pull "${DOCKER_IMAGE}"
2022-05-18T04:04:20.3769894Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:04:20.3770197Z env:
2022-05-18T04:04:20.3770422Z   IN_CI: 1
2022-05-18T04:04:20.3770674Z   IS_GHA: 1
2022-05-18T04:04:20.3770935Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:04:20.3771619Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:04:20.3772117Z ##[endgroup]
2022-05-18T04:04:20.6384826Z 6deab82db6a72ca54cd3e3322ee4f13864536734: Pulling from pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7
2022-05-18T04:04:20.6385314Z 11323ed2c653: Pulling fs layer
2022-05-18T04:04:20.6385593Z 9b0c32b3202c: Pulling fs layer
2022-05-18T04:04:20.6385864Z 55d4aa3df964: Pulling fs layer
2022-05-18T04:04:20.6386145Z ced0e45f533f: Pulling fs layer
2022-05-18T04:04:20.6386436Z a6d5f855f26c: Pulling fs layer
2022-05-18T04:04:20.6386702Z 532188ad0a5d: Pulling fs layer
2022-05-18T04:04:20.6387002Z 53b0132b34a2: Pulling fs layer
2022-05-18T04:04:20.6387288Z d63f711e9949: Pulling fs layer
2022-05-18T04:04:20.6387555Z 776e7a7e28b2: Pulling fs layer
2022-05-18T04:04:20.6387816Z 69004237646f: Pulling fs layer
2022-05-18T04:04:20.6388077Z a0a6f96a62d8: Pulling fs layer
2022-05-18T04:04:20.6388343Z 7918ac79e586: Pulling fs layer
2022-05-18T04:04:20.6388596Z 517f3f32e512: Pulling fs layer
2022-05-18T04:04:20.6388864Z 7c88fb71bf11: Pulling fs layer
2022-05-18T04:04:20.6389120Z a6d5f855f26c: Waiting
2022-05-18T04:04:20.6389360Z 7b920d7a1988: Pulling fs layer
2022-05-18T04:04:20.6389624Z 0ba8a6800faf: Pulling fs layer
2022-05-18T04:04:20.6389879Z 532188ad0a5d: Waiting
2022-05-18T04:04:20.6390118Z 6d58a87851d7: Pulling fs layer
2022-05-18T04:04:20.6390379Z b06b299e7454: Pulling fs layer
2022-05-18T04:04:20.6390642Z b046a45d4ca8: Pulling fs layer
2022-05-18T04:04:20.6390946Z acf3886a01ad: Pulling fs layer
2022-05-18T04:04:20.6391291Z 166228572fc8: Pulling fs layer
2022-05-18T04:04:20.6391553Z 6d680b004bdb: Pulling fs layer
2022-05-18T04:04:20.6391806Z 4d9d54d04be5: Pulling fs layer
2022-05-18T04:04:20.6392071Z 55e19101ee96: Pulling fs layer
2022-05-18T04:04:20.6392325Z ced0e45f533f: Waiting
2022-05-18T04:04:20.6392565Z d57378452c6c: Pulling fs layer
2022-05-18T04:04:20.6392823Z 4097195e70a4: Pulling fs layer
2022-05-18T04:04:20.6393081Z e90775d597ae: Pulling fs layer
2022-05-18T04:04:20.6393328Z d63f711e9949: Waiting
2022-05-18T04:04:20.6393548Z 7c88fb71bf11: Waiting
2022-05-18T04:04:20.6393799Z 342cb5b8793f: Pulling fs layer
2022-05-18T04:04:20.6394045Z 7918ac79e586: Waiting
2022-05-18T04:04:20.6394277Z ec9f4694245d: Pulling fs layer
2022-05-18T04:04:20.6394523Z 7b920d7a1988: Waiting
2022-05-18T04:04:20.6394768Z 5ff41a564c23: Pulling fs layer
2022-05-18T04:04:20.6395001Z 776e7a7e28b2: Waiting
2022-05-18T04:04:20.6395312Z acf3886a01ad: Waiting
2022-05-18T04:04:20.6395563Z 5e9e1c5c2b02: Pulling fs layer
2022-05-18T04:04:20.6395797Z a0a6f96a62d8: Waiting
2022-05-18T04:04:20.6396027Z 69004237646f: Waiting
2022-05-18T04:04:20.6396256Z 0ba8a6800faf: Waiting
2022-05-18T04:04:20.6396476Z 166228572fc8: Waiting
2022-05-18T04:04:20.6396723Z 85cae8860e8b: Pulling fs layer
2022-05-18T04:04:20.6397166Z 7bd074c80c3f: Pulling fs layer
2022-05-18T04:04:20.6397420Z 4d9d54d04be5: Waiting
2022-05-18T04:04:20.6397668Z 7ebce38575d6: Pulling fs layer
2022-05-18T04:04:20.6397916Z b06b299e7454: Waiting
2022-05-18T04:04:20.6398133Z 342cb5b8793f: Waiting
2022-05-18T04:04:20.6398364Z 5ff41a564c23: Waiting
2022-05-18T04:04:20.6398615Z 3dcf0fc78ba8: Pulling fs layer
2022-05-18T04:04:20.6398849Z 517f3f32e512: Waiting
2022-05-18T04:04:20.6399097Z de93ffc12e40: Pulling fs layer
2022-05-18T04:04:20.6399344Z 4097195e70a4: Waiting
2022-05-18T04:04:20.6399573Z e90775d597ae: Waiting
2022-05-18T04:04:20.6399788Z 6d680b004bdb: Waiting
2022-05-18T04:04:20.6400018Z ec9f4694245d: Waiting
2022-05-18T04:04:20.6400245Z d57378452c6c: Waiting
2022-05-18T04:04:20.6400456Z 5e9e1c5c2b02: Waiting
2022-05-18T04:04:20.6400706Z fd0f553736b3: Pulling fs layer
2022-05-18T04:04:20.6400971Z 6b52bc4fc524: Pulling fs layer
2022-05-18T04:04:20.6401325Z f709baccd3f5: Pulling fs layer
2022-05-18T04:04:20.6401592Z 25dff8b9a054: Pulling fs layer
2022-05-18T04:04:20.6401866Z bcd88fe424d2: Pulling fs layer
2022-05-18T04:04:20.6402106Z de93ffc12e40: Waiting
2022-05-18T04:04:20.6402345Z 6b52bc4fc524: Waiting
2022-05-18T04:04:20.6402580Z f709baccd3f5: Waiting
2022-05-18T04:04:20.6402808Z 8710652e57c7: Pulling fs layer
2022-05-18T04:04:20.6403067Z 050758b5b900: Pulling fs layer
2022-05-18T04:04:20.6403332Z e104e8ddd08b: Pulling fs layer
2022-05-18T04:04:20.6403572Z bcd88fe424d2: Waiting
2022-05-18T04:04:20.6403808Z fd0f553736b3: Waiting
2022-05-18T04:04:20.6404031Z 8710652e57c7: Waiting
2022-05-18T04:04:20.6404256Z b0c972c96382: Pulling fs layer
2022-05-18T04:04:20.6404498Z 050758b5b900: Waiting
2022-05-18T04:04:20.6404776Z e104e8ddd08b: Waiting
2022-05-18T04:04:20.6405006Z 053d59c76970: Pulling fs layer
2022-05-18T04:04:20.6405267Z 30dcacd2ffe2: Pulling fs layer
2022-05-18T04:04:20.6405534Z 1c1fd12e267d: Pulling fs layer
2022-05-18T04:04:20.6405769Z 053d59c76970: Waiting
2022-05-18T04:04:20.6406002Z 30dcacd2ffe2: Waiting
2022-05-18T04:04:20.6406239Z b0c972c96382: Waiting
2022-05-18T04:04:20.6406452Z 1c1fd12e267d: Waiting
2022-05-18T04:04:20.7904202Z 9b0c32b3202c: Download complete
2022-05-18T04:04:20.7961135Z 55d4aa3df964: Verifying Checksum
2022-05-18T04:04:20.7961439Z 55d4aa3df964: Download complete
2022-05-18T04:04:20.8775413Z ced0e45f533f: Verifying Checksum
2022-05-18T04:04:20.8775726Z ced0e45f533f: Download complete
2022-05-18T04:04:20.8798039Z a6d5f855f26c: Download complete
2022-05-18T04:04:20.9697215Z 11323ed2c653: Download complete
2022-05-18T04:04:20.9705521Z 53b0132b34a2: Download complete
2022-05-18T04:04:21.2025167Z 776e7a7e28b2: Verifying Checksum
2022-05-18T04:04:21.2025490Z 776e7a7e28b2: Download complete
2022-05-18T04:04:21.7629694Z 11323ed2c653: Pull complete
2022-05-18T04:04:22.0619777Z 9b0c32b3202c: Pull complete
2022-05-18T04:04:22.3697716Z 55d4aa3df964: Pull complete
2022-05-18T04:04:22.4767629Z ced0e45f533f: Pull complete
2022-05-18T04:04:22.6002554Z a6d5f855f26c: Pull complete
2022-05-18T04:04:26.8124296Z 69004237646f: Verifying Checksum
2022-05-18T04:04:26.8124700Z 69004237646f: Download complete
2022-05-18T04:04:26.8925720Z a0a6f96a62d8: Verifying Checksum
2022-05-18T04:04:26.8926064Z a0a6f96a62d8: Download complete
2022-05-18T04:04:28.5293608Z 532188ad0a5d: Verifying Checksum
2022-05-18T04:04:28.5294054Z 532188ad0a5d: Download complete
2022-05-18T04:04:28.6073362Z 517f3f32e512: Download complete
2022-05-18T04:04:28.6869920Z 7c88fb71bf11: Download complete
2022-05-18T04:04:28.7671822Z 7918ac79e586: Verifying Checksum
2022-05-18T04:04:28.7672170Z 7918ac79e586: Download complete
2022-05-18T04:04:28.7701477Z 7b920d7a1988: Verifying Checksum
2022-05-18T04:04:28.7701783Z 7b920d7a1988: Download complete
2022-05-18T04:04:28.8482691Z 6d58a87851d7: Verifying Checksum
2022-05-18T04:04:28.8482997Z 6d58a87851d7: Download complete
2022-05-18T04:04:28.8537258Z 0ba8a6800faf: Download complete
2022-05-18T04:04:28.9344712Z b06b299e7454: Verifying Checksum
2022-05-18T04:04:28.9345031Z b06b299e7454: Download complete
2022-05-18T04:04:29.0082928Z acf3886a01ad: Verifying Checksum
2022-05-18T04:04:29.0083302Z acf3886a01ad: Download complete
2022-05-18T04:04:29.0851607Z 166228572fc8: Download complete
2022-05-18T04:04:29.1760558Z 6d680b004bdb: Verifying Checksum
2022-05-18T04:04:29.1760905Z 6d680b004bdb: Download complete
2022-05-18T04:04:29.2466541Z 4d9d54d04be5: Download complete
2022-05-18T04:04:29.3394200Z 55e19101ee96: Verifying Checksum
2022-05-18T04:04:29.3394562Z 55e19101ee96: Download complete
2022-05-18T04:04:29.4376465Z d57378452c6c: Verifying Checksum
2022-05-18T04:04:29.4376799Z d57378452c6c: Download complete
2022-05-18T04:04:29.8503791Z d63f711e9949: Verifying Checksum
2022-05-18T04:04:29.8504179Z d63f711e9949: Download complete
2022-05-18T04:04:29.9335964Z e90775d597ae: Verifying Checksum
2022-05-18T04:04:29.9336345Z e90775d597ae: Download complete
2022-05-18T04:04:30.0315845Z 342cb5b8793f: Download complete
2022-05-18T04:04:30.1460778Z ec9f4694245d: Verifying Checksum
2022-05-18T04:04:30.1461149Z ec9f4694245d: Download complete
2022-05-18T04:04:30.2435396Z 5ff41a564c23: Download complete
2022-05-18T04:04:30.3247432Z 5e9e1c5c2b02: Verifying Checksum
2022-05-18T04:04:30.3247831Z 5e9e1c5c2b02: Download complete
2022-05-18T04:04:30.3836138Z 4097195e70a4: Verifying Checksum
2022-05-18T04:04:30.3836553Z 4097195e70a4: Download complete
2022-05-18T04:04:30.4497752Z 7bd074c80c3f: Verifying Checksum
2022-05-18T04:04:30.4498124Z 7bd074c80c3f: Download complete
2022-05-18T04:04:30.5334832Z 7ebce38575d6: Verifying Checksum
2022-05-18T04:04:30.5335195Z 7ebce38575d6: Download complete
2022-05-18T04:04:30.7919518Z 3dcf0fc78ba8: Verifying Checksum
2022-05-18T04:04:30.7919909Z 3dcf0fc78ba8: Download complete
2022-05-18T04:04:30.8764573Z de93ffc12e40: Download complete
2022-05-18T04:04:30.9603873Z fd0f553736b3: Verifying Checksum
2022-05-18T04:04:30.9604247Z fd0f553736b3: Download complete
2022-05-18T04:04:31.0459181Z 6b52bc4fc524: Download complete
2022-05-18T04:04:32.3895732Z 85cae8860e8b: Verifying Checksum
2022-05-18T04:04:32.3896124Z 85cae8860e8b: Download complete
2022-05-18T04:04:32.4690038Z 25dff8b9a054: Download complete
2022-05-18T04:04:32.5586788Z bcd88fe424d2: Verifying Checksum
2022-05-18T04:04:32.5587177Z bcd88fe424d2: Download complete
2022-05-18T04:04:32.8512144Z 8710652e57c7: Verifying Checksum
2022-05-18T04:04:32.8512535Z 8710652e57c7: Download complete
2022-05-18T04:04:32.9273282Z 050758b5b900: Verifying Checksum
2022-05-18T04:04:32.9273617Z 050758b5b900: Download complete
2022-05-18T04:04:33.1299232Z e104e8ddd08b: Download complete
2022-05-18T04:04:33.1972442Z b0c972c96382: Download complete
2022-05-18T04:04:33.7842072Z 053d59c76970: Verifying Checksum
2022-05-18T04:04:33.7842457Z 053d59c76970: Download complete
2022-05-18T04:04:34.0663852Z 30dcacd2ffe2: Download complete
2022-05-18T04:04:34.1317036Z 1c1fd12e267d: Verifying Checksum
2022-05-18T04:04:34.1317368Z 1c1fd12e267d: Download complete
2022-05-18T04:04:34.5048993Z f709baccd3f5: Verifying Checksum
2022-05-18T04:04:34.5049393Z f709baccd3f5: Download complete
2022-05-18T04:04:37.3479924Z 532188ad0a5d: Pull complete
2022-05-18T04:04:37.4612322Z 53b0132b34a2: Pull complete
2022-05-18T04:04:49.3144855Z d63f711e9949: Pull complete
2022-05-18T04:04:49.4368431Z 776e7a7e28b2: Pull complete
2022-05-18T04:04:49.8915693Z b046a45d4ca8: Verifying Checksum
2022-05-18T04:04:49.8916048Z b046a45d4ca8: Download complete
2022-05-18T04:04:55.8948359Z 69004237646f: Pull complete
2022-05-18T04:04:56.0155420Z a0a6f96a62d8: Pull complete
2022-05-18T04:05:00.3531481Z 7918ac79e586: Pull complete
2022-05-18T04:05:00.4601766Z 517f3f32e512: Pull complete
2022-05-18T04:05:01.2102392Z 7c88fb71bf11: Pull complete
2022-05-18T04:05:02.7041435Z 7b920d7a1988: Pull complete
2022-05-18T04:05:04.5518917Z 0ba8a6800faf: Pull complete
2022-05-18T04:05:06.1748582Z 6d58a87851d7: Pull complete
2022-05-18T04:05:08.0819418Z b06b299e7454: Pull complete
2022-05-18T04:05:41.4476833Z b046a45d4ca8: Pull complete
2022-05-18T04:05:43.2933448Z acf3886a01ad: Pull complete
2022-05-18T04:05:45.1657282Z 166228572fc8: Pull complete
2022-05-18T04:05:47.1366209Z 6d680b004bdb: Pull complete
2022-05-18T04:05:48.9968392Z 4d9d54d04be5: Pull complete
2022-05-18T04:05:50.7997414Z 55e19101ee96: Pull complete
2022-05-18T04:05:52.6175488Z d57378452c6c: Pull complete
2022-05-18T04:05:56.6499444Z 4097195e70a4: Pull complete
2022-05-18T04:05:58.5237331Z e90775d597ae: Pull complete
2022-05-18T04:06:00.3869083Z 342cb5b8793f: Pull complete
2022-05-18T04:06:00.5338041Z ec9f4694245d: Pull complete
2022-05-18T04:06:00.6559880Z 5ff41a564c23: Pull complete
2022-05-18T04:06:00.7524183Z 5e9e1c5c2b02: Pull complete
2022-05-18T04:06:07.6627345Z 85cae8860e8b: Pull complete
2022-05-18T04:06:09.1808436Z 7bd074c80c3f: Pull complete
2022-05-18T04:06:11.7960526Z 7ebce38575d6: Pull complete
2022-05-18T04:06:14.0618477Z 3dcf0fc78ba8: Pull complete
2022-05-18T04:06:16.4556468Z de93ffc12e40: Pull complete
2022-05-18T04:06:19.3703240Z fd0f553736b3: Pull complete
2022-05-18T04:06:21.4065501Z 6b52bc4fc524: Pull complete
2022-05-18T04:06:27.9702868Z f709baccd3f5: Pull complete
2022-05-18T04:06:28.0840483Z 25dff8b9a054: Pull complete
2022-05-18T04:06:28.2232480Z bcd88fe424d2: Pull complete
2022-05-18T04:06:28.3454800Z 8710652e57c7: Pull complete
2022-05-18T04:06:28.4490651Z 050758b5b900: Pull complete
2022-05-18T04:06:29.2504926Z e104e8ddd08b: Pull complete
2022-05-18T04:06:29.3679417Z b0c972c96382: Pull complete
2022-05-18T04:06:31.1262468Z 053d59c76970: Pull complete
2022-05-18T04:06:31.2290783Z 30dcacd2ffe2: Pull complete
2022-05-18T04:06:31.3572678Z 1c1fd12e267d: Pull complete
2022-05-18T04:06:31.3687934Z Digest: sha256:9737b662edb86afcd12a9367db6178a57889543632c0b710c5058abe14dc048f
2022-05-18T04:06:31.3726377Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:06:31.3753538Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:06:31.3848902Z ##[group]Run nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a
2022-05-18T04:06:31.3849242Z with:
2022-05-18T04:06:31.3849452Z   timeout_minutes: 10
2022-05-18T04:06:31.3849694Z   max_attempts: 3
2022-05-18T04:06:31.3850069Z   command: set -ex
bash .github/scripts/install_nvidia_utils_linux.sh
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2022-05-18T04:06:31.3850424Z   retry_wait_seconds: 10
2022-05-18T04:06:31.3850685Z   polling_interval_seconds: 1
2022-05-18T04:06:31.3851117Z   warning_on_retry: true
2022-05-18T04:06:31.3851359Z   continue_on_error: false
2022-05-18T04:06:31.3851600Z env:
2022-05-18T04:06:31.3851812Z   IN_CI: 1
2022-05-18T04:06:31.3852015Z   IS_GHA: 1
2022-05-18T04:06:31.3852260Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:06:31.3852514Z ##[endgroup]
2022-05-18T04:06:31.4304447Z 
2022-05-18T04:06:31.4378608Z + bash .github/scripts/install_nvidia_utils_linux.sh
2022-05-18T04:06:31.4379447Z + sudo yum install -y yum-utils
2022-05-18T04:06:31.4382119Z == Installing nvidia container toolkit for amzn2 ==
2022-05-18T04:06:31.9235634Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:06:32.2823507Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2022-05-18T04:06:32.2823930Z Nothing to do
2022-05-18T04:06:32.3030851Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-05-18T04:06:32.7723659Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:06:32.8257947Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-05-18T04:06:32.8258641Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2022-05-18T04:06:32.8259179Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2022-05-18T04:06:32.8415653Z + sudo yum install -y nvidia-docker2
2022-05-18T04:06:33.3093822Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:06:33.3508894Z Retrieving key from https://nvidia.github.io/libnvidia-container/gpgkey
2022-05-18T04:06:33.3653087Z Importing GPG key 0xF796ECB0:
2022-05-18T04:06:33.3653798Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-05-18T04:06:33.3654219Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-05-18T04:06:33.3654747Z  From       : https://nvidia.github.io/libnvidia-container/gpgkey
2022-05-18T04:06:34.5628714Z Retrieving key from https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-05-18T04:06:34.5698988Z Importing GPG key 0xF796ECB0:
2022-05-18T04:06:34.5699756Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-05-18T04:06:34.5700198Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-05-18T04:06:34.5700734Z  From       : https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-05-18T04:06:34.8012428Z Retrieving key from https://nvidia.github.io/nvidia-docker/gpgkey
2022-05-18T04:06:34.8085453Z Importing GPG key 0xF796ECB0:
2022-05-18T04:06:34.8086276Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-05-18T04:06:34.8086746Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-05-18T04:06:34.8087252Z  From       : https://nvidia.github.io/nvidia-docker/gpgkey
2022-05-18T04:06:38.3156768Z Resolving Dependencies
2022-05-18T04:06:38.3162730Z --> Running transaction check
2022-05-18T04:06:38.3163238Z ---> Package nvidia-docker2.noarch 0:2.10.0-1 will be installed
2022-05-18T04:06:38.3190239Z --> Processing Dependency: nvidia-container-toolkit >= 1.9.0-1 for package: nvidia-docker2-2.10.0-1.noarch
2022-05-18T04:06:38.3519202Z --> Running transaction check
2022-05-18T04:06:38.3519717Z ---> Package nvidia-container-toolkit.x86_64 0:1.9.0-1 will be installed
2022-05-18T04:06:38.3529005Z --> Processing Dependency: libnvidia-container-tools < 2.0.0 for package: nvidia-container-toolkit-1.9.0-1.x86_64
2022-05-18T04:06:38.3651055Z --> Processing Dependency: libnvidia-container-tools >= 1.9.0-1 for package: nvidia-container-toolkit-1.9.0-1.x86_64
2022-05-18T04:06:38.3651621Z --> Running transaction check
2022-05-18T04:06:38.3652062Z ---> Package libnvidia-container-tools.x86_64 0:1.9.0-1 will be installed
2022-05-18T04:06:38.3681218Z --> Processing Dependency: libnvidia-container1(x86-64) >= 1.9.0-1 for package: libnvidia-container-tools-1.9.0-1.x86_64
2022-05-18T04:06:38.3711738Z --> Processing Dependency: libnvidia-container.so.1(NVC_1.0)(64bit) for package: libnvidia-container-tools-1.9.0-1.x86_64
2022-05-18T04:06:38.3712512Z --> Processing Dependency: libnvidia-container.so.1()(64bit) for package: libnvidia-container-tools-1.9.0-1.x86_64
2022-05-18T04:06:38.3712980Z --> Running transaction check
2022-05-18T04:06:38.3713418Z ---> Package libnvidia-container1.x86_64 0:1.9.0-1 will be installed
2022-05-18T04:06:38.6618810Z --> Finished Dependency Resolution
2022-05-18T04:06:38.7260901Z 
2022-05-18T04:06:38.7261677Z Dependencies Resolved
2022-05-18T04:06:38.7274486Z 
2022-05-18T04:06:38.7274944Z ================================================================================
2022-05-18T04:06:38.7275665Z  Package                     Arch     Version       Repository             Size
2022-05-18T04:06:38.7276347Z ================================================================================
2022-05-18T04:06:38.7276811Z Installing:
2022-05-18T04:06:38.7278850Z  nvidia-docker2              noarch   2.10.0-1      libnvidia-container   8.7 k
2022-05-18T04:06:38.7279545Z Installing for dependencies:
2022-05-18T04:06:38.7280479Z  libnvidia-container-tools   x86_64   1.9.0-1       libnvidia-container    48 k
2022-05-18T04:06:38.7281464Z  libnvidia-container1        x86_64   1.9.0-1       libnvidia-container   1.0 M
2022-05-18T04:06:38.7282474Z  nvidia-container-toolkit    x86_64   1.9.0-1       libnvidia-container   1.5 M
2022-05-18T04:06:38.7282989Z 
2022-05-18T04:06:38.7283210Z Transaction Summary
2022-05-18T04:06:38.7283744Z ================================================================================
2022-05-18T04:06:38.7284344Z Install  1 Package (+3 Dependent packages)
2022-05-18T04:06:38.7284714Z 
2022-05-18T04:06:38.7284950Z Total download size: 2.5 M
2022-05-18T04:06:38.7285452Z Installed size: 7.4 M
2022-05-18T04:06:38.7285936Z Downloading packages:
2022-05-18T04:06:38.8304402Z --------------------------------------------------------------------------------
2022-05-18T04:06:38.8304841Z Total                                               25 MB/s | 2.5 MB  00:00     
2022-05-18T04:06:38.8353699Z Running transaction check
2022-05-18T04:06:38.8418515Z Running transaction test
2022-05-18T04:06:38.8582634Z Transaction test succeeded
2022-05-18T04:06:38.8584415Z Running transaction
2022-05-18T04:06:39.0740364Z   Installing : libnvidia-container1-1.9.0-1.x86_64                          1/4 
2022-05-18T04:06:39.2787292Z   Installing : libnvidia-container-tools-1.9.0-1.x86_64                     2/4 
2022-05-18T04:06:39.2985761Z   Installing : nvidia-container-toolkit-1.9.0-1.x86_64                      3/4 
2022-05-18T04:06:39.3369417Z   Installing : nvidia-docker2-2.10.0-1.noarch                               4/4 
2022-05-18T04:06:39.3635908Z   Verifying  : libnvidia-container-tools-1.9.0-1.x86_64                     1/4 
2022-05-18T04:06:39.3731137Z   Verifying  : nvidia-container-toolkit-1.9.0-1.x86_64                      2/4 
2022-05-18T04:06:39.3814096Z   Verifying  : nvidia-docker2-2.10.0-1.noarch                               3/4 
2022-05-18T04:06:39.4443091Z   Verifying  : libnvidia-container1-1.9.0-1.x86_64                          4/4 
2022-05-18T04:06:39.4443589Z 
2022-05-18T04:06:39.4443772Z Installed:
2022-05-18T04:06:39.4444488Z   nvidia-docker2.noarch 0:2.10.0-1                                              
2022-05-18T04:06:39.4444881Z 
2022-05-18T04:06:39.4445106Z Dependency Installed:
2022-05-18T04:06:39.4445975Z   libnvidia-container-tools.x86_64 0:1.9.0-1                                    
2022-05-18T04:06:39.4447176Z   libnvidia-container1.x86_64 0:1.9.0-1                                         
2022-05-18T04:06:39.4448097Z   nvidia-container-toolkit.x86_64 0:1.9.0-1                                     
2022-05-18T04:06:39.4448987Z 
2022-05-18T04:06:39.4449371Z Complete!
2022-05-18T04:06:39.5380055Z + sudo systemctl restart docker
2022-05-18T04:06:47.0506648Z == Installing nvidia driver NVIDIA-Linux-x86_64-510.60.02.run ==
2022-05-18T04:06:47.0507590Z + sudo yum groupinstall -y 'Development Tools'
2022-05-18T04:06:47.5343744Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:06:48.5537296Z Resolving Dependencies
2022-05-18T04:06:48.5542168Z --> Running transaction check
2022-05-18T04:06:48.5543669Z ---> Package autoconf.noarch 0:2.69-11.amzn2 will be installed
2022-05-18T04:06:48.5782150Z --> Processing Dependency: m4 >= 1.4.14 for package: autoconf-2.69-11.amzn2.noarch
2022-05-18T04:06:48.6116578Z --> Processing Dependency: perl(Data::Dumper) for package: autoconf-2.69-11.amzn2.noarch
2022-05-18T04:06:48.6117205Z ---> Package automake.noarch 0:1.13.4-3.1.amzn2 will be installed
2022-05-18T04:06:48.6178519Z --> Processing Dependency: perl(Thread::Queue) for package: automake-1.13.4-3.1.amzn2.noarch
2022-05-18T04:06:48.6186287Z --> Processing Dependency: perl(TAP::Parser) for package: automake-1.13.4-3.1.amzn2.noarch
2022-05-18T04:06:48.6198036Z ---> Package bison.x86_64 0:3.0.4-6.amzn2.0.2 will be installed
2022-05-18T04:06:48.6303640Z ---> Package byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 will be installed
2022-05-18T04:06:48.6313495Z ---> Package cscope.x86_64 0:15.8-10.amzn2.0.2 will be installed
2022-05-18T04:06:48.6365917Z ---> Package ctags.x86_64 0:5.8-13.amzn2.0.2 will be installed
2022-05-18T04:06:48.6378765Z ---> Package diffstat.x86_64 0:1.57-4.amzn2.0.2 will be installed
2022-05-18T04:06:48.6390285Z ---> Package doxygen.x86_64 1:1.8.5-4.amzn2 will be installed
2022-05-18T04:06:48.6485135Z ---> Package elfutils.x86_64 0:0.176-2.amzn2 will be installed
2022-05-18T04:06:48.6650299Z ---> Package flex.x86_64 0:2.5.37-3.amzn2.0.3 will be installed
2022-05-18T04:06:48.6682105Z ---> Package gcc.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:06:48.6844402Z --> Processing Dependency: cpp = 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:06:48.6859984Z --> Processing Dependency: libsanitizer >= 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:06:48.6908771Z --> Processing Dependency: libquadmath >= 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:06:48.6963334Z --> Processing Dependency: libmpx >= 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:06:48.7012833Z --> Processing Dependency: libitm >= 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:06:48.7058203Z --> Processing Dependency: libcilkrts >= 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:06:48.7107627Z --> Processing Dependency: libatomic >= 7.3.1-14.amzn2 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:06:48.7154946Z --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:06:48.7300209Z --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:06:48.7323506Z --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-14.amzn2.x86_64
2022-05-18T04:06:48.7346847Z ---> Package gcc-c++.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:06:48.7384285Z ---> Package gcc-gfortran.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:06:48.7426715Z --> Processing Dependency: libgfortran.so.4()(64bit) for package: gcc-gfortran-7.3.1-14.amzn2.x86_64
2022-05-18T04:06:48.7484159Z ---> Package indent.x86_64 0:2.2.11-13.amzn2.0.2 will be installed
2022-05-18T04:06:48.7507055Z ---> Package intltool.noarch 0:0.50.2-7.amzn2 will be installed
2022-05-18T04:06:48.7559978Z --> Processing Dependency: perl(XML::Parser) for package: intltool-0.50.2-7.amzn2.noarch
2022-05-18T04:06:48.7576756Z --> Processing Dependency: gettext-devel for package: intltool-0.50.2-7.amzn2.noarch
2022-05-18T04:06:48.7597879Z ---> Package libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 will be installed
2022-05-18T04:06:48.7635452Z ---> Package patch.x86_64 0:2.7.1-12.amzn2.0.2 will be installed
2022-05-18T04:06:48.7676847Z ---> Package patchutils.x86_64 0:0.3.3-4.amzn2.0.1 will be installed
2022-05-18T04:06:48.7709110Z ---> Package rcs.x86_64 0:5.9.0-5.amzn2.0.2 will be installed
2022-05-18T04:06:48.7752508Z ---> Package rpm-build.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-05-18T04:06:48.7989000Z --> Processing Dependency: /usr/bin/gdb-add-index for package: rpm-build-4.11.3-48.amzn2.0.2.x86_64
2022-05-18T04:06:48.8010341Z ---> Package rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-05-18T04:06:48.8046389Z ---> Package subversion.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-05-18T04:06:48.8210342Z --> Processing Dependency: subversion-libs(x86-64) = 1.7.14-16.amzn2.0.1 for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8231111Z --> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8231773Z --> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8232409Z --> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8233046Z --> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8233677Z --> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8234294Z --> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8234929Z --> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8235572Z --> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8236213Z --> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8236819Z --> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8237444Z --> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8238077Z --> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8238713Z --> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8239329Z --> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8239943Z --> Processing Dependency: libneon.so.27()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8259809Z --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8282346Z --> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-05-18T04:06:48.8307811Z ---> Package swig.x86_64 0:3.0.12-11.amzn2.0.3 will be installed
2022-05-18T04:06:48.8334487Z ---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.13 will be installed
2022-05-18T04:06:48.8384620Z --> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.13.noarch
2022-05-18T04:06:48.8402795Z --> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.13.noarch
2022-05-18T04:06:48.8417143Z --> Processing Dependency: go-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.13.noarch
2022-05-18T04:06:48.8584504Z ---> Package systemtap.x86_64 0:4.4-1.amzn2.0.2 will be installed
2022-05-18T04:06:48.8601227Z --> Processing Dependency: systemtap-devel = 4.4-1.amzn2.0.2 for package: systemtap-4.4-1.amzn2.0.2.x86_64
2022-05-18T04:06:48.8613071Z --> Processing Dependency: systemtap-client = 4.4-1.amzn2.0.2 for package: systemtap-4.4-1.amzn2.0.2.x86_64
2022-05-18T04:06:48.8624818Z --> Running transaction check
2022-05-18T04:06:48.8626470Z ---> Package apr.x86_64 0:1.7.0-9.amzn2 will be installed
2022-05-18T04:06:48.8711488Z ---> Package apr-util.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-05-18T04:06:48.8759908Z --> Processing Dependency: apr-util-bdb(x86-64) = 1.6.1-5.amzn2.0.2 for package: apr-util-1.6.1-5.amzn2.0.2.x86_64
2022-05-18T04:06:48.8774753Z ---> Package cpp.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:06:48.8860378Z ---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed
2022-05-18T04:06:48.8876330Z ---> Package gdb.x86_64 0:8.0.1-36.amzn2.0.1 will be installed
2022-05-18T04:06:48.8957384Z ---> Package gettext-devel.x86_64 0:0.19.8.1-3.amzn2 will be installed
2022-05-18T04:06:48.9023345Z --> Processing Dependency: gettext-common-devel = 0.19.8.1-3.amzn2 for package: gettext-devel-0.19.8.1-3.amzn2.x86_64
2022-05-18T04:06:48.9032684Z ---> Package glibc-devel.x86_64 0:2.26-58.amzn2 will be installed
2022-05-18T04:06:48.9151525Z --> Processing Dependency: glibc-headers = 2.26-58.amzn2 for package: glibc-devel-2.26-58.amzn2.x86_64
2022-05-18T04:06:48.9176313Z --> Processing Dependency: glibc-headers for package: glibc-devel-2.26-58.amzn2.x86_64
2022-05-18T04:06:48.9176881Z ---> Package go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.1 will be installed
2022-05-18T04:06:48.9182437Z ---> Package libatomic.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:06:48.9205390Z ---> Package libcilkrts.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:06:48.9245131Z ---> Package libgfortran.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:06:48.9292064Z ---> Package libitm.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:06:48.9318162Z ---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed
2022-05-18T04:06:48.9337689Z ---> Package libmpx.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:06:48.9362512Z ---> Package libquadmath.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:06:48.9400064Z ---> Package libsanitizer.x86_64 0:7.3.1-14.amzn2 will be installed
2022-05-18T04:06:48.9460719Z ---> Package m4.x86_64 0:1.4.16-10.amzn2.0.2 will be installed
2022-05-18T04:06:48.9485585Z ---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed
2022-05-18T04:06:48.9516162Z ---> Package neon.x86_64 0:0.30.0-3.amzn2.0.2 will be installed
2022-05-18T04:06:48.9600855Z --> Processing Dependency: libgnutls.so.28(GNUTLS_2_12)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-05-18T04:06:48.9639452Z --> Processing Dependency: libgnutls.so.28(GNUTLS_1_4)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-05-18T04:06:48.9640084Z --> Processing Dependency: libproxy.so.1()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-05-18T04:06:48.9661350Z --> Processing Dependency: libpakchois.so.0()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-05-18T04:06:48.9681435Z --> Processing Dependency: libgnutls.so.28()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-05-18T04:06:48.9688368Z ---> Package perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 will be installed
2022-05-18T04:06:48.9744492Z ---> Package perl-Test-Harness.noarch 0:3.28-3.amzn2 will be installed
2022-05-18T04:06:48.9883908Z ---> Package perl-Thread-Queue.noarch 0:3.02-2.amzn2 will be installed
2022-05-18T04:06:48.9898796Z ---> Package perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 will be installed
2022-05-18T04:06:48.9921167Z ---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed
2022-05-18T04:06:48.9921723Z ---> Package subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-05-18T04:06:48.9970202Z ---> Package systemtap-client.x86_64 0:4.4-1.amzn2.0.2 will be installed
2022-05-18T04:06:49.0218034Z --> Processing Dependency: mokutil for package: systemtap-client-4.4-1.amzn2.0.2.x86_64
2022-05-18T04:06:49.0233782Z --> Processing Dependency: libavahi-common.so.3()(64bit) for package: systemtap-client-4.4-1.amzn2.0.2.x86_64
2022-05-18T04:06:49.0260427Z --> Processing Dependency: libavahi-client.so.3()(64bit) for package: systemtap-client-4.4-1.amzn2.0.2.x86_64
2022-05-18T04:06:49.0261042Z ---> Package systemtap-devel.x86_64 0:4.4-1.amzn2.0.2 will be installed
2022-05-18T04:06:49.0385633Z --> Processing Dependency: kernel-devel-uname-r for package: systemtap-devel-4.4-1.amzn2.0.2.x86_64
2022-05-18T04:06:49.1289694Z --> Running transaction check
2022-05-18T04:06:49.1290177Z ---> Package apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-05-18T04:06:49.1303739Z ---> Package avahi-libs.x86_64 0:0.6.31-20.amzn2 will be installed
2022-05-18T04:06:49.1337440Z ---> Package gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 will be installed
2022-05-18T04:06:49.1338000Z ---> Package glibc-headers.x86_64 0:2.26-58.amzn2 will be installed
2022-05-18T04:06:49.1381995Z --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-58.amzn2.x86_64
2022-05-18T04:06:49.2341464Z --> Processing Dependency: kernel-headers for package: glibc-headers-2.26-58.amzn2.x86_64
2022-05-18T04:06:49.2342367Z ---> Package gnutls.x86_64 0:3.3.29-9.amzn2.0.1 will be installed
2022-05-18T04:06:49.2421803Z --> Processing Dependency: trousers >= 0.3.11.2 for package: gnutls-3.3.29-9.amzn2.0.1.x86_64
2022-05-18T04:06:49.2451035Z ---> Package kernel-devel.x86_64 0:4.14.276-211.499.amzn2 will be installed
2022-05-18T04:06:49.2480053Z --> Processing Dependency: elfutils-libelf-devel for package: kernel-devel-4.14.276-211.499.amzn2.x86_64
2022-05-18T04:06:49.2501106Z ---> Package libproxy.x86_64 0:0.4.11-10.amzn2.0.3 will be installed
2022-05-18T04:06:49.2541411Z --> Processing Dependency: libmodman.so.1()(64bit) for package: libproxy-0.4.11-10.amzn2.0.3.x86_64
2022-05-18T04:06:49.2561586Z ---> Package mokutil.x86_64 1:0.3.0-10.amzn2.0.1 will be installed
2022-05-18T04:06:49.2610610Z --> Processing Dependency: libefivar.so.1(libefivar.so.0)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-05-18T04:06:49.2633474Z --> Processing Dependency: libefivar.so.1(LIBEFIVAR_0.24)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-05-18T04:06:49.2634145Z --> Processing Dependency: libefivar.so.1()(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-05-18T04:06:49.2634677Z ---> Package pakchois.x86_64 0:0.4-10.amzn2.0.2 will be installed
2022-05-18T04:06:49.2655436Z --> Running transaction check
2022-05-18T04:06:49.2655919Z ---> Package efivar-libs.x86_64 0:31-4.amzn2.0.4 will be installed
2022-05-18T04:06:49.2682732Z ---> Package elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 will be installed
2022-05-18T04:06:49.2696276Z --> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.amzn2.x86_64
2022-05-18T04:06:49.2721559Z ---> Package kernel-headers.x86_64 0:4.14.276-211.499.amzn2 will be installed
2022-05-18T04:06:49.2722093Z ---> Package libmodman.x86_64 0:2.0.1-8.amzn2.0.2 will be installed
2022-05-18T04:06:49.2751128Z ---> Package trousers.x86_64 0:0.3.14-2.amzn2.0.2 will be installed
2022-05-18T04:06:49.2816245Z --> Running transaction check
2022-05-18T04:06:49.2816707Z ---> Package zlib-devel.x86_64 0:1.2.7-19.amzn2.0.1 will be installed
2022-05-18T04:06:49.5400073Z --> Finished Dependency Resolution
2022-05-18T04:06:49.6509858Z 
2022-05-18T04:06:49.6510482Z Dependencies Resolved
2022-05-18T04:06:49.6620389Z 
2022-05-18T04:06:49.6620731Z ================================================================================
2022-05-18T04:06:49.6621120Z  Package                 Arch     Version                    Repository    Size
2022-05-18T04:06:49.6621482Z ================================================================================
2022-05-18T04:06:49.6621819Z Installing for group install "Development Tools":
2022-05-18T04:06:49.6622853Z  autoconf                noarch   2.69-11.amzn2              amzn2-core   701 k
2022-05-18T04:06:49.6623293Z  automake                noarch   1.13.4-3.1.amzn2           amzn2-core   679 k
2022-05-18T04:06:49.6623996Z  bison                   x86_64   3.0.4-6.amzn2.0.2          amzn2-core   674 k
2022-05-18T04:06:49.6624457Z  byacc                   x86_64   1.9.20130304-3.amzn2.0.2   amzn2-core    66 k
2022-05-18T04:06:49.6624899Z  cscope                  x86_64   15.8-10.amzn2.0.2          amzn2-core   204 k
2022-05-18T04:06:49.6625313Z  ctags                   x86_64   5.8-13.amzn2.0.2           amzn2-core   157 k
2022-05-18T04:06:49.6625746Z  diffstat                x86_64   1.57-4.amzn2.0.2           amzn2-core    35 k
2022-05-18T04:06:49.6626191Z  doxygen                 x86_64   1:1.8.5-4.amzn2            amzn2-core   3.5 M
2022-05-18T04:06:49.6626611Z  elfutils                x86_64   0.176-2.amzn2              amzn2-core   307 k
2022-05-18T04:06:49.6627047Z  flex                    x86_64   2.5.37-3.amzn2.0.3         amzn2-core   291 k
2022-05-18T04:06:49.6627474Z  gcc                     x86_64   7.3.1-14.amzn2             amzn2-core    22 M
2022-05-18T04:06:49.6627918Z  gcc-c++                 x86_64   7.3.1-14.amzn2             amzn2-core    13 M
2022-05-18T04:06:49.6628339Z  gcc-gfortran            x86_64   7.3.1-14.amzn2             amzn2-core    11 M
2022-05-18T04:06:49.6628779Z  indent                  x86_64   2.2.11-13.amzn2.0.2        amzn2-core   150 k
2022-05-18T04:06:49.6629220Z  intltool                noarch   0.50.2-7.amzn2             amzn2-core    59 k
2022-05-18T04:06:49.6629638Z  libtool                 x86_64   2.4.2-22.2.amzn2.0.2       amzn2-core   588 k
2022-05-18T04:06:49.6630068Z  patch                   x86_64   2.7.1-12.amzn2.0.2         amzn2-core   110 k
2022-05-18T04:06:49.6630498Z  patchutils              x86_64   0.3.3-4.amzn2.0.1          amzn2-core   104 k
2022-05-18T04:06:49.6630939Z  rcs                     x86_64   5.9.0-5.amzn2.0.2          amzn2-core   231 k
2022-05-18T04:06:49.6631356Z  rpm-build               x86_64   4.11.3-48.amzn2.0.2        amzn2-core   150 k
2022-05-18T04:06:49.6631810Z  rpm-sign                x86_64   4.11.3-48.amzn2.0.2        amzn2-core    50 k
2022-05-18T04:06:49.6632255Z  subversion              x86_64   1.7.14-16.amzn2.0.1        amzn2-core   1.0 M
2022-05-18T04:06:49.6632664Z  swig                    x86_64   3.0.12-11.amzn2.0.3        amzn2-core   1.4 M
2022-05-18T04:06:49.6633120Z  system-rpm-config       noarch   9.1.0-76.amzn2.0.13        amzn2-core    89 k
2022-05-18T04:06:49.6633584Z  systemtap               x86_64   4.4-1.amzn2.0.2            amzn2-core    12 k
2022-05-18T04:06:49.6633912Z Installing for dependencies:
2022-05-18T04:06:49.6634308Z  apr                     x86_64   1.7.0-9.amzn2              amzn2-core   122 k
2022-05-18T04:06:49.6634738Z  apr-util                x86_64   1.6.1-5.amzn2.0.2          amzn2-core    99 k
2022-05-18T04:06:49.6635178Z  apr-util-bdb            x86_64   1.6.1-5.amzn2.0.2          amzn2-core    19 k
2022-05-18T04:06:49.6635652Z  avahi-libs              x86_64   0.6.31-20.amzn2            amzn2-core    61 k
2022-05-18T04:06:49.6636255Z  cpp                     x86_64   7.3.1-14.amzn2             amzn2-core   9.2 M
2022-05-18T04:06:49.6636680Z  dwz                     x86_64   0.11-3.amzn2.0.3           amzn2-core    98 k
2022-05-18T04:06:49.6637177Z  efivar-libs             x86_64   31-4.amzn2.0.4             amzn2-core    68 k
2022-05-18T04:06:49.6637626Z  elfutils-libelf-devel   x86_64   0.176-2.amzn2              amzn2-core    40 k
2022-05-18T04:06:49.6638080Z  gdb                     x86_64   8.0.1-36.amzn2.0.1         amzn2-core   3.1 M
2022-05-18T04:06:49.6638601Z  gettext-common-devel    noarch   0.19.8.1-3.amzn2           amzn2-core   410 k
2022-05-18T04:06:49.6639050Z  gettext-devel           x86_64   0.19.8.1-3.amzn2           amzn2-core   320 k
2022-05-18T04:06:49.6639492Z  glibc-devel             x86_64   2.26-58.amzn2              amzn2-core   994 k
2022-05-18T04:06:49.6639946Z  glibc-headers           x86_64   2.26-58.amzn2              amzn2-core   514 k
2022-05-18T04:06:49.6640385Z  gnutls                  x86_64   3.3.29-9.amzn2.0.1         amzn2-core   661 k
2022-05-18T04:06:49.6640880Z  go-srpm-macros          noarch   3.0.15-23.amzn2.0.1        amzn2-core    23 k
2022-05-18T04:06:49.6641360Z  kernel-devel            x86_64   4.14.276-211.499.amzn2     amzn2-core    13 M
2022-05-18T04:06:49.6641819Z  kernel-headers          x86_64   4.14.276-211.499.amzn2     amzn2-core   1.2 M
2022-05-18T04:06:49.6642247Z  libatomic               x86_64   7.3.1-14.amzn2             amzn2-core    46 k
2022-05-18T04:06:49.6642689Z  libcilkrts              x86_64   7.3.1-14.amzn2             amzn2-core    85 k
2022-05-18T04:06:49.6643123Z  libgfortran             x86_64   7.3.1-14.amzn2             amzn2-core   536 k
2022-05-18T04:06:49.6643553Z  libitm                  x86_64   7.3.1-14.amzn2             amzn2-core    84 k
2022-05-18T04:06:49.6643960Z  libmodman               x86_64   2.0.1-8.amzn2.0.2          amzn2-core    29 k
2022-05-18T04:06:49.6644392Z  libmpc                  x86_64   1.0.1-3.amzn2.0.2          amzn2-core    52 k
2022-05-18T04:06:49.6644836Z  libmpx                  x86_64   7.3.1-14.amzn2             amzn2-core    51 k
2022-05-18T04:06:49.6645248Z  libproxy                x86_64   0.4.11-10.amzn2.0.3        amzn2-core    61 k
2022-05-18T04:06:49.6645687Z  libquadmath             x86_64   7.3.1-14.amzn2             amzn2-core   189 k
2022-05-18T04:06:49.6646123Z  libsanitizer            x86_64   7.3.1-14.amzn2             amzn2-core   641 k
2022-05-18T04:06:49.6646553Z  m4                      x86_64   1.4.16-10.amzn2.0.2        amzn2-core   256 k
2022-05-18T04:06:49.6646958Z  mokutil                 x86_64   1:0.3.0-10.amzn2.0.1       amzn2-core    39 k
2022-05-18T04:06:49.6647383Z  mpfr                    x86_64   3.1.1-4.amzn2.0.2          amzn2-core   208 k
2022-05-18T04:06:49.6647802Z  neon                    x86_64   0.30.0-3.amzn2.0.2         amzn2-core   166 k
2022-05-18T04:06:49.6648210Z  pakchois                x86_64   0.4-10.amzn2.0.2           amzn2-core    14 k
2022-05-18T04:06:49.6648664Z  perl-Data-Dumper        x86_64   2.145-3.amzn2.0.2          amzn2-core    48 k
2022-05-18T04:06:49.6649141Z  perl-Test-Harness       noarch   3.28-3.amzn2               amzn2-core   302 k
2022-05-18T04:06:49.6649614Z  perl-Thread-Queue       noarch   3.02-2.amzn2               amzn2-core    17 k
2022-05-18T04:06:49.6650184Z  perl-XML-Parser         x86_64   2.41-10.amzn2.0.2          amzn2-core   223 k
2022-05-18T04:06:49.6650653Z  perl-srpm-macros        noarch   1-8.amzn2.0.1              amzn2-core   4.7 k
2022-05-18T04:06:49.6651118Z  subversion-libs         x86_64   1.7.14-16.amzn2.0.1        amzn2-core   912 k
2022-05-18T04:06:49.6651551Z  systemtap-client        x86_64   4.4-1.amzn2.0.2            amzn2-core   3.7 M
2022-05-18T04:06:49.6652008Z  systemtap-devel         x86_64   4.4-1.amzn2.0.2            amzn2-core   2.3 M
2022-05-18T04:06:49.6652453Z  trousers                x86_64   0.3.14-2.amzn2.0.2         amzn2-core   294 k
2022-05-18T04:06:49.6652883Z  zlib-devel              x86_64   1.2.7-19.amzn2.0.1         amzn2-core    50 k
2022-05-18T04:06:49.6653138Z 
2022-05-18T04:06:49.6653258Z Transaction Summary
2022-05-18T04:06:49.6653543Z ================================================================================
2022-05-18T04:06:49.6653860Z Install  25 Packages (+42 Dependent packages)
2022-05-18T04:06:49.6654062Z 
2022-05-18T04:06:49.6654176Z Total download size: 96 M
2022-05-18T04:06:49.6654444Z Installed size: 303 M
2022-05-18T04:06:49.6654707Z Downloading packages:
2022-05-18T04:06:49.6672275Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-05-18T04:06:51.2472277Z --------------------------------------------------------------------------------
2022-05-18T04:06:51.2472741Z Total                                               61 MB/s |  96 MB  00:01     
2022-05-18T04:06:51.3552062Z Running transaction check
2022-05-18T04:06:51.4367801Z Running transaction test
2022-05-18T04:06:51.8791287Z Transaction test succeeded
2022-05-18T04:06:51.8794283Z Running transaction
2022-05-18T04:06:52.4182923Z   Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64                               1/67 
2022-05-18T04:06:52.4710864Z   Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64                             2/67 
2022-05-18T04:06:52.5131129Z   Installing : m4-1.4.16-10.amzn2.0.2.x86_64                               3/67 
2022-05-18T04:06:52.5439167Z   Installing : apr-1.7.0-9.amzn2.x86_64                                    4/67 
2022-05-18T04:06:52.5733355Z   Installing : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                       5/67 
2022-05-18T04:06:52.6100074Z   Installing : apr-util-1.6.1-5.amzn2.0.2.x86_64                           6/67 
2022-05-18T04:06:52.6638586Z   Installing : avahi-libs-0.6.31-20.amzn2.x86_64                           7/67 
2022-05-18T04:06:52.7126273Z   Installing : libquadmath-7.3.1-14.amzn2.x86_64                           8/67 
2022-05-18T04:06:52.7396444Z   Installing : patch-2.7.1-12.amzn2.0.2.x86_64                             9/67 
2022-05-18T04:06:52.8250339Z   Installing : perl-Thread-Queue-3.02-2.amzn2.noarch                      10/67 
2022-05-18T04:06:53.9247585Z   Installing : libgfortran-7.3.1-14.amzn2.x86_64                          11/67 
2022-05-18T04:06:53.9890047Z   Installing : cpp-7.3.1-14.amzn2.x86_64                                  12/67 
2022-05-18T04:06:54.0472270Z   Installing : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   13/67 
2022-05-18T04:06:54.0784299Z   Installing : elfutils-0.176-2.amzn2.x86_64                              14/67 
2022-05-18T04:06:54.1052012Z   Installing : dwz-0.11-3.amzn2.0.3.x86_64                                15/67 
2022-05-18T04:06:54.1430375Z   Installing : efivar-libs-31-4.amzn2.0.4.x86_64                          16/67 
2022-05-18T04:06:54.8661014Z   Installing : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        17/67 
2022-05-18T04:06:54.9900643Z   Installing : systemtap-client-4.4-1.amzn2.0.2.x86_64                    18/67 
2022-05-18T04:06:55.1573133Z   Installing : trousers-0.3.14-2.amzn2.0.2.x86_64                         19/67 
2022-05-18T04:06:55.1938025Z   Installing : gnutls-3.3.29-9.amzn2.0.1.x86_64                           20/67 
2022-05-18T04:06:55.2158515Z   Installing : zlib-devel-1.2.7-19.amzn2.0.1.x86_64                       21/67 
2022-05-18T04:06:55.2411670Z   Installing : elfutils-libelf-devel-0.176-2.amzn2.x86_64                 22/67 
2022-05-18T04:06:55.6410608Z   Installing : libcilkrts-7.3.1-14.amzn2.x86_64                           23/67 
2022-05-18T04:06:55.6765080Z   Installing : gdb-8.0.1-36.amzn2.0.1.x86_64                              24/67 
2022-05-18T04:06:55.9719319Z   Installing : libitm-7.3.1-14.amzn2.x86_64                               25/67 
2022-05-18T04:06:56.1606269Z   Installing : kernel-headers-4.14.276-211.499.amzn2.x86_64               26/67 
2022-05-18T04:06:56.3026243Z   Installing : glibc-headers-2.26-58.amzn2.x86_64                         27/67 
2022-05-18T04:06:56.3502844Z   Installing : glibc-devel-2.26-58.amzn2.x86_64                           28/67 
2022-05-18T04:06:56.3837172Z   Installing : libmpx-7.3.1-14.amzn2.x86_64                               29/67 
2022-05-18T04:06:56.4111649Z   Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch                      30/67 
2022-05-18T04:06:56.4486848Z   Installing : system-rpm-config-9.1.0-76.amzn2.0.13.noarch               31/67 
2022-05-18T04:06:56.4706463Z   Installing : go-srpm-macros-3.0.15-23.amzn2.0.1.noarch                  32/67 
2022-05-18T04:06:56.5748245Z   Installing : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  33/67 
2022-05-18T04:06:56.6251912Z   Installing : autoconf-2.69-11.amzn2.noarch                              34/67 
2022-05-18T04:06:56.7074299Z   Installing : gettext-common-devel-0.19.8.1-3.amzn2.noarch               35/67 
2022-05-18T04:06:56.7986194Z   Installing : gettext-devel-0.19.8.1-3.amzn2.x86_64                      36/67 
2022-05-18T04:06:56.9065228Z   Installing : perl-Test-Harness-3.28-3.amzn2.noarch                      37/67 
2022-05-18T04:06:56.9468967Z   Installing : automake-1.13.4-3.1.amzn2.noarch                           38/67 
2022-05-18T04:06:56.9858794Z   Installing : libmodman-2.0.1-8.amzn2.0.2.x86_64                         39/67 
2022-05-18T04:06:57.1071220Z   Installing : libproxy-0.4.11-10.amzn2.0.3.x86_64                        40/67 
2022-05-18T04:06:57.1430873Z   Installing : libsanitizer-7.3.1-14.amzn2.x86_64                         41/67 
2022-05-18T04:06:57.2046284Z   Installing : pakchois-0.4-10.amzn2.0.2.x86_64                           42/67 
2022-05-18T04:06:57.3429754Z   Installing : neon-0.30.0-3.amzn2.0.2.x86_64                             43/67 
2022-05-18T04:06:57.3792691Z   Installing : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 44/67 
2022-05-18T04:06:59.4577777Z   Installing : libatomic-7.3.1-14.amzn2.x86_64                            45/67 
2022-05-18T04:07:03.3807744Z   Installing : gcc-7.3.1-14.amzn2.x86_64                                  46/67 
2022-05-18T04:07:16.9527813Z   Installing : kernel-devel-4.14.276-211.499.amzn2.x86_64                 47/67 
2022-05-18T04:07:16.9973135Z   Installing : systemtap-devel-4.4-1.amzn2.0.2.x86_64                     48/67 
2022-05-18T04:07:18.2528005Z   Installing : systemtap-4.4-1.amzn2.0.2.x86_64                           49/67 
2022-05-18T04:07:18.3659335Z   Installing : gcc-gfortran-7.3.1-14.amzn2.x86_64                         50/67 
2022-05-18T04:07:20.0174890Z   Installing : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        51/67 
2022-05-18T04:07:20.2207334Z   Installing : gcc-c++-7.3.1-14.amzn2.x86_64                              52/67 
2022-05-18T04:07:20.3113202Z   Installing : subversion-1.7.14-16.amzn2.0.1.x86_64                      53/67 
2022-05-18T04:07:20.3510310Z   Installing : intltool-0.50.2-7.amzn2.noarch                             54/67 
2022-05-18T04:07:20.4114423Z   Installing : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       55/67 
2022-05-18T04:07:20.5228714Z   Installing : flex-2.5.37-3.amzn2.0.3.x86_64                             56/67 
2022-05-18T04:07:20.5922348Z   Installing : bison-3.0.4-6.amzn2.0.2.x86_64                             57/67 
2022-05-18T04:07:20.6428872Z   Installing : rcs-5.9.0-5.amzn2.0.2.x86_64                               58/67 
2022-05-18T04:07:20.6950784Z   Installing : indent-2.2.11-13.amzn2.0.2.x86_64                          59/67 
2022-05-18T04:07:21.4004075Z   Installing : patchutils-0.3.3-4.amzn2.0.1.x86_64                        60/67 
2022-05-18T04:07:21.4482672Z   Installing : 1:doxygen-1.8.5-4.amzn2.x86_64                             61/67 
2022-05-18T04:07:21.4986643Z   Installing : diffstat-1.57-4.amzn2.0.2.x86_64                           62/67 
2022-05-18T04:07:21.5470988Z   Installing : cscope-15.8-10.amzn2.0.2.x86_64                            63/67 
2022-05-18T04:07:21.8588441Z   Installing : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      64/67 
2022-05-18T04:07:21.9170935Z   Installing : swig-3.0.12-11.amzn2.0.3.x86_64                            65/67 
2022-05-18T04:07:21.9367089Z   Installing : ctags-5.8-13.amzn2.0.2.x86_64                              66/67 
2022-05-18T04:07:22.0084123Z   Installing : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                        67/67 
2022-05-18T04:07:22.0202243Z   Verifying  : systemtap-4.4-1.amzn2.0.2.x86_64                            1/67 
2022-05-18T04:07:22.0307042Z   Verifying  : perl-Thread-Queue-3.02-2.amzn2.noarch                       2/67 
2022-05-18T04:07:22.0404248Z   Verifying  : gettext-devel-0.19.8.1-3.amzn2.x86_64                       3/67 
2022-05-18T04:07:22.0493263Z   Verifying  : glibc-headers-2.26-58.amzn2.x86_64                          4/67 
2022-05-18T04:07:22.0578016Z   Verifying  : patch-2.7.1-12.amzn2.0.2.x86_64                             5/67 
2022-05-18T04:07:22.0659524Z   Verifying  : flex-2.5.37-3.amzn2.0.3.x86_64                              6/67 
2022-05-18T04:07:22.0749731Z   Verifying  : systemtap-client-4.4-1.amzn2.0.2.x86_64                     7/67 
2022-05-18T04:07:22.0840623Z   Verifying  : libmpc-1.0.1-3.amzn2.0.2.x86_64                             8/67 
2022-05-18T04:07:22.0934133Z   Verifying  : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                         9/67 
2022-05-18T04:07:22.1028191Z   Verifying  : ctags-5.8-13.amzn2.0.2.x86_64                              10/67 
2022-05-18T04:07:22.1121284Z   Verifying  : swig-3.0.12-11.amzn2.0.3.x86_64                            11/67 
2022-05-18T04:07:22.1212524Z   Verifying  : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      12/67 
2022-05-18T04:07:22.1297087Z   Verifying  : libatomic-7.3.1-14.amzn2.x86_64                            13/67 
2022-05-18T04:07:22.1380165Z   Verifying  : pakchois-0.4-10.amzn2.0.2.x86_64                           14/67 
2022-05-18T04:07:22.1460405Z   Verifying  : libgfortran-7.3.1-14.amzn2.x86_64                          15/67 
2022-05-18T04:07:22.1546973Z   Verifying  : go-srpm-macros-3.0.15-23.amzn2.0.1.noarch                  16/67 
2022-05-18T04:07:22.1627472Z   Verifying  : libproxy-0.4.11-10.amzn2.0.3.x86_64                        17/67 
2022-05-18T04:07:22.1716597Z   Verifying  : cscope-15.8-10.amzn2.0.2.x86_64                            18/67 
2022-05-18T04:07:22.1791324Z   Verifying  : diffstat-1.57-4.amzn2.0.2.x86_64                           19/67 
2022-05-18T04:07:22.1883596Z   Verifying  : 1:doxygen-1.8.5-4.amzn2.x86_64                             20/67 
2022-05-18T04:07:22.1967023Z   Verifying  : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        21/67 
2022-05-18T04:07:22.2058208Z   Verifying  : libsanitizer-7.3.1-14.amzn2.x86_64                         22/67 
2022-05-18T04:07:22.2142434Z   Verifying  : gnutls-3.3.29-9.amzn2.0.1.x86_64                           23/67 
2022-05-18T04:07:22.2228609Z   Verifying  : libmodman-2.0.1-8.amzn2.0.2.x86_64                         24/67 
2022-05-18T04:07:22.2332594Z   Verifying  : cpp-7.3.1-14.amzn2.x86_64                                  25/67 
2022-05-18T04:07:22.2410585Z   Verifying  : perl-Test-Harness-3.28-3.amzn2.noarch                      26/67 
2022-05-18T04:07:22.2489027Z   Verifying  : autoconf-2.69-11.amzn2.noarch                              27/67 
2022-05-18T04:07:22.2571184Z   Verifying  : intltool-0.50.2-7.amzn2.noarch                             28/67 
2022-05-18T04:07:22.2718928Z   Verifying  : kernel-devel-4.14.276-211.499.amzn2.x86_64                 29/67 
2022-05-18T04:07:22.2804699Z   Verifying  : apr-util-1.6.1-5.amzn2.0.2.x86_64                          30/67 
2022-05-18T04:07:22.2906124Z   Verifying  : libquadmath-7.3.1-14.amzn2.x86_64                          31/67 
2022-05-18T04:07:22.2998602Z   Verifying  : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       32/67 
2022-05-18T04:07:22.3071969Z   Verifying  : gettext-common-devel-0.19.8.1-3.amzn2.noarch               33/67 
2022-05-18T04:07:22.3172324Z   Verifying  : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  34/67 
2022-05-18T04:07:22.3261657Z   Verifying  : elfutils-libelf-devel-0.176-2.amzn2.x86_64                 35/67 
2022-05-18T04:07:22.3369284Z   Verifying  : perl-srpm-macros-1-8.amzn2.0.1.noarch                      36/67 
2022-05-18T04:07:22.3482637Z   Verifying  : libmpx-7.3.1-14.amzn2.x86_64                               37/67 
2022-05-18T04:07:22.3574885Z   Verifying  : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 38/67 
2022-05-18T04:07:22.3659728Z   Verifying  : automake-1.13.4-3.1.amzn2.noarch                           39/67 
2022-05-18T04:07:22.3735817Z   Verifying  : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                      40/67 
2022-05-18T04:07:22.3815909Z   Verifying  : glibc-devel-2.26-58.amzn2.x86_64                           41/67 
2022-05-18T04:07:22.3902582Z   Verifying  : avahi-libs-0.6.31-20.amzn2.x86_64                          42/67 
2022-05-18T04:07:22.3986993Z   Verifying  : kernel-headers-4.14.276-211.499.amzn2.x86_64               43/67 
2022-05-18T04:07:22.4069566Z   Verifying  : bison-3.0.4-6.amzn2.0.2.x86_64                             44/67 
2022-05-18T04:07:22.4169115Z   Verifying  : libitm-7.3.1-14.amzn2.x86_64                               45/67 
2022-05-18T04:07:22.4285378Z   Verifying  : gdb-8.0.1-36.amzn2.0.1.x86_64                              46/67 
2022-05-18T04:07:22.4370016Z   Verifying  : gcc-7.3.1-14.amzn2.x86_64                                  47/67 
2022-05-18T04:07:22.4457396Z   Verifying  : patchutils-0.3.3-4.amzn2.0.1.x86_64                        48/67 
2022-05-18T04:07:22.4542761Z   Verifying  : gcc-gfortran-7.3.1-14.amzn2.x86_64                         49/67 
2022-05-18T04:07:22.4622905Z   Verifying  : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        50/67 
2022-05-18T04:07:22.4703244Z   Verifying  : indent-2.2.11-13.amzn2.0.2.x86_64                          51/67 
2022-05-18T04:07:22.4800404Z   Verifying  : subversion-1.7.14-16.amzn2.0.1.x86_64                      52/67 
2022-05-18T04:07:22.4886433Z   Verifying  : libcilkrts-7.3.1-14.amzn2.x86_64                           53/67 
2022-05-18T04:07:22.4974969Z   Verifying  : apr-1.7.0-9.amzn2.x86_64                                   54/67 
2022-05-18T04:07:22.5080514Z   Verifying  : system-rpm-config-9.1.0-76.amzn2.0.13.noarch               55/67 
2022-05-18T04:07:22.5242760Z   Verifying  : gcc-c++-7.3.1-14.amzn2.x86_64                              56/67 
2022-05-18T04:07:22.5325808Z   Verifying  : zlib-devel-1.2.7-19.amzn2.0.1.x86_64                       57/67 
2022-05-18T04:07:22.5413047Z   Verifying  : mpfr-3.1.1-4.amzn2.0.2.x86_64                              58/67 
2022-05-18T04:07:22.5494522Z   Verifying  : trousers-0.3.14-2.amzn2.0.2.x86_64                         59/67 
2022-05-18T04:07:22.5587307Z   Verifying  : neon-0.30.0-3.amzn2.0.2.x86_64                             60/67 
2022-05-18T04:07:22.5684923Z   Verifying  : efivar-libs-31-4.amzn2.0.4.x86_64                          61/67 
2022-05-18T04:07:22.5769178Z   Verifying  : dwz-0.11-3.amzn2.0.3.x86_64                                62/67 
2022-05-18T04:07:22.5867981Z   Verifying  : rcs-5.9.0-5.amzn2.0.2.x86_64                               63/67 
2022-05-18T04:07:22.5957709Z   Verifying  : systemtap-devel-4.4-1.amzn2.0.2.x86_64                     64/67 
2022-05-18T04:07:22.6043274Z   Verifying  : elfutils-0.176-2.amzn2.x86_64                              65/67 
2022-05-18T04:07:22.6125606Z   Verifying  : m4-1.4.16-10.amzn2.0.2.x86_64                              66/67 
2022-05-18T04:07:22.6913641Z   Verifying  : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   67/67 
2022-05-18T04:07:22.6914111Z 
2022-05-18T04:07:22.6914352Z Installed:
2022-05-18T04:07:22.6917554Z   autoconf.noarch 0:2.69-11.amzn2                                               
2022-05-18T04:07:22.6918078Z   automake.noarch 0:1.13.4-3.1.amzn2                                            
2022-05-18T04:07:22.6918672Z   bison.x86_64 0:3.0.4-6.amzn2.0.2                                              
2022-05-18T04:07:22.6919379Z   byacc.x86_64 0:1.9.20130304-3.amzn2.0.2                                       
2022-05-18T04:07:22.6919872Z   cscope.x86_64 0:15.8-10.amzn2.0.2                                             
2022-05-18T04:07:22.6920652Z   ctags.x86_64 0:5.8-13.amzn2.0.2                                               
2022-05-18T04:07:22.6921200Z   diffstat.x86_64 0:1.57-4.amzn2.0.2                                            
2022-05-18T04:07:22.6921753Z   doxygen.x86_64 1:1.8.5-4.amzn2                                                
2022-05-18T04:07:22.6922196Z   elfutils.x86_64 0:0.176-2.amzn2                                               
2022-05-18T04:07:22.6922760Z   flex.x86_64 0:2.5.37-3.amzn2.0.3                                              
2022-05-18T04:07:22.6923258Z   gcc.x86_64 0:7.3.1-14.amzn2                                                   
2022-05-18T04:07:22.6923790Z   gcc-c++.x86_64 0:7.3.1-14.amzn2                                               
2022-05-18T04:07:22.6925410Z   gcc-gfortran.x86_64 0:7.3.1-14.amzn2                                          
2022-05-18T04:07:22.6925928Z   indent.x86_64 0:2.2.11-13.amzn2.0.2                                           
2022-05-18T04:07:22.6926532Z   intltool.noarch 0:0.50.2-7.amzn2                                              
2022-05-18T04:07:22.6927132Z   libtool.x86_64 0:2.4.2-22.2.amzn2.0.2                                         
2022-05-18T04:07:22.6927657Z   patch.x86_64 0:2.7.1-12.amzn2.0.2                                             
2022-05-18T04:07:22.6928164Z   patchutils.x86_64 0:0.3.3-4.amzn2.0.1                                         
2022-05-18T04:07:22.6929137Z   rcs.x86_64 0:5.9.0-5.amzn2.0.2                                                
2022-05-18T04:07:22.6929614Z   rpm-build.x86_64 0:4.11.3-48.amzn2.0.2                                        
2022-05-18T04:07:22.6930544Z   rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2                                         
2022-05-18T04:07:22.6931636Z   subversion.x86_64 0:1.7.14-16.amzn2.0.1                                       
2022-05-18T04:07:22.6932377Z   swig.x86_64 0:3.0.12-11.amzn2.0.3                                             
2022-05-18T04:07:22.6932929Z   system-rpm-config.noarch 0:9.1.0-76.amzn2.0.13                                
2022-05-18T04:07:22.6933493Z   systemtap.x86_64 0:4.4-1.amzn2.0.2                                            
2022-05-18T04:07:22.6933685Z 
2022-05-18T04:07:22.6933855Z Dependency Installed:
2022-05-18T04:07:22.6934546Z   apr.x86_64 0:1.7.0-9.amzn2                                                    
2022-05-18T04:07:22.6935089Z   apr-util.x86_64 0:1.6.1-5.amzn2.0.2                                           
2022-05-18T04:07:22.6935548Z   apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2                                       
2022-05-18T04:07:22.6936113Z   avahi-libs.x86_64 0:0.6.31-20.amzn2                                           
2022-05-18T04:07:22.6936618Z   cpp.x86_64 0:7.3.1-14.amzn2                                                   
2022-05-18T04:07:22.6937106Z   dwz.x86_64 0:0.11-3.amzn2.0.3                                                 
2022-05-18T04:07:22.6937551Z   efivar-libs.x86_64 0:31-4.amzn2.0.4                                           
2022-05-18T04:07:22.6938129Z   elfutils-libelf-devel.x86_64 0:0.176-2.amzn2                                  
2022-05-18T04:07:22.6938658Z   gdb.x86_64 0:8.0.1-36.amzn2.0.1                                               
2022-05-18T04:07:22.6939127Z   gettext-common-devel.noarch 0:0.19.8.1-3.amzn2                                
2022-05-18T04:07:22.6939683Z   gettext-devel.x86_64 0:0.19.8.1-3.amzn2                                       
2022-05-18T04:07:22.6940179Z   glibc-devel.x86_64 0:2.26-58.amzn2                                            
2022-05-18T04:07:22.6940710Z   glibc-headers.x86_64 0:2.26-58.amzn2                                          
2022-05-18T04:07:22.6941150Z   gnutls.x86_64 0:3.3.29-9.amzn2.0.1                                            
2022-05-18T04:07:22.6941660Z   go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.1                                   
2022-05-18T04:07:22.6942660Z   kernel-devel.x86_64 0:4.14.276-211.499.amzn2                                  
2022-05-18T04:07:22.6943124Z   kernel-headers.x86_64 0:4.14.276-211.499.amzn2                                
2022-05-18T04:07:22.6943887Z   libatomic.x86_64 0:7.3.1-14.amzn2                                             
2022-05-18T04:07:22.6944403Z   libcilkrts.x86_64 0:7.3.1-14.amzn2                                            
2022-05-18T04:07:22.6944923Z   libgfortran.x86_64 0:7.3.1-14.amzn2                                           
2022-05-18T04:07:22.6945358Z   libitm.x86_64 0:7.3.1-14.amzn2                                                
2022-05-18T04:07:22.6945875Z   libmodman.x86_64 0:2.0.1-8.amzn2.0.2                                          
2022-05-18T04:07:22.6946380Z   libmpc.x86_64 0:1.0.1-3.amzn2.0.2                                             
2022-05-18T04:07:22.6946814Z   libmpx.x86_64 0:7.3.1-14.amzn2                                                
2022-05-18T04:07:22.6947396Z   libproxy.x86_64 0:0.4.11-10.amzn2.0.3                                         
2022-05-18T04:07:22.6947899Z   libquadmath.x86_64 0:7.3.1-14.amzn2                                           
2022-05-18T04:07:22.6994893Z   libsanitizer.x86_64 0:7.3.1-14.amzn2                                          
2022-05-18T04:07:22.6995373Z   m4.x86_64 0:1.4.16-10.amzn2.0.2                                               
2022-05-18T04:07:22.6995754Z   mokutil.x86_64 1:0.3.0-10.amzn2.0.1                                           
2022-05-18T04:07:22.6996152Z   mpfr.x86_64 0:3.1.1-4.amzn2.0.2                                               
2022-05-18T04:07:22.6996602Z   neon.x86_64 0:0.30.0-3.amzn2.0.2                                              
2022-05-18T04:07:22.6997047Z   pakchois.x86_64 0:0.4-10.amzn2.0.2                                            
2022-05-18T04:07:22.6997484Z   perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2                                   
2022-05-18T04:07:22.6997958Z   perl-Test-Harness.noarch 0:3.28-3.amzn2                                       
2022-05-18T04:07:22.6998428Z   perl-Thread-Queue.noarch 0:3.02-2.amzn2                                       
2022-05-18T04:07:22.6998894Z   perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2                                    
2022-05-18T04:07:22.6999381Z   perl-srpm-macros.noarch 0:1-8.amzn2.0.1                                       
2022-05-18T04:07:22.6999855Z   subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1                                  
2022-05-18T04:07:22.7000315Z   systemtap-client.x86_64 0:4.4-1.amzn2.0.2                                     
2022-05-18T04:07:22.7000808Z   systemtap-devel.x86_64 0:4.4-1.amzn2.0.2                                      
2022-05-18T04:07:22.7001261Z   trousers.x86_64 0:0.3.14-2.amzn2.0.2                                          
2022-05-18T04:07:22.7001698Z   zlib-devel.x86_64 0:1.2.7-19.amzn2.0.1                                        
2022-05-18T04:07:22.7001910Z 
2022-05-18T04:07:22.7001996Z Complete!
2022-05-18T04:07:22.7309553Z ++ uname -r
2022-05-18T04:07:22.7315735Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.252-195.483.amzn2.x86_64'
2022-05-18T04:07:23.2584894Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-05-18T04:07:23.2750910Z Existing lock /var/run/yum.pid: another copy is running as pid 36407.
2022-05-18T04:07:23.2751382Z Another app is currently holding the yum lock; waiting for it to exit...
2022-05-18T04:07:23.2763202Z   The other application is: yum
2022-05-18T04:07:23.2763533Z     Memory :  43 M RSS (261 MB VSZ)
2022-05-18T04:07:23.2763950Z     Started: Wed May 18 04:07:22 2022 - 00:01 ago
2022-05-18T04:07:23.2764272Z     State  : Running, pid: 36407
2022-05-18T04:07:25.2788605Z Another app is currently holding the yum lock; waiting for it to exit...
2022-05-18T04:07:25.2798030Z   The other application is: yum
2022-05-18T04:07:25.2798344Z     Memory : 155 M RSS (374 MB VSZ)
2022-05-18T04:07:25.2798879Z     Started: Wed May 18 04:07:22 2022 - 00:03 ago
2022-05-18T04:07:25.2799195Z     State  : Running, pid: 36407
2022-05-18T04:07:28.6098733Z Resolving Dependencies
2022-05-18T04:07:28.6105971Z --> Running transaction check
2022-05-18T04:07:28.6106532Z ---> Package kernel-devel.x86_64 0:4.14.252-195.483.amzn2 will be installed
2022-05-18T04:07:28.9656846Z --> Finished Dependency Resolution
2022-05-18T04:07:29.0913379Z 
2022-05-18T04:07:29.0914160Z Dependencies Resolved
2022-05-18T04:07:29.0918145Z 
2022-05-18T04:07:29.0918347Z ================================================================================
2022-05-18T04:07:29.0918926Z  Package          Arch       Version                       Repository      Size
2022-05-18T04:07:29.0919391Z ================================================================================
2022-05-18T04:07:29.0919912Z Installing:
2022-05-18T04:07:29.0920904Z  kernel-devel     x86_64     4.14.252-195.483.amzn2        amzn2-core      13 M
2022-05-18T04:07:29.0921342Z 
2022-05-18T04:07:29.0921583Z Transaction Summary
2022-05-18T04:07:29.0922077Z ================================================================================
2022-05-18T04:07:29.0922475Z Install  1 Package
2022-05-18T04:07:29.0922645Z 
2022-05-18T04:07:29.0922787Z Total download size: 13 M
2022-05-18T04:07:29.0923064Z Installed size: 60 M
2022-05-18T04:07:29.0923553Z Downloading packages:
2022-05-18T04:07:29.0932390Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-05-18T04:07:29.3997203Z Running transaction check
2022-05-18T04:07:29.4189162Z Running transaction test
2022-05-18T04:07:29.8316605Z Transaction test succeeded
2022-05-18T04:07:29.8318951Z Running transaction
2022-05-18T04:07:47.9429314Z   Installing : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-05-18T04:07:48.0285576Z   Verifying  : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-05-18T04:07:48.0285955Z 
2022-05-18T04:07:48.0286083Z Installed:
2022-05-18T04:07:48.0286499Z   kernel-devel.x86_64 0:4.14.252-195.483.amzn2                                  
2022-05-18T04:07:48.0286724Z 
2022-05-18T04:07:48.0286850Z Complete!
2022-05-18T04:07:48.0653631Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-510.60.02.run
2022-05-18T04:07:51.3820079Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm
2022-05-18T04:07:52.6011938Z Verifying archive integrity... OK
2022-05-18T04:08:16.6974418Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 510.60.02..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2022-05-18T04:08:16.8522099Z 
2022-05-18T04:08:16.8526270Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver.
2022-05-18T04:08:16.8526654Z 
2022-05-18T04:08:31.0087174Z 
2022-05-18T04:08:31.0088688Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system.  If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver.
2022-05-18T04:08:31.0090010Z 
2022-05-18T04:08:39.3935596Z + sudo rm -fv /tmp/nvidia_driver
2022-05-18T04:08:39.4458033Z removed ‘/tmp/nvidia_driver’
2022-05-18T04:08:39.4476498Z + nvidia-smi
2022-05-18T04:08:47.6698583Z Wed May 18 04:08:47 2022       
2022-05-18T04:08:47.6699287Z +-----------------------------------------------------------------------------+
2022-05-18T04:08:47.6699837Z | NVIDIA-SMI 510.60.02    Driver Version: 510.60.02    CUDA Version: 11.6     |
2022-05-18T04:08:47.6700345Z |-------------------------------+----------------------+----------------------+
2022-05-18T04:08:47.6701267Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2022-05-18T04:08:47.6701791Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2022-05-18T04:08:47.6703011Z |                               |                      |               MIG M. |
2022-05-18T04:08:47.6703847Z |===============================+======================+======================|
2022-05-18T04:08:47.6754154Z |   0  Tesla M60           Off  | 00000000:00:1B.0 Off |                    0 |
2022-05-18T04:08:47.6754531Z | N/A   31C    P0    37W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-05-18T04:08:47.6754873Z |                               |                      |                  N/A |
2022-05-18T04:08:47.6755383Z +-------------------------------+----------------------+----------------------+
2022-05-18T04:08:47.6837382Z |   1  Tesla M60           Off  | 00000000:00:1C.0 Off |                    0 |
2022-05-18T04:08:47.6838294Z | N/A   28C    P0    37W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-05-18T04:08:47.6838678Z |                               |                      |                  N/A |
2022-05-18T04:08:47.6839191Z +-------------------------------+----------------------+----------------------+
2022-05-18T04:08:47.6891077Z |   2  Tesla M60           Off  | 00000000:00:1D.0 Off |                    0 |
2022-05-18T04:08:47.6891617Z | N/A   30C    P0    38W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-05-18T04:08:47.6892052Z |                               |                      |                  N/A |
2022-05-18T04:08:47.6892557Z +-------------------------------+----------------------+----------------------+
2022-05-18T04:08:47.6946201Z |   3  Tesla M60           Off  | 00000000:00:1E.0 Off |                    0 |
2022-05-18T04:08:47.6946569Z | N/A   26C    P0    40W / 150W |      0MiB /  7680MiB |     97%      Default |
2022-05-18T04:08:47.6947203Z |                               |                      |                  N/A |
2022-05-18T04:08:47.6948198Z +-------------------------------+----------------------+----------------------+
2022-05-18T04:08:47.6948591Z                                                                                
2022-05-18T04:08:47.6949392Z +-----------------------------------------------------------------------------+
2022-05-18T04:08:47.6950244Z | Processes:                                                                  |
2022-05-18T04:08:47.6950743Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2022-05-18T04:08:47.6951107Z |        ID   ID                                                   Usage      |
2022-05-18T04:08:47.6951401Z |=============================================================================|
2022-05-18T04:08:47.6959528Z |  No running processes found                                                 |
2022-05-18T04:08:47.6960294Z +-----------------------------------------------------------------------------+
2022-05-18T04:08:48.7946959Z + echo 'GPU_FLAG=--gpus all'
2022-05-18T04:08:49.5756272Z Command completed after 1 attempt(s).
2022-05-18T04:08:49.5756551Z 
2022-05-18T04:08:49.5838961Z Prepare all required actions
2022-05-18T04:08:49.5839498Z Getting action download info
2022-05-18T04:08:50.0153599Z Download action repository 'seemethere/download-artifact-s3@v3' (SHA:64048a097659c8ca71ceacbb3c01cee9ed6f1b05)
2022-05-18T04:08:50.2025169Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869)
2022-05-18T04:08:50.3218977Z ##[group]Run ./.github/actions/download-build-artifacts
2022-05-18T04:08:50.3219277Z with:
2022-05-18T04:08:50.3219575Z   name: linux-bionic-cuda10.2-py3.9-gcc7
2022-05-18T04:08:50.3219865Z env:
2022-05-18T04:08:50.3220066Z   IN_CI: 1
2022-05-18T04:08:50.3220295Z   IS_GHA: 1
2022-05-18T04:08:50.3220560Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:08:50.3220813Z   GPU_FLAG: --gpus all
2022-05-18T04:08:50.3221072Z ##[endgroup]
2022-05-18T04:08:50.3252096Z ##[group]Run seemethere/download-artifact-s3@v3
2022-05-18T04:08:50.3252406Z with:
2022-05-18T04:08:50.3252732Z   name: linux-bionic-cuda10.2-py3.9-gcc7
2022-05-18T04:08:50.3253025Z   s3-bucket: gha-artifacts
2022-05-18T04:08:50.3253308Z   region: us-east-1
2022-05-18T04:08:50.3253547Z env:
2022-05-18T04:08:50.3253744Z   IN_CI: 1
2022-05-18T04:08:50.3253984Z   IS_GHA: 1
2022-05-18T04:08:50.3254239Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:08:50.3254493Z   GPU_FLAG: --gpus all
2022-05-18T04:08:50.3254747Z ##[endgroup]
2022-05-18T04:08:51.0597850Z Found 1 objects with prefix pytorch/pytorch/2342799949/1/linux-bionic-cuda10.2-py3.9-gcc7/
2022-05-18T04:08:51.0598530Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-05-18T04:08:56.5771314Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-05-18T04:08:56.5771694Z 
2022-05-18T04:08:56.5773057Z Artifact download has finished successfully
2022-05-18T04:08:56.5923926Z ##[group]Run unzip -o artifacts.zip
2022-05-18T04:08:56.5924252Z unzip -o artifacts.zip
2022-05-18T04:08:56.5939196Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:08:56.5939486Z env:
2022-05-18T04:08:56.5939698Z   IN_CI: 1
2022-05-18T04:08:56.5939896Z   IS_GHA: 1
2022-05-18T04:08:56.5940135Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:08:56.5940392Z   GPU_FLAG: --gpus all
2022-05-18T04:08:56.5940620Z ##[endgroup]
2022-05-18T04:08:56.6032840Z Archive:  artifacts.zip
2022-05-18T04:08:56.6033573Z    creating: dist/
2022-05-18T04:08:58.4608987Z   inflating: dist/torch-1.12.0a0+git3b23752-cp39-cp39-linux_x86_64.whl  
2022-05-18T04:08:58.4609419Z    creating: build/custom_test_artifacts/
2022-05-18T04:08:58.4609846Z    creating: build/custom_test_artifacts/custom-op-build/
2022-05-18T04:08:58.4610321Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-05-18T04:08:58.4617634Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-05-18T04:08:58.4618256Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-05-18T04:08:58.4618841Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-05-18T04:08:58.4619415Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-05-18T04:08:58.4619972Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-05-18T04:08:58.4622364Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-05-18T04:08:58.4624602Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-05-18T04:08:58.4625183Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-05-18T04:08:58.4625745Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-05-18T04:08:58.4628906Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-05-18T04:08:58.4631061Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-05-18T04:08:58.4632906Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-05-18T04:08:58.4633821Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-05-18T04:08:58.4636036Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-05-18T04:08:58.4637397Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-05-18T04:08:58.4637988Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-05-18T04:08:58.4638567Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-05-18T04:08:58.4685510Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-05-18T04:08:58.4686297Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-05-18T04:08:58.4687042Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-05-18T04:08:58.4688644Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-05-18T04:08:58.4690224Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-05-18T04:08:58.4691641Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-05-18T04:08:58.4693003Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_30.cubin  
2022-05-18T04:08:58.4694503Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-05-18T04:08:58.4695965Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-05-18T04:08:58.4732827Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-05-18T04:08:58.4769142Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-05-18T04:08:58.4770618Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-05-18T04:08:58.4772008Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_30.cubin  
2022-05-18T04:08:58.4773291Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-05-18T04:08:58.4774658Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-05-18T04:08:58.4775996Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-05-18T04:08:58.4777286Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-05-18T04:08:58.4778783Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-05-18T04:08:58.4842422Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-05-18T04:08:58.4904912Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-05-18T04:08:58.4906252Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-05-18T04:08:58.4907424Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-05-18T04:08:58.4908687Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-05-18T04:08:58.4909888Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-05-18T04:08:58.4911000Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-05-18T04:08:58.4912205Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-05-18T04:08:58.4913432Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-05-18T04:08:58.4914664Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-05-18T04:08:58.4915893Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-05-18T04:08:58.4917147Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-05-18T04:08:58.4918515Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-05-18T04:08:58.4919617Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-05-18T04:08:58.4920712Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-05-18T04:08:58.4921888Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-05-18T04:08:58.4938103Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-05-18T04:08:58.5049283Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-05-18T04:08:58.5050470Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-05-18T04:08:58.5051735Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-05-18T04:08:58.5053056Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-05-18T04:08:58.5054322Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-05-18T04:08:58.5055549Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-05-18T04:08:58.5056858Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-05-18T04:08:58.5058113Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-05-18T04:08:58.5059310Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-05-18T04:08:58.5060427Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-05-18T04:08:58.5061557Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-05-18T04:08:58.5079496Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-05-18T04:08:58.5161004Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-05-18T04:08:58.5162542Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-05-18T04:08:58.5163811Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-05-18T04:08:58.5164963Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-05-18T04:08:58.5166037Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-05-18T04:08:58.5167138Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-05-18T04:08:58.5168250Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2022-05-18T04:08:58.5171664Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-05-18T04:08:58.5172843Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-05-18T04:08:58.5173859Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-05-18T04:08:58.5265468Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-05-18T04:08:58.5329539Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-05-18T04:08:58.5330464Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-05-18T04:08:58.5331356Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-05-18T04:08:58.5338720Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-05-18T04:08:58.5339829Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-05-18T04:08:58.5340963Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-05-18T04:08:58.5342604Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-05-18T04:08:58.5343756Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-05-18T04:08:58.5344957Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-05-18T04:08:58.5346123Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-05-18T04:08:58.5347304Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-05-18T04:08:58.5348416Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-05-18T04:08:58.5350604Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-05-18T04:08:58.5352527Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-05-18T04:08:58.5355395Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-05-18T04:08:58.5356679Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-05-18T04:08:58.5358191Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-05-18T04:08:58.5359709Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-05-18T04:08:58.5360930Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-05-18T04:08:58.5362090Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-05-18T04:08:58.5407926Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-05-18T04:08:58.5409385Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-05-18T04:08:58.5410938Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-05-18T04:08:58.5412476Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-05-18T04:08:58.5413993Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-05-18T04:08:58.5415395Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-05-18T04:08:58.5416846Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_30.cubin  
2022-05-18T04:08:58.5418252Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-05-18T04:08:58.5419750Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-05-18T04:08:58.5454965Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-05-18T04:08:58.5491758Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-05-18T04:08:58.5493239Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-05-18T04:08:58.5494583Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_30.cubin  
2022-05-18T04:08:58.5495878Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-05-18T04:08:58.5497134Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-05-18T04:08:58.5498580Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-05-18T04:08:58.5499877Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-05-18T04:08:58.5501233Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-05-18T04:08:58.5561925Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-05-18T04:08:58.5624311Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-05-18T04:08:58.5625639Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-05-18T04:08:58.5626784Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-05-18T04:08:58.5627869Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-05-18T04:08:58.5628989Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-05-18T04:08:58.5630101Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-05-18T04:08:58.5631345Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-05-18T04:08:58.5632580Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-05-18T04:08:58.5633824Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-05-18T04:08:58.5635018Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-05-18T04:08:58.5636308Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-05-18T04:08:58.5637716Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-05-18T04:08:58.5638859Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-05-18T04:08:58.5639929Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-05-18T04:08:58.5641077Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-05-18T04:08:58.5657974Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-05-18T04:08:58.5723017Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-05-18T04:08:58.5724312Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-05-18T04:08:58.5725532Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-05-18T04:08:58.5726668Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-05-18T04:08:58.5727727Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-05-18T04:08:58.5729024Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-05-18T04:08:58.5730031Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2022-05-18T04:08:58.5731294Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-05-18T04:08:58.5732720Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-05-18T04:08:58.5733919Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-05-18T04:08:58.5783853Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-05-18T04:08:58.5784861Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-05-18T04:08:58.5785800Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-05-18T04:08:58.5793000Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-05-18T04:08:58.5794326Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-05-18T04:08:58.5795533Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-05-18T04:08:58.5796712Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-05-18T04:08:58.5797866Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-05-18T04:08:58.5799168Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-05-18T04:08:58.5800432Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-05-18T04:08:58.5801667Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-05-18T04:08:58.5802848Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-05-18T04:08:58.5805074Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-05-18T04:08:58.5806978Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-05-18T04:08:58.5809180Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-05-18T04:08:58.5810557Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-05-18T04:08:58.5812370Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-05-18T04:08:58.5814101Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-05-18T04:08:58.5815402Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-05-18T04:08:58.5816636Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-05-18T04:08:58.5862950Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-05-18T04:08:58.5864501Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-05-18T04:08:58.5866060Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-05-18T04:08:58.5867589Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-05-18T04:08:58.5869160Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-05-18T04:08:58.5870701Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-05-18T04:08:58.5872401Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_30.cubin  
2022-05-18T04:08:58.5873904Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-05-18T04:08:58.5875360Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-05-18T04:08:58.5910750Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-05-18T04:08:58.5946937Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-05-18T04:08:58.5948273Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-05-18T04:08:58.5949223Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_30.cubin  
2022-05-18T04:08:58.5950392Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-05-18T04:08:58.5951784Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-05-18T04:08:58.5953145Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-05-18T04:08:58.5954477Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-05-18T04:08:58.5955511Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-05-18T04:08:58.6016399Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-05-18T04:08:58.6078494Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-05-18T04:08:58.6079212Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-05-18T04:08:58.6079785Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-05-18T04:08:58.6080358Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-05-18T04:08:58.6080938Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-05-18T04:08:58.6081519Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-05-18T04:08:58.6082351Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-05-18T04:08:58.6083232Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-05-18T04:08:58.6084626Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-05-18T04:08:58.6085937Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-05-18T04:08:58.6087262Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-05-18T04:08:58.6088562Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-05-18T04:08:58.6089914Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-05-18T04:08:58.6091213Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-05-18T04:08:58.6092447Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-05-18T04:08:58.6096366Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-05-18T04:08:58.6244993Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-05-18T04:08:58.6246299Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-05-18T04:08:58.6247561Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-05-18T04:08:58.6248955Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-05-18T04:08:58.6250280Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-05-18T04:08:58.6251577Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-05-18T04:08:58.6252938Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-05-18T04:08:58.6254386Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-05-18T04:08:58.6255643Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-05-18T04:08:58.6256810Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-05-18T04:08:58.6258067Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-05-18T04:08:58.6275467Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-05-18T04:08:58.6334298Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-05-18T04:08:58.6335710Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-05-18T04:08:58.6337047Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-05-18T04:08:58.6338252Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-05-18T04:08:58.6339402Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-05-18T04:08:58.6340595Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-05-18T04:08:58.6341672Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2022-05-18T04:08:58.6344003Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-05-18T04:08:58.6344835Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-05-18T04:08:58.6345762Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-05-18T04:08:58.6467503Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-05-18T04:08:58.6513326Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-05-18T04:08:58.6513730Z    creating: build/lib/
2022-05-18T04:08:58.6514033Z   inflating: build/lib/libclog.a     
2022-05-18T04:08:58.6582253Z   inflating: build/lib/libgtest.a    
2022-05-18T04:08:58.6593596Z   inflating: build/lib/libpthreadpool.a  
2022-05-18T04:08:58.6685388Z   inflating: build/lib/libbenchmark.a  
2022-05-18T04:08:58.6795456Z   inflating: build/lib/libprotobuf-lite.a  
2022-05-18T04:08:58.6828094Z   inflating: build/lib/libtensorpipe_uv.a  
2022-05-18T04:08:58.6885517Z   inflating: build/lib/libasmjit.a   
2022-05-18T04:08:58.7021899Z   inflating: build/lib/libgloo.a     
2022-05-18T04:08:58.7564854Z   inflating: build/lib/libprotobuf.a  
2022-05-18T04:08:58.7584574Z   inflating: build/lib/libfmt.a      
2022-05-18T04:08:58.7585379Z   inflating: build/lib/libcaffe2_nvrtc.so  
2022-05-18T04:08:58.7586188Z   inflating: build/lib/libfoxi_loader.a  
2022-05-18T04:08:58.7653090Z   inflating: build/lib/libc10.so     
2022-05-18T04:08:58.7653824Z   inflating: build/lib/libtorch_global_deps.so  
2022-05-18T04:08:58.7664244Z   inflating: build/lib/libcpuinfo.a  
2022-05-18T04:08:58.7673187Z   inflating: build/lib/libcpuinfo_internals.a  
2022-05-18T04:08:58.7689061Z   inflating: build/lib/libqnnpack.a  
2022-05-18T04:08:58.8277266Z   inflating: build/lib/libprotoc.a   
2022-05-18T04:08:58.8278728Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-05-18T04:08:58.8304741Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-05-18T04:08:58.8323821Z   inflating: build/lib/libgmock.a    
2022-05-18T04:08:58.8324408Z   inflating: build/lib/libgtest_main.a  
2022-05-18T04:08:58.8324771Z   inflating: build/lib/libbenchmark_main.a  
2022-05-18T04:08:58.8347946Z   inflating: build/lib/libnnpack.a   
2022-05-18T04:08:59.6693267Z   inflating: build/lib/libdnnl.a     
2022-05-18T04:08:59.7368428Z   inflating: build/lib/libtensorpipe.a  
2022-05-18T04:08:59.7412244Z   inflating: build/lib/libc10_cuda.so  
2022-05-18T04:08:59.8976679Z   inflating: build/lib/libfbgemm.a   
2022-05-18T04:08:59.8977337Z   inflating: build/lib/libgmock_main.a  
2022-05-18T04:09:00.0136851Z   inflating: build/lib/libdnnl_graph.a  
2022-05-18T04:09:00.0576330Z   inflating: build/lib/libkineto.a   
2022-05-18T04:09:00.0874922Z   inflating: build/lib/libtensorpipe_cuda.a  
2022-05-18T04:09:00.0922305Z   inflating: build/lib/libcaffe2_protos.a  
2022-05-18T04:09:00.0971401Z   inflating: build/lib/libonnx_proto.a  
2022-05-18T04:09:00.1117131Z   inflating: build/lib/libXNNPACK.a  
2022-05-18T04:09:00.1798440Z   inflating: build/lib/libonnx.a     
2022-05-18T04:09:00.2239432Z   inflating: build/lib/libgloo_cuda.a  
2022-05-18T04:09:02.3819446Z   inflating: build/lib/libtorch_cpu.so  
2022-05-18T04:09:04.4232738Z   inflating: build/lib/libtorch_cuda.so  
2022-05-18T04:09:04.4233464Z   inflating: build/lib/libtorch.so   
2022-05-18T04:09:04.4237367Z   inflating: build/lib/libc10d_cuda_test.so  
2022-05-18T04:09:05.0539115Z   inflating: build/lib/libtorch_cuda_linalg.so  
2022-05-18T04:09:05.0563212Z   inflating: build/lib/libjitbackend_test.so  
2022-05-18T04:09:05.0594679Z   inflating: build/lib/libbackend_with_compiler.so  
2022-05-18T04:09:05.0648945Z   inflating: build/lib/libtorchbind_test.so  
2022-05-18T04:09:05.0652748Z   inflating: build/lib/libshm.so     
2022-05-18T04:09:05.2268084Z   inflating: build/lib/libtorch_python.so  
2022-05-18T04:09:05.2307036Z   inflating: build/lib/libnnapi_backend.so  
2022-05-18T04:09:05.2307712Z    creating: build/bin/
2022-05-18T04:09:05.2361396Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-05-18T04:09:05.2418328Z   inflating: build/bin/c10_DeviceGuard_test  
2022-05-18T04:09:05.2473229Z   inflating: build/bin/c10_Device_test  
2022-05-18T04:09:05.2536831Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-05-18T04:09:05.2588982Z   inflating: build/bin/c10_StreamGuard_test  
2022-05-18T04:09:05.2650630Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-05-18T04:09:05.2711174Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-05-18T04:09:05.2773391Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-05-18T04:09:05.2826140Z   inflating: build/bin/c10_Array_test  
2022-05-18T04:09:05.2884571Z   inflating: build/bin/c10_Bitset_test  
2022-05-18T04:09:05.2940811Z   inflating: build/bin/c10_C++17_test  
2022-05-18T04:09:05.3013020Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-05-18T04:09:05.3067639Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-05-18T04:09:05.3122570Z   inflating: build/bin/c10_Half_test  
2022-05-18T04:09:05.3184436Z   inflating: build/bin/c10_LeftRight_test  
2022-05-18T04:09:05.3254114Z   inflating: build/bin/c10_Metaprogramming_test  
2022-05-18T04:09:05.3414313Z   inflating: build/bin/c10_SmallVectorTest  
2022-05-18T04:09:05.3469881Z   inflating: build/bin/c10_Synchronized_test  
2022-05-18T04:09:05.3533157Z   inflating: build/bin/c10_ThreadLocal_test  
2022-05-18T04:09:05.3589488Z   inflating: build/bin/c10_TypeIndex_test  
2022-05-18T04:09:05.3645335Z   inflating: build/bin/c10_TypeList_test  
2022-05-18T04:09:05.3698755Z   inflating: build/bin/c10_TypeTraits_test  
2022-05-18T04:09:05.3755681Z   inflating: build/bin/c10_accumulate_test  
2022-05-18T04:09:05.3815533Z   inflating: build/bin/c10_bfloat16_test  
2022-05-18T04:09:05.3874830Z   inflating: build/bin/c10_complex_math_test  
2022-05-18T04:09:05.3936005Z   inflating: build/bin/c10_complex_test  
2022-05-18T04:09:05.4056690Z   inflating: build/bin/c10_either_test  
2022-05-18T04:09:05.4113476Z   inflating: build/bin/c10_exception_test  
2022-05-18T04:09:05.4168569Z   inflating: build/bin/c10_flags_test  
2022-05-18T04:09:05.4355380Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-05-18T04:09:05.4410524Z   inflating: build/bin/c10_irange_test  
2022-05-18T04:09:05.4473968Z   inflating: build/bin/c10_logging_test  
2022-05-18T04:09:05.4541664Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-05-18T04:09:05.4624227Z   inflating: build/bin/c10_optional_test  
2022-05-18T04:09:05.4683281Z   inflating: build/bin/c10_registry_test  
2022-05-18T04:09:05.4749070Z   inflating: build/bin/c10_string_view_test  
2022-05-18T04:09:05.4805272Z   inflating: build/bin/c10_tempfile_test  
2022-05-18T04:09:05.4867092Z   inflating: build/bin/c10_typeid_test  
2022-05-18T04:09:05.4927851Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-05-18T04:09:05.5459865Z   inflating: build/bin/protoc-3.13.0.0  
2022-05-18T04:09:05.5995152Z   inflating: build/bin/protoc        
2022-05-18T04:09:05.6048784Z   inflating: build/bin/c10_cuda_CUDATest  
2022-05-18T04:09:05.6377117Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-05-18T04:09:05.6742475Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-05-18T04:09:05.6800489Z   inflating: build/bin/HashStoreTest  
2022-05-18T04:09:05.6860222Z   inflating: build/bin/FileStoreTest  
2022-05-18T04:09:05.6926115Z   inflating: build/bin/TCPStoreTest  
2022-05-18T04:09:05.6942239Z   inflating: build/bin/ProcessGroupMPITest  
2022-05-18T04:09:05.6944501Z   inflating: build/bin/example_allreduce  
2022-05-18T04:09:05.7003248Z   inflating: build/bin/Dimname_test  
2022-05-18T04:09:05.7066338Z   inflating: build/bin/scalar_test   
2022-05-18T04:09:05.7132545Z   inflating: build/bin/apply_utils_test  
2022-05-18T04:09:05.7197839Z   inflating: build/bin/basic         
2022-05-18T04:09:05.7262241Z   inflating: build/bin/atest         
2022-05-18T04:09:05.7325441Z   inflating: build/bin/NamedTensor_test  
2022-05-18T04:09:05.7383858Z   inflating: build/bin/broadcast_test  
2022-05-18T04:09:05.7438353Z   inflating: build/bin/wrapdim_test  
2022-05-18T04:09:05.7516812Z   inflating: build/bin/Dict_test     
2022-05-18T04:09:05.7570251Z   inflating: build/bin/dlconvertor_test  
2022-05-18T04:09:05.7631065Z   inflating: build/bin/half_test     
2022-05-18T04:09:05.7692625Z   inflating: build/bin/native_test   
2022-05-18T04:09:05.7693003Z   inflating: build/bin/verify_api_visibility  
2022-05-18T04:09:05.7751404Z   inflating: build/bin/undefined_tensor_test  
2022-05-18T04:09:05.7753391Z   inflating: build/bin/thread_init_test  
2022-05-18T04:09:05.7816443Z   inflating: build/bin/scalar_tensor_test  
2022-05-18T04:09:05.7877791Z   inflating: build/bin/test_parallel  
2022-05-18T04:09:05.7933214Z   inflating: build/bin/weakref_test  
2022-05-18T04:09:05.7986891Z   inflating: build/bin/lazy_tensor_test  
2022-05-18T04:09:05.8049179Z   inflating: build/bin/quantized_test  
2022-05-18T04:09:05.8104223Z   inflating: build/bin/operators_test  
2022-05-18T04:09:05.8165631Z   inflating: build/bin/extension_backend_test  
2022-05-18T04:09:05.8222617Z   inflating: build/bin/math_kernel_test  
2022-05-18T04:09:05.8278371Z   inflating: build/bin/memory_overlapping_test  
2022-05-18T04:09:05.8331425Z   inflating: build/bin/variant_test  
2022-05-18T04:09:05.8415860Z   inflating: build/bin/tensor_iterator_test  
2022-05-18T04:09:05.8472163Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-05-18T04:09:05.8535764Z   inflating: build/bin/cpu_generator_test  
2022-05-18T04:09:05.8590802Z   inflating: build/bin/reportMemoryUsage_test  
2022-05-18T04:09:05.8644088Z   inflating: build/bin/reduce_ops_test  
2022-05-18T04:09:05.8701308Z   inflating: build/bin/memory_format_test  
2022-05-18T04:09:05.8772762Z   inflating: build/bin/pow_test      
2022-05-18T04:09:05.8829568Z   inflating: build/bin/mobile_memory_cleanup  
2022-05-18T04:09:05.8884108Z   inflating: build/bin/dispatch_key_set_test  
2022-05-18T04:09:05.8949679Z   inflating: build/bin/IListRef_test  
2022-05-18T04:09:05.9070508Z   inflating: build/bin/List_test     
2022-05-18T04:09:05.9127087Z   inflating: build/bin/stride_properties_test  
2022-05-18T04:09:05.9202077Z   inflating: build/bin/vmap_test     
2022-05-18T04:09:05.9333637Z   inflating: build/bin/kernel_function_legacy_test  
2022-05-18T04:09:05.9438851Z   inflating: build/bin/kernel_function_test  
2022-05-18T04:09:05.9577394Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-05-18T04:09:05.9690026Z   inflating: build/bin/kernel_lambda_test  
2022-05-18T04:09:05.9755337Z   inflating: build/bin/kernel_stackbased_test  
2022-05-18T04:09:05.9859672Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-05-18T04:09:05.9914919Z   inflating: build/bin/CppSignature_test  
2022-05-18T04:09:05.9966855Z   inflating: build/bin/op_allowlist_test  
2022-05-18T04:09:06.0280998Z   inflating: build/bin/op_registration_test  
2022-05-18T04:09:06.0378015Z   inflating: build/bin/cpu_rng_test  
2022-05-18T04:09:06.0436123Z   inflating: build/bin/inline_container_test  
2022-05-18T04:09:06.0505782Z   inflating: build/bin/KernelFunction_test  
2022-05-18T04:09:06.0571863Z   inflating: build/bin/type_test     
2022-05-18T04:09:06.0636446Z   inflating: build/bin/cuda_atomic_ops_test  
2022-05-18T04:09:06.0739529Z   inflating: build/bin/ivalue_test   
2022-05-18T04:09:06.0814893Z   inflating: build/bin/cuda_complex_math_test  
2022-05-18T04:09:06.0880385Z   inflating: build/bin/cuda_complex_test  
2022-05-18T04:09:06.0937535Z   inflating: build/bin/cuda_apply_test  
2022-05-18T04:09:06.0993595Z   inflating: build/bin/cuda_integer_divider_test  
2022-05-18T04:09:06.1059997Z   inflating: build/bin/cuda_stream_test  
2022-05-18T04:09:06.1121028Z   inflating: build/bin/backend_fallback_test  
2022-05-18T04:09:06.1180652Z   inflating: build/bin/cuda_caching_host_allocator_test  
2022-05-18T04:09:06.1237927Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2022-05-18T04:09:06.1292511Z   inflating: build/bin/cuda_dlconvertor_test  
2022-05-18T04:09:06.1347092Z   inflating: build/bin/cuda_half_test  
2022-05-18T04:09:06.1403625Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2022-05-18T04:09:06.1491081Z   inflating: build/bin/cuda_cub_test  
2022-05-18T04:09:06.1543857Z   inflating: build/bin/cuda_optional_test  
2022-05-18T04:09:06.1608609Z   inflating: build/bin/cuda_distributions_test  
2022-05-18T04:09:06.1667583Z   inflating: build/bin/cuda_vectorized_test  
2022-05-18T04:09:06.1720489Z   inflating: build/bin/cuda_cudnn_test  
2022-05-18T04:09:06.1785077Z   inflating: build/bin/cuda_generator_test  
2022-05-18T04:09:06.1855863Z   inflating: build/bin/ProcessGroupGlooTest  
2022-05-18T04:09:06.1919753Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2022-05-18T04:09:06.1985641Z   inflating: build/bin/ProcessGroupNCCLTest  
2022-05-18T04:09:06.2003466Z   inflating: build/bin/tutorial_tensorexpr  
2022-05-18T04:09:06.2068455Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2022-05-18T04:09:06.2125696Z   inflating: build/bin/test_dist_autograd  
2022-05-18T04:09:06.2202824Z   inflating: build/bin/test_cpp_rpc  
2022-05-18T04:09:06.2278306Z   inflating: build/bin/test_mobile_nnc  
2022-05-18T04:09:06.2280217Z   inflating: build/bin/parallel_benchmark  
2022-05-18T04:09:06.2292117Z   inflating: build/bin/aot_model_compiler_test  
2022-05-18T04:09:06.3220134Z   inflating: build/bin/test_tensorexpr  
2022-05-18T04:09:06.3610819Z   inflating: build/bin/test_lazy     
2022-05-18T04:09:06.3616384Z   inflating: build/bin/torch_shm_manager  
2022-05-18T04:09:06.3751896Z   inflating: build/bin/nvfuser_bench  
2022-05-18T04:09:06.5073964Z   inflating: build/bin/test_api      
2022-05-18T04:09:06.6050800Z   inflating: build/bin/test_jit      
2022-05-18T04:09:06.6051465Z   inflating: .pytorch-test-times.json  
2022-05-18T04:09:06.6087116Z ##[group]Run df -H
2022-05-18T04:09:06.6087559Z df -H
2022-05-18T04:09:06.6103399Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:09:06.6103708Z env:
2022-05-18T04:09:06.6103930Z   IN_CI: 1
2022-05-18T04:09:06.6104140Z   IS_GHA: 1
2022-05-18T04:09:06.6104395Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:09:06.6104666Z   GPU_FLAG: --gpus all
2022-05-18T04:09:06.6104898Z ##[endgroup]
2022-05-18T04:09:06.6148779Z Filesystem      Size  Used Avail Use% Mounted on
2022-05-18T04:09:06.6149149Z devtmpfs        258G     0  258G   0% /dev
2022-05-18T04:09:06.6149438Z tmpfs           258G     0  258G   0% /dev/shm
2022-05-18T04:09:06.6149726Z tmpfs           258G  725k  258G   1% /run
2022-05-18T04:09:06.6150377Z tmpfs           258G     0  258G   0% /sys/fs/cgroup
2022-05-18T04:09:06.6151086Z /dev/xvda1      162G   22G  140G  14% /
2022-05-18T04:09:06.6151381Z tmpfs            52G     0   52G   0% /run/user/0
2022-05-18T04:09:06.6891331Z ##[group]Run .github/scripts/parse_ref.py
2022-05-18T04:09:06.6891715Z .github/scripts/parse_ref.py
2022-05-18T04:09:06.6906827Z shell: /usr/bin/bash -e {0}
2022-05-18T04:09:06.6907065Z env:
2022-05-18T04:09:06.6907286Z   IN_CI: 1
2022-05-18T04:09:06.6907512Z   IS_GHA: 1
2022-05-18T04:09:06.6907743Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:09:06.6908015Z   GPU_FLAG: --gpus all
2022-05-18T04:09:06.6908277Z ##[endgroup]
2022-05-18T04:09:13.2699853Z ##[group]Run set -x
2022-05-18T04:09:13.2700417Z set -x
2022-05-18T04:09:13.2700658Z 
2022-05-18T04:09:13.2700940Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-05-18T04:09:13.2701280Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-05-18T04:09:13.2701648Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-05-18T04:09:13.2702303Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-05-18T04:09:13.2702570Z else
2022-05-18T04:09:13.2702865Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-05-18T04:09:13.2703151Z fi
2022-05-18T04:09:13.2703378Z 
2022-05-18T04:09:13.2703689Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-05-18T04:09:13.2704048Z export COMMIT_MESSAGES
2022-05-18T04:09:13.2704308Z 
2022-05-18T04:09:13.2704606Z # detached container should get cleaned up by teardown_ec2_linux
2022-05-18T04:09:13.2705055Z # TODO: Stop building test binaries as part of the build phase
2022-05-18T04:09:13.2705447Z # Used for GPU_FLAG since that doesn't play nice
2022-05-18T04:09:13.2705767Z # shellcheck disable=SC2086,SC2090
2022-05-18T04:09:13.2706102Z container_name=$(docker run \
2022-05-18T04:09:13.2706399Z   ${GPU_FLAG:-} \
2022-05-18T04:09:13.2706659Z   -e BUILD_ENVIRONMENT \
2022-05-18T04:09:13.2706945Z   -e PR_NUMBER \
2022-05-18T04:09:13.2707252Z   -e CUSTOM_TEST_ARTIFACT_BUILD_DIR \
2022-05-18T04:09:13.2707538Z   -e GITHUB_ACTIONS \
2022-05-18T04:09:13.2707801Z   -e IN_CI \
2022-05-18T04:09:13.2708050Z   -e IS_GHA \
2022-05-18T04:09:13.2708284Z   -e BRANCH \
2022-05-18T04:09:13.2708539Z   -e SHA1 \
2022-05-18T04:09:13.2708800Z   -e AWS_DEFAULT_REGION \
2022-05-18T04:09:13.2709079Z   -e IN_WHEEL_TEST \
2022-05-18T04:09:13.2709334Z   -e SHARD_NUMBER \
2022-05-18T04:09:13.2709604Z   -e JOB_BASE_NAME \
2022-05-18T04:09:13.2709875Z   -e TEST_CONFIG \
2022-05-18T04:09:13.2710131Z   -e NUM_TEST_SHARDS \
2022-05-18T04:09:13.2710399Z   -e PR_BODY \
2022-05-18T04:09:13.2710674Z   -e COMMIT_MESSAGES \
2022-05-18T04:09:13.2710963Z   -e PYTORCH_RETRY_TEST_CASES \
2022-05-18T04:09:13.2711252Z   -e PR_LABELS \
2022-05-18T04:09:13.2711545Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-05-18T04:09:13.2711818Z   -e SCCACHE_BUCKET \
2022-05-18T04:09:13.2712079Z   -e XLA_CUDA \
2022-05-18T04:09:13.2712362Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-05-18T04:09:13.2712686Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-05-18T04:09:13.2713009Z   --ulimit stack=10485760:83886080 \
2022-05-18T04:09:13.2713321Z   --security-opt seccomp=unconfined \
2022-05-18T04:09:13.2713792Z   --cap-add=SYS_PTRACE \
2022-05-18T04:09:13.2714038Z   --ipc=host \
2022-05-18T04:09:13.2714294Z   --shm-size="${SHM_SIZE}" \
2022-05-18T04:09:13.2714546Z   --tty \
2022-05-18T04:09:13.2714759Z   --detach \
2022-05-18T04:09:13.2715021Z   --name="${container_name}" \
2022-05-18T04:09:13.2715287Z   --user jenkins \
2022-05-18T04:09:13.2715902Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-05-18T04:09:13.2716241Z   -w /var/lib/jenkins/workspace \
2022-05-18T04:09:13.2716526Z   "${DOCKER_IMAGE}"
2022-05-18T04:09:13.2716751Z )
2022-05-18T04:09:13.2717093Z docker exec -t "${container_name}" sh -c "pip install dist/*.whl && ${TEST_COMMAND}"
2022-05-18T04:09:13.2732121Z shell: /usr/bin/bash -e {0}
2022-05-18T04:09:13.2732367Z env:
2022-05-18T04:09:13.2732576Z   IN_CI: 1
2022-05-18T04:09:13.2732770Z   IS_GHA: 1
2022-05-18T04:09:13.2733162Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:09:13.2733527Z   GPU_FLAG: --gpus all
2022-05-18T04:09:13.2733847Z   BUILD_ENVIRONMENT: linux-bionic-cuda10.2-py3.9-gcc7
2022-05-18T04:09:13.2734159Z   PR_NUMBER: 
2022-05-18T04:09:13.2734394Z   BRANCH: master
2022-05-18T04:09:13.2734677Z   CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts
2022-05-18T04:09:13.2735018Z   SHA1: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T04:09:13.2735319Z   PYTORCH_RETRY_TEST_CASES: 1
2022-05-18T04:09:13.2735635Z   JOB_BASE_NAME: linux-bionic-cuda10.2-py3.9-gcc7-test
2022-05-18T04:09:13.2735950Z   TEST_CONFIG: multigpu
2022-05-18T04:09:13.2736194Z   SHARD_NUMBER: 1
2022-05-18T04:09:13.2736433Z   NUM_TEST_SHARDS: 1
2022-05-18T04:09:13.2736650Z   PR_BODY: 
2022-05-18T04:09:13.2736949Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-05-18T04:09:13.2737252Z   SHM_SIZE: 2g
2022-05-18T04:09:13.2737729Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:09:13.2738212Z   XLA_CUDA: 
2022-05-18T04:09:13.2738557Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-05-18T04:09:13.2738894Z ##[endgroup]
2022-05-18T04:09:13.2773200Z + [[ multigpu == \m\u\l\t\i\g\p\u ]]
2022-05-18T04:09:13.2773799Z + TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-05-18T04:09:13.2778789Z ++ git cherry -v origin/master
2022-05-18T04:09:13.2818236Z + COMMIT_MESSAGES=
2022-05-18T04:09:13.2819230Z + export COMMIT_MESSAGES
2022-05-18T04:09:13.2830471Z +++ nproc --ignore=2
2022-05-18T04:09:13.5328822Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e CUSTOM_TEST_ARTIFACT_BUILD_DIR -e GITHUB_ACTIONS -e IN_CI -e IS_GHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e JOB_BASE_NAME -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PR_LABELS -e MAX_JOBS=62 -e SCCACHE_BUCKET -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_2342799949 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:09:33.4254342Z + container_name=3bd9d49d70bf00640e887a904f809d68654d815afb203dfacb3a7962baa0db74
2022-05-18T04:09:33.4255558Z + docker exec -t 3bd9d49d70bf00640e887a904f809d68654d815afb203dfacb3a7962baa0db74 sh -c 'pip install dist/*.whl && .jenkins/pytorch/multigpu-test.sh'
2022-05-18T04:09:33.9452091Z Processing ./dist/torch-1.12.0a0+git3b23752-cp39-cp39-linux_x86_64.whl
2022-05-18T04:09:34.0465226Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.9/site-packages (from torch==1.12.0a0+git3b23752) (4.2.0)
2022-05-18T04:09:34.6014293Z Installing collected packages: torch
2022-05-18T04:09:44.3065191Z Successfully installed torch-1.12.0a0+git3b23752
2022-05-18T04:09:44.3716718Z ++++ dirname .jenkins/pytorch/common.sh
2022-05-18T04:09:44.3729236Z +++ cd .jenkins/pytorch
2022-05-18T04:09:44.3757998Z +++ pwd -P
2022-05-18T04:09:44.3758390Z ++ SCRIPT_DIR=/var/lib/jenkins/workspace/.jenkins/pytorch
2022-05-18T04:09:44.3758893Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7 == *linux* ]]
2022-05-18T04:09:44.3759704Z +++ find /etc/apt/ -type f -name '*.list'
2022-05-18T04:09:44.3760449Z ++ sudo sed -i 's/.*nvidia.*/# &/' /etc/apt/sources.list /etc/apt/sources.list.d/cuda.list /etc/apt/sources.list.d/nvidia-ml.list /etc/apt/sources.list.d/ubuntu-toolchain-r-ubuntu-test-bionic.list
2022-05-18T04:09:44.3835472Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7 == *rocm* ]]
2022-05-18T04:09:44.3836203Z ++ echo ENTERED_USER_LAND
2022-05-18T04:09:44.3836752Z ENTERED_USER_LAND
2022-05-18T04:09:44.3837196Z ++ export IN_CI=1
2022-05-18T04:09:44.3839891Z ++ IN_CI=1
2022-05-18T04:09:44.3840276Z ++ declare -f -t trap_add
2022-05-18T04:09:44.3840772Z ++ trap_add cleanup EXIT
2022-05-18T04:09:44.3841079Z ++ trap_add_cmd=cleanup
2022-05-18T04:09:44.3841313Z ++ shift
2022-05-18T04:09:44.3841560Z ++ for trap_add_name in "$@"
2022-05-18T04:09:44.3848186Z ++++ trap -p EXIT
2022-05-18T04:09:44.3850734Z +++ eval 'extract_trap_cmd '
2022-05-18T04:09:44.3851020Z ++++ extract_trap_cmd
2022-05-18T04:09:44.3851347Z ++++ printf '%s\n' ''
2022-05-18T04:09:44.3851631Z +++ printf '%s\n' cleanup
2022-05-18T04:09:44.3851929Z ++ trap -- '
2022-05-18T04:09:44.3852201Z cleanup' EXIT
2022-05-18T04:09:44.3858065Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7 != *win-* ]]
2022-05-18T04:09:44.3858399Z ++ which sccache
2022-05-18T04:09:44.3871478Z ++ sccache --stop-server
2022-05-18T04:09:44.3903970Z ++ true
2022-05-18T04:09:44.3904358Z ++ rm -f /var/lib/jenkins/sccache_error.log
2022-05-18T04:09:44.3915137Z ++ [[ -n '' ]]
2022-05-18T04:09:44.3915573Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7 == *rocm* ]]
2022-05-18T04:09:44.3915946Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
2022-05-18T04:09:44.3916278Z ++ SCCACHE_IDLE_TIMEOUT=1200
2022-05-18T04:09:44.3916597Z ++ RUST_LOG=sccache::server=error
2022-05-18T04:09:44.3916925Z ++ sccache --start-server
2022-05-18T04:09:44.3938165Z sccache: Starting the server...
2022-05-18T04:09:44.4168101Z ++ sccache --zero-stats
2022-05-18T04:09:44.4197092Z Compile requests                      0
2022-05-18T04:09:44.4197818Z Compile requests executed             0
2022-05-18T04:09:44.4198449Z Cache hits                            0
2022-05-18T04:09:44.4199035Z Cache misses                          0
2022-05-18T04:09:44.4199608Z Cache timeouts                        0
2022-05-18T04:09:44.4200207Z Cache read errors                     0
2022-05-18T04:09:44.4200483Z Forced recaches                       0
2022-05-18T04:09:44.4200776Z Cache write errors                    0
2022-05-18T04:09:44.4201080Z Compilation failures                  0
2022-05-18T04:09:44.4201340Z Cache errors                          0
2022-05-18T04:09:44.4201733Z Non-cacheable compilations            0
2022-05-18T04:09:44.4202091Z Non-cacheable calls                   0
2022-05-18T04:09:44.4202417Z Non-compilation calls                 0
2022-05-18T04:09:44.4202725Z Unsupported compiler calls            0
2022-05-18T04:09:44.4203042Z Average cache write               0.000 s
2022-05-18T04:09:44.4203322Z Average cache read miss           0.000 s
2022-05-18T04:09:44.4203605Z Average cache read hit            0.000 s
2022-05-18T04:09:44.4203913Z Failed distributed compilations       0
2022-05-18T04:09:44.4204629Z Cache location                  S3, bucket: Bucket(name=ossci-compiler-cache-circleci-v2, base_url=http://ossci-compiler-cache-circleci-v2.s3.amazonaws.com/)
2022-05-18T04:09:44.4205311Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7-test == *-build ]]
2022-05-18T04:09:44.4205799Z ++ which ccache
2022-05-18T04:09:44.4212311Z ++ '[' -z linux-bionic-cuda10.2-py3.9-gcc7 ']'
2022-05-18T04:09:44.4212835Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7 == *linux-trusty-py3.6-gcc7* ]]
2022-05-18T04:09:44.4213211Z ++ BUILD_TEST_LIBTORCH=0
2022-05-18T04:09:44.4213488Z ++ [[ multigpu == *xla* ]]
2022-05-18T04:09:44.4213869Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7 == *centos* ]]
2022-05-18T04:09:44.4214332Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7 == *linux-bionic* ]]
2022-05-18T04:09:44.4214650Z ++ which conda
2022-05-18T04:09:44.4222795Z /opt/conda/bin/conda
2022-05-18T04:09:44.4223123Z ++ conda install -q -y cmake
2022-05-18T04:09:54.0903102Z Collecting package metadata (current_repodata.json): ...working... done
2022-05-18T04:09:54.6020268Z Solving environment: ...working... done
2022-05-18T04:09:54.6860129Z 
2022-05-18T04:09:54.6860457Z ## Package Plan ##
2022-05-18T04:09:54.6860671Z 
2022-05-18T04:09:54.6861434Z   environment location: /opt/conda
2022-05-18T04:09:54.6861674Z 
2022-05-18T04:09:54.6861822Z   added / updated specs:
2022-05-18T04:09:54.6862635Z     - cmake
2022-05-18T04:09:54.6862897Z 
2022-05-18T04:09:54.6862925Z 
2022-05-18T04:09:54.6863214Z The following packages will be downloaded:
2022-05-18T04:09:54.6863921Z 
2022-05-18T04:09:54.6864110Z     package                    |            build
2022-05-18T04:09:54.6864661Z     ---------------------------|-----------------
2022-05-18T04:09:54.6865113Z     bzip2-1.0.8                |       h7b6447c_0          78 KB
2022-05-18T04:09:54.6865531Z     c-ares-1.18.1              |       h7f8727e_0         114 KB
2022-05-18T04:09:54.6865931Z     cmake-3.22.1               |       h1fce559_0         7.3 MB
2022-05-18T04:09:54.6866322Z     expat-2.4.4                |       h295c915_0         169 KB
2022-05-18T04:09:54.6866737Z     krb5-1.19.2                |       hac12032_0         1.2 MB
2022-05-18T04:09:54.6867109Z     libcurl-7.82.0             |       h0b77cf5_0         342 KB
2022-05-18T04:09:54.6867517Z     libedit-3.1.20210910       |       h7f8727e_0         166 KB
2022-05-18T04:09:54.6867904Z     libev-4.33                 |       h7f8727e_1         111 KB
2022-05-18T04:09:54.6868320Z     libnghttp2-1.46.0          |       hce63b2e_0         680 KB
2022-05-18T04:09:54.6868705Z     libssh2-1.10.0             |       h8f2d780_0         274 KB
2022-05-18T04:09:54.6869089Z     libuv-1.40.0               |       h7b6447c_0         736 KB
2022-05-18T04:09:54.6869473Z     lz4-c-1.9.3                |       h295c915_1         185 KB
2022-05-18T04:09:54.6869833Z     rhash-1.4.1                |       h3c74f83_1         203 KB
2022-05-18T04:09:54.6870271Z     zstd-1.5.2                 |       ha4553b6_0         488 KB
2022-05-18T04:09:54.6870699Z     ------------------------------------------------------------
2022-05-18T04:09:54.6871022Z                                            Total:        12.0 MB
2022-05-18T04:09:54.6871210Z 
2022-05-18T04:09:54.6871379Z The following NEW packages will be INSTALLED:
2022-05-18T04:09:54.6871585Z 
2022-05-18T04:09:54.6871952Z   bzip2              pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0
2022-05-18T04:09:54.6872468Z   c-ares             pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0
2022-05-18T04:09:54.6872958Z   cmake              pkgs/main/linux-64::cmake-3.22.1-h1fce559_0
2022-05-18T04:09:54.6873412Z   expat              pkgs/main/linux-64::expat-2.4.4-h295c915_0
2022-05-18T04:09:54.6873889Z   krb5               pkgs/main/linux-64::krb5-1.19.2-hac12032_0
2022-05-18T04:09:54.6874368Z   libcurl            pkgs/main/linux-64::libcurl-7.82.0-h0b77cf5_0
2022-05-18T04:09:54.6874859Z   libedit            pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0
2022-05-18T04:09:54.6875344Z   libev              pkgs/main/linux-64::libev-4.33-h7f8727e_1
2022-05-18T04:09:54.6875853Z   libnghttp2         pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0
2022-05-18T04:09:54.6876357Z   libssh2            pkgs/main/linux-64::libssh2-1.10.0-h8f2d780_0
2022-05-18T04:09:54.6876814Z   libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0
2022-05-18T04:09:54.6877281Z   lz4-c              pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1
2022-05-18T04:09:54.6877744Z   rhash              pkgs/main/linux-64::rhash-1.4.1-h3c74f83_1
2022-05-18T04:09:54.6878193Z   zstd               pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0
2022-05-18T04:09:54.6878404Z 
2022-05-18T04:09:54.6878695Z The following packages will be SUPERSEDED by a higher-priority channel:
2022-05-18T04:09:54.6878950Z 
2022-05-18T04:09:54.6879352Z   certifi            conda-forge::certifi-2021.10.8-py39hf~ --> pkgs/main::certifi-2021.10.8-py39h06a4308_2
2022-05-18T04:09:54.6880176Z   conda              conda-forge::conda-4.12.0-py39hf3d152~ --> pkgs/main::conda-4.12.0-py39h06a4308_0
2022-05-18T04:09:54.6880452Z 
2022-05-18T04:09:54.6880470Z 
2022-05-18T04:09:55.7375040Z Preparing transaction: ...working... done
2022-05-18T04:09:56.3294833Z Verifying transaction: ...working... done
2022-05-18T04:09:59.0743424Z Executing transaction: ...working... done
2022-05-18T04:10:00.3453025Z ++ [[ linux-bionic-cuda10.2-py3.9-gcc7 == *centos* ]]
2022-05-18T04:10:00.3453559Z + echo 'Testing pytorch (distributed only)'
2022-05-18T04:10:00.3453884Z Testing pytorch (distributed only)
2022-05-18T04:10:00.3454574Z + '[' -n 1 ']'
2022-05-18T04:10:00.3454959Z + pip_install 'unittest-xml-reporting<=3.2.0,>=2.0.0'
2022-05-18T04:10:00.3455437Z + pip install --progress-bar off 'unittest-xml-reporting<=3.2.0,>=2.0.0'
2022-05-18T04:10:00.7724921Z Requirement already satisfied: unittest-xml-reporting<=3.2.0,>=2.0.0 in /opt/conda/lib/python3.9/site-packages (3.2.0)
2022-05-18T04:10:00.7744760Z Requirement already satisfied: lxml in /opt/conda/lib/python3.9/site-packages (from unittest-xml-reporting<=3.2.0,>=2.0.0) (4.8.0)
2022-05-18T04:10:01.3239093Z + python test/run_test.py --verbose -i distributed/test_c10d_common
2022-05-18T04:10:10.9441827Z Ignoring disabled issues:  []
2022-05-18T04:10:10.9567815Z /var/lib/jenkins/workspace/test/run_test.py:894: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-05-18T04:10:10.9568386Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) == "11.6":
2022-05-18T04:10:10.9568777Z Selected tests:
2022-05-18T04:10:10.9569043Z  distributed/test_c10d_common
2022-05-18T04:10:10.9666334Z Prioritized test from test file changes.
2022-05-18T04:10:10.9666666Z reordering tests for PR:
2022-05-18T04:10:10.9666918Z prioritized: []
2022-05-18T04:10:10.9667415Z the rest: ['distributed/test_c10d_common']
2022-05-18T04:10:10.9667613Z 
2022-05-18T04:10:11.0274093Z Running distributed/test_c10d_common ... [2022-05-18 04:10:11.026845]
2022-05-18T04:10:11.0275429Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:10:11.026906]
2022-05-18T04:10:11.9952194Z test_debug_level (__main__.CommTest)
2022-05-18T04:10:11.9952972Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-05-18T04:10:11.9953783Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-05-18T04:10:11.9954536Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-05-18T04:10:11.9955351Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-05-18T04:10:11.9956197Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest)
2022-05-18T04:10:11.9957004Z test_collectives (__main__.PythonProcessGroupExtensionTest)
2022-05-18T04:10:11.9957777Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest)
2022-05-18T04:10:11.9958422Z test_send_recv (__main__.PythonProcessGroupExtensionTest)
2022-05-18T04:10:12.9379895Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:10:12.9390179Z 
2022-05-18T04:10:12.9390581Z Running tests...
2022-05-18T04:10:12.9391045Z ----------------------------------------------------------------------
2022-05-18T04:10:14.5866482Z   test_debug_level (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:14.6291176Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 669
2022-05-18T04:10:14.6399828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 670
2022-05-18T04:10:15.5705937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:15.5737567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:15.7445731Z ok (2.805s)
2022-05-18T04:10:15.7445953Z 
2022-05-18T04:10:15.7446647Z ----------------------------------------------------------------------
2022-05-18T04:10:15.7447018Z Ran 1 test in 2.806s
2022-05-18T04:10:15.7447186Z 
2022-05-18T04:10:15.7447300Z OK
2022-05-18T04:10:15.7447442Z 
2022-05-18T04:10:15.7448042Z Generating XML reports...
2022-05-18T04:10:15.7491496Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220518041012.xml
2022-05-18T04:10:16.9427969Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:10:16.9439441Z 
2022-05-18T04:10:16.9439757Z Running tests...
2022-05-18T04:10:16.9440637Z ----------------------------------------------------------------------
2022-05-18T04:10:18.5815183Z   test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:18.5986201Z ok (1.655s)
2022-05-18T04:10:18.5986392Z 
2022-05-18T04:10:18.5986881Z ----------------------------------------------------------------------
2022-05-18T04:10:18.5987241Z Ran 1 test in 1.655s
2022-05-18T04:10:18.5987406Z 
2022-05-18T04:10:18.5987513Z OK
2022-05-18T04:10:18.5987654Z 
2022-05-18T04:10:18.5990480Z Generating XML reports...
2022-05-18T04:10:18.6019517Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041016.xml
2022-05-18T04:10:19.7703925Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:10:19.7714028Z 
2022-05-18T04:10:19.7714214Z Running tests...
2022-05-18T04:10:19.7714683Z ----------------------------------------------------------------------
2022-05-18T04:10:21.4243253Z   test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:21.4415300Z ok (1.670s)
2022-05-18T04:10:21.4415489Z 
2022-05-18T04:10:21.4416027Z ----------------------------------------------------------------------
2022-05-18T04:10:21.4416369Z Ran 1 test in 1.670s
2022-05-18T04:10:21.4416547Z 
2022-05-18T04:10:21.4416643Z OK
2022-05-18T04:10:21.4416783Z 
2022-05-18T04:10:21.4416914Z Generating XML reports...
2022-05-18T04:10:21.4449324Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041019.xml
2022-05-18T04:10:22.6519833Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:10:22.6528658Z 
2022-05-18T04:10:22.6529040Z Running tests...
2022-05-18T04:10:22.6530370Z ----------------------------------------------------------------------
2022-05-18T04:10:24.3015237Z   test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:24.3191183Z ok (1.666s)
2022-05-18T04:10:24.3191398Z 
2022-05-18T04:10:24.3191806Z ----------------------------------------------------------------------
2022-05-18T04:10:24.3192369Z Ran 1 test in 1.666s
2022-05-18T04:10:24.3192553Z 
2022-05-18T04:10:24.3192658Z OK
2022-05-18T04:10:24.3192813Z 
2022-05-18T04:10:24.3192952Z Generating XML reports...
2022-05-18T04:10:24.3225091Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041022.xml
2022-05-18T04:10:25.5173537Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:10:25.5182345Z 
2022-05-18T04:10:25.5182742Z Running tests...
2022-05-18T04:10:25.5183243Z ----------------------------------------------------------------------
2022-05-18T04:10:27.1395877Z   test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:27.1567807Z ok (1.638s)
2022-05-18T04:10:27.1567969Z 
2022-05-18T04:10:27.1568674Z ----------------------------------------------------------------------
2022-05-18T04:10:27.1569832Z Ran 1 test in 1.639s
2022-05-18T04:10:27.1570164Z 
2022-05-18T04:10:27.1570499Z OK
2022-05-18T04:10:27.1570662Z 
2022-05-18T04:10:27.1570803Z Generating XML reports...
2022-05-18T04:10:27.1601416Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041025.xml
2022-05-18T04:10:28.3362072Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:10:28.3371850Z 
2022-05-18T04:10:28.3372428Z Running tests...
2022-05-18T04:10:28.3373019Z ----------------------------------------------------------------------
2022-05-18T04:10:30.0106936Z   test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:30.0513524Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1132
2022-05-18T04:10:30.0623298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1133
2022-05-18T04:10:30.0730728Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1134
2022-05-18T04:10:30.0831431Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1135
2022-05-18T04:10:31.0577365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:31.0606117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:31.0797218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:31.0825401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:31.2880700Z ok (2.951s)
2022-05-18T04:10:31.2881164Z 
2022-05-18T04:10:31.2881747Z ----------------------------------------------------------------------
2022-05-18T04:10:31.2882186Z Ran 1 test in 2.951s
2022-05-18T04:10:31.2882357Z 
2022-05-18T04:10:31.2882459Z OK
2022-05-18T04:10:31.2882581Z 
2022-05-18T04:10:31.2882718Z Generating XML reports...
2022-05-18T04:10:31.2936003Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041028.xml
2022-05-18T04:10:32.5563827Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:10:32.5574737Z 
2022-05-18T04:10:32.5575079Z Running tests...
2022-05-18T04:10:32.5575554Z ----------------------------------------------------------------------
2022-05-18T04:10:34.2076774Z   test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:34.2504108Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1463
2022-05-18T04:10:34.2616812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1464
2022-05-18T04:10:34.2726712Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1465
2022-05-18T04:10:34.2830366Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1466
2022-05-18T04:10:35.2839428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:35.2846375Z [W socket.cpp:558] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-05-18T04:10:35.2848071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:35.2928283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:35.2938126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:35.2951568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:35.2962160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:36.2853304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:36.2955128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:36.2955913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:36.2956896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:36.3030361Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:36.3053353Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:38.0942364Z ok (5.536s)
2022-05-18T04:10:38.0942749Z 
2022-05-18T04:10:38.0943911Z ----------------------------------------------------------------------
2022-05-18T04:10:38.0944604Z Ran 1 test in 5.537s
2022-05-18T04:10:38.0944896Z 
2022-05-18T04:10:38.0945045Z OK
2022-05-18T04:10:38.0945291Z 
2022-05-18T04:10:38.0945500Z Generating XML reports...
2022-05-18T04:10:38.0988552Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041032.xml
2022-05-18T04:10:39.3594915Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:10:39.3605306Z 
2022-05-18T04:10:39.3605715Z Running tests...
2022-05-18T04:10:39.3606188Z ----------------------------------------------------------------------
2022-05-18T04:10:41.0041548Z   test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:41.0474386Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1803
2022-05-18T04:10:41.0585817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1804
2022-05-18T04:10:41.0690607Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1805
2022-05-18T04:10:41.0803742Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1806
2022-05-18T04:10:41.9950794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:41.9951368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:42.0361517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:42.0564109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:42.1850480Z ok (2.824s)
2022-05-18T04:10:42.1850715Z 
2022-05-18T04:10:42.1851142Z ----------------------------------------------------------------------
2022-05-18T04:10:42.1851503Z Ran 1 test in 2.825s
2022-05-18T04:10:42.1851647Z 
2022-05-18T04:10:42.1851743Z OK
2022-05-18T04:10:42.1851897Z 
2022-05-18T04:10:42.1852037Z Generating XML reports...
2022-05-18T04:10:42.1896526Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041039.xml
2022-05-18T04:10:43.3888556Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:10:43.3898296Z 
2022-05-18T04:10:43.3898696Z Running tests...
2022-05-18T04:10:43.3899169Z ----------------------------------------------------------------------
2022-05-18T04:10:45.0588519Z   test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:10:45.1028213Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2134
2022-05-18T04:10:45.1142761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2135
2022-05-18T04:10:45.1246218Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2136
2022-05-18T04:10:45.1349902Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2137
2022-05-18T04:10:46.0663336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:46.0669076Z [W socket.cpp:558] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-05-18T04:10:46.0685866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:46.0692024Z [W socket.cpp:558] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-05-18T04:10:46.0738285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:46.0744651Z [W socket.cpp:558] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-05-18T04:10:46.0773193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:47.0679382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:47.0700388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:47.0753621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:47.0789884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:47.0790684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:47.0804177Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:47.0857457Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:47.0885830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:48.9461933Z ok (5.556s)
2022-05-18T04:10:48.9462522Z 
2022-05-18T04:10:48.9462949Z ----------------------------------------------------------------------
2022-05-18T04:10:48.9463315Z Ran 1 test in 5.556s
2022-05-18T04:10:48.9463548Z 
2022-05-18T04:10:48.9463624Z OK
2022-05-18T04:10:48.9463858Z 
2022-05-18T04:10:48.9463994Z Generating XML reports...
2022-05-18T04:10:48.9508930Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041043.xml
2022-05-18T04:10:50.8531594Z 
2022-05-18T04:10:50.8531982Z real	0m49.529s
2022-05-18T04:10:50.8532305Z user	1m12.983s
2022-05-18T04:10:50.8532548Z sys	1m46.026s
2022-05-18T04:10:50.8533105Z + python test/run_test.py --verbose -i distributed/test_c10d_gloo
2022-05-18T04:11:00.4923610Z Ignoring disabled issues:  []
2022-05-18T04:11:00.5050545Z /var/lib/jenkins/workspace/test/run_test.py:894: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-05-18T04:11:00.5051104Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) == "11.6":
2022-05-18T04:11:00.5051470Z Selected tests:
2022-05-18T04:11:00.5051737Z  distributed/test_c10d_gloo
2022-05-18T04:11:00.5146895Z Prioritized test from test file changes.
2022-05-18T04:11:00.5147206Z reordering tests for PR:
2022-05-18T04:11:00.5147487Z prioritized: []
2022-05-18T04:11:00.5147991Z the rest: ['distributed/test_c10d_gloo']
2022-05-18T04:11:00.5148187Z 
2022-05-18T04:11:00.5157037Z Running distributed/test_c10d_gloo ... [2022-05-18 04:11:00.515246]
2022-05-18T04:11:00.5157772Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:11:00.515319]
2022-05-18T04:11:01.4781436Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cpu>, <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>]>
2022-05-18T04:11:01.4783240Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest)
2022-05-18T04:11:01.4783647Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest)
2022-05-18T04:11:01.4784006Z test_gloo_barrier_device_ids (__main__.CommTest)
2022-05-18T04:11:01.4784325Z test_gloo_warn_not_in_group (__main__.CommTest)
2022-05-18T04:11:01.4784692Z test_sequence_num_incremented_gloo_default (__main__.CommTest)
2022-05-18T04:11:01.4785237Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest)
2022-05-18T04:11:01.4786122Z test_sequence_num_set_default_pg_gloo (__main__.CommTest)
2022-05-18T04:11:01.4786740Z test_sequence_num_set_gloo_new_group (__main__.CommTest)
2022-05-18T04:11:01.4796186Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-05-18T04:11:01.4802122Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4802603Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4803114Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4803693Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4804205Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4804738Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4805250Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4805740Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4806203Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4806703Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4807213Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4807709Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4808240Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4808726Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4809194Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4809639Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4810095Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4810541Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4810968Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4811457Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4811936Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4812468Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4812968Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4813442Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4813955Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4814422Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4814856Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4815268Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4815713Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4816143Z test_ignored_output (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4816567Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4817015Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4817438Z test_save_load_checkpoint (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4817835Z test_sparse_gradients (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4818328Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4818771Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4819213Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-05-18T04:11:01.4819576Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:11:01.4825838Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooTest testMethod=test_allgather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]>
2022-05-18T04:11:01.4831017Z test_allgather_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4831407Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4831855Z test_allgather_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4832246Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4832630Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4833042Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4833434Z test_allgather_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4833792Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4834197Z test_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4834624Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4835063Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4835499Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4835890Z test_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4836274Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4836656Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4837051Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4837503Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4837894Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4838279Z test_allreduce_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4838655Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4839017Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4839397Z test_broadcast_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4839776Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4840147Z test_broadcast_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4840491Z test_broadcast_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4840871Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4841239Z test_empty_tensors (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4841577Z test_gather_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4841943Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4842427Z test_gather_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4842795Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4843180Z test_gather_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4843545Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4843934Z test_multi_device_constructor (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4844292Z test_reduce_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4844658Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4845022Z test_reduce_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4845365Z test_reduce_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4845727Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4846090Z test_round_robin (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4846446Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4846824Z test_scatter_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4847191Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4847538Z test_scatter_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4847898Z test_scatter_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4848269Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4848644Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4849009Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4849413Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4850023Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:11:01.4850887Z <unittest.suite.TestSuite tests=[<__main__.ReducerTest testMethod=test_forward_backward>, <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]>
2022-05-18T04:11:01.4851702Z test_forward_backward (__main__.ReducerTest)
2022-05-18T04:11:01.4852050Z test_forward_backward_optimizer (__main__.ReducerTest)
2022-05-18T04:11:01.4852560Z test_forward_backward_unused_parameters (__main__.ReducerTest)
2022-05-18T04:11:01.4852931Z test_multi_dtype_multi_bucket (__main__.ReducerTest)
2022-05-18T04:11:01.4853262Z test_multi_dtype_single_bucket (__main__.ReducerTest)
2022-05-18T04:11:01.4853614Z test_single_dtype_single_bucket (__main__.ReducerTest)
2022-05-18T04:11:01.4854048Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_logging_init>]>
2022-05-18T04:11:01.4854444Z test_logging_init (__main__.RendezvousEnvTest)
2022-05-18T04:11:01.4854771Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:11:01.4855195Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_gloo>]>
2022-05-18T04:11:01.4855604Z test_default_store_timeout_gloo (__main__.TimeoutTest)
2022-05-18T04:11:02.4590989Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:11:02.4600737Z 
2022-05-18T04:11:02.4601126Z Running tests...
2022-05-18T04:11:02.4601766Z ----------------------------------------------------------------------
2022-05-18T04:11:04.1191404Z   test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:04.1610851Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2609
2022-05-18T04:11:04.1720052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2610
2022-05-18T04:11:05.1315629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:05.1589437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:05.3764760Z ok (2.916s)
2022-05-18T04:11:05.3765005Z 
2022-05-18T04:11:05.3765447Z ----------------------------------------------------------------------
2022-05-18T04:11:05.3765814Z Ran 1 test in 2.916s
2022-05-18T04:11:05.3765974Z 
2022-05-18T04:11:05.3766078Z OK
2022-05-18T04:11:05.3766228Z 
2022-05-18T04:11:05.3766376Z Generating XML reports...
2022-05-18T04:11:05.3808162Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041102.xml
2022-05-18T04:11:06.6053011Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:11:06.6064928Z 
2022-05-18T04:11:06.6065426Z Running tests...
2022-05-18T04:11:06.6066022Z ----------------------------------------------------------------------
2022-05-18T04:11:08.2886041Z   test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:08.3316956Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2814
2022-05-18T04:11:08.3431242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2815
2022-05-18T04:11:09.3102706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:09.3119806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:11.0512992Z ok (4.444s)
2022-05-18T04:11:11.0513223Z 
2022-05-18T04:11:11.0513672Z ----------------------------------------------------------------------
2022-05-18T04:11:11.0514023Z Ran 1 test in 4.445s
2022-05-18T04:11:11.0514202Z 
2022-05-18T04:11:11.0514308Z OK
2022-05-18T04:11:11.0514452Z 
2022-05-18T04:11:11.0514973Z Generating XML reports...
2022-05-18T04:11:11.0556997Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041106.xml
2022-05-18T04:11:12.2998698Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:11:12.3010340Z 
2022-05-18T04:11:12.3011004Z Running tests...
2022-05-18T04:11:12.3011487Z ----------------------------------------------------------------------
2022-05-18T04:11:13.9576853Z   test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:13.9997037Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3021
2022-05-18T04:11:14.0102655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3022
2022-05-18T04:11:14.9775358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:14.9970469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:15.0080239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:15.0080790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:15.0081662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:15.0082362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:15.2148214Z ok (2.913s)
2022-05-18T04:11:15.2148466Z 
2022-05-18T04:11:15.2152563Z ----------------------------------------------------------------------
2022-05-18T04:11:15.2152924Z Ran 1 test in 2.914s
2022-05-18T04:11:15.2153114Z 
2022-05-18T04:11:15.2153217Z OK
2022-05-18T04:11:15.2153434Z 
2022-05-18T04:11:15.2153581Z Generating XML reports...
2022-05-18T04:11:15.2192835Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041112.xml
2022-05-18T04:11:16.5031710Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:11:16.5042089Z 
2022-05-18T04:11:16.5042675Z Running tests...
2022-05-18T04:11:16.5043168Z ----------------------------------------------------------------------
2022-05-18T04:11:18.1738871Z   test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:18.2160035Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3226
2022-05-18T04:11:18.2274579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3227
2022-05-18T04:11:19.2139410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:19.2404703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:19.2552809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:19.2553383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:19.2554509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:19.2555241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:19.2557964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:11:19.2658671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:11:19.2659437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:11:19.2662543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:11:20.9354467Z ok (4.431s)
2022-05-18T04:11:20.9354703Z 
2022-05-18T04:11:20.9355165Z ----------------------------------------------------------------------
2022-05-18T04:11:20.9355519Z Ran 1 test in 4.431s
2022-05-18T04:11:20.9355667Z 
2022-05-18T04:11:20.9355773Z OK
2022-05-18T04:11:20.9355919Z 
2022-05-18T04:11:20.9356062Z Generating XML reports...
2022-05-18T04:11:20.9398463Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041116.xml
2022-05-18T04:11:22.2476604Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:11:22.2487246Z 
2022-05-18T04:11:22.2487885Z Running tests...
2022-05-18T04:11:22.2488405Z ----------------------------------------------------------------------
2022-05-18T04:11:23.9129839Z   test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:23.9557525Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3436
2022-05-18T04:11:23.9671441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3437
2022-05-18T04:11:24.9427865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:24.9905347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:25.0059725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:25.0060266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:25.0061157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:25.0062151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:25.0271415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:11:25.0271975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:11:25.0272703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:11:25.0273397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:11:26.6751742Z ok (4.426s)
2022-05-18T04:11:26.6751983Z 
2022-05-18T04:11:26.6752396Z ----------------------------------------------------------------------
2022-05-18T04:11:26.6752778Z Ran 1 test in 4.427s
2022-05-18T04:11:26.6752953Z 
2022-05-18T04:11:26.6753064Z OK
2022-05-18T04:11:26.6753210Z 
2022-05-18T04:11:26.6753362Z Generating XML reports...
2022-05-18T04:11:26.6796797Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041122.xml
2022-05-18T04:11:27.9283015Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:11:27.9291812Z 
2022-05-18T04:11:27.9292124Z Running tests...
2022-05-18T04:11:27.9292728Z ----------------------------------------------------------------------
2022-05-18T04:11:29.5420519Z   test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:29.5831263Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3649
2022-05-18T04:11:29.5940461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3650
2022-05-18T04:11:30.5243254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:30.5635396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:30.6984439Z ok (2.769s)
2022-05-18T04:11:30.6984648Z 
2022-05-18T04:11:30.6985078Z ----------------------------------------------------------------------
2022-05-18T04:11:30.6985865Z Ran 1 test in 2.769s
2022-05-18T04:11:30.6986037Z 
2022-05-18T04:11:30.6986136Z OK
2022-05-18T04:11:30.6986276Z 
2022-05-18T04:11:30.6986394Z Generating XML reports...
2022-05-18T04:11:30.7029192Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041127.xml
2022-05-18T04:11:31.9503731Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:11:31.9513634Z 
2022-05-18T04:11:31.9513955Z Running tests...
2022-05-18T04:11:31.9514421Z ----------------------------------------------------------------------
2022-05-18T04:11:33.6216382Z   test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:33.6639792Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3848
2022-05-18T04:11:33.6743825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3849
2022-05-18T04:11:34.6214741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:34.6215311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:34.6324737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:34.6325286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:34.6326147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:34.6326937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:34.8791887Z ok (2.928s)
2022-05-18T04:11:34.8792076Z 
2022-05-18T04:11:34.8792405Z ----------------------------------------------------------------------
2022-05-18T04:11:34.8792765Z Ran 1 test in 2.928s
2022-05-18T04:11:34.8792914Z 
2022-05-18T04:11:34.8793024Z OK
2022-05-18T04:11:34.8793163Z 
2022-05-18T04:11:34.8793298Z Generating XML reports...
2022-05-18T04:11:34.8839896Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041131.xml
2022-05-18T04:11:36.1489934Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:11:36.1500596Z 
2022-05-18T04:11:36.1500790Z Running tests...
2022-05-18T04:11:36.1501295Z ----------------------------------------------------------------------
2022-05-18T04:11:37.8018290Z   test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:37.8452090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4053
2022-05-18T04:11:37.8571313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4054
2022-05-18T04:11:38.8165179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:38.8454818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:38.8577597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:38.8578152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:38.8579011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:38.8579704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:11:38.8791212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:11:38.8791760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:11:38.8792489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:11:38.8793490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:11:39.0615856Z ok (2.911s)
2022-05-18T04:11:39.0616098Z 
2022-05-18T04:11:39.0616544Z ----------------------------------------------------------------------
2022-05-18T04:11:39.0616924Z Ran 1 test in 2.912s
2022-05-18T04:11:39.0617068Z 
2022-05-18T04:11:39.0617179Z OK
2022-05-18T04:11:39.0617320Z 
2022-05-18T04:11:39.0617470Z Generating XML reports...
2022-05-18T04:11:39.0660512Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041136.xml
2022-05-18T04:11:40.2983689Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:11:40.2994763Z 
2022-05-18T04:11:40.2995228Z Running tests...
2022-05-18T04:11:40.2995711Z ----------------------------------------------------------------------
2022-05-18T04:11:40.3004236Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-05-18T04:11:41.9629582Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:42.0040166Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4264
2022-05-18T04:11:42.0154645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4265
2022-05-18T04:11:43.0194083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:43.0203442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:44.3644385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpom__k65i
2022-05-18T04:11:44.3645032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpom__k65i/_remote_module_non_scriptable.py
2022-05-18T04:11:44.3670104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11vdnfl_
2022-05-18T04:11:44.3670748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11vdnfl_/_remote_module_non_scriptable.py
2022-05-18T04:11:44.9242094Z ok (4.624s)
2022-05-18T04:11:44.9242336Z 
2022-05-18T04:11:44.9242784Z ----------------------------------------------------------------------
2022-05-18T04:11:44.9243141Z Ran 1 test in 4.625s
2022-05-18T04:11:44.9243315Z 
2022-05-18T04:11:44.9243391Z OK
2022-05-18T04:11:44.9243535Z 
2022-05-18T04:11:44.9243684Z Generating XML reports...
2022-05-18T04:11:44.9287975Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041140.xml
2022-05-18T04:11:46.1945370Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:11:46.1956345Z 
2022-05-18T04:11:46.1956745Z Running tests...
2022-05-18T04:11:46.1957282Z ----------------------------------------------------------------------
2022-05-18T04:11:46.1965161Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:11:47.8797124Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:47.9225343Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4479
2022-05-18T04:11:47.9342793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4480
2022-05-18T04:11:48.8849463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:48.8868791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:50.2264678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqio4q74
2022-05-18T04:11:50.2265306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqio4q74/_remote_module_non_scriptable.py
2022-05-18T04:11:50.2304962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3xr_t4m
2022-05-18T04:11:50.2305850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3xr_t4m/_remote_module_non_scriptable.py
2022-05-18T04:11:50.8428111Z ok (4.647s)
2022-05-18T04:11:50.8428287Z 
2022-05-18T04:11:50.8428718Z ----------------------------------------------------------------------
2022-05-18T04:11:50.8429069Z Ran 1 test in 4.647s
2022-05-18T04:11:50.8429240Z 
2022-05-18T04:11:50.8431117Z OK
2022-05-18T04:11:50.8431307Z 
2022-05-18T04:11:50.8431429Z Generating XML reports...
2022-05-18T04:11:50.8476596Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041146.xml
2022-05-18T04:11:52.1037398Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:11:52.1045879Z 
2022-05-18T04:11:52.1046340Z Running tests...
2022-05-18T04:11:52.1046861Z ----------------------------------------------------------------------
2022-05-18T04:11:52.1056956Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:11:53.7838459Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:53.8272649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4694
2022-05-18T04:11:53.8385259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4695
2022-05-18T04:11:54.8178651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:54.8440982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:56.1743458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu73l81qf
2022-05-18T04:11:56.1744094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu73l81qf/_remote_module_non_scriptable.py
2022-05-18T04:11:56.1897926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvz41fc2d
2022-05-18T04:11:56.1898916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvz41fc2d/_remote_module_non_scriptable.py
2022-05-18T04:11:56.3923199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.3923782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.4275325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.4275840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.4463728Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:11:56.4464541Z   warnings.warn(
2022-05-18T04:11:56.4465635Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:11:56.4466383Z   warnings.warn(
2022-05-18T04:11:56.4593772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.4594297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.4856176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.4856716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.5214780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.5215672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.5523715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.5524257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:11:56.9476239Z ok (4.843s)
2022-05-18T04:11:56.9476445Z 
2022-05-18T04:11:56.9476982Z ----------------------------------------------------------------------
2022-05-18T04:11:56.9477321Z Ran 1 test in 4.843s
2022-05-18T04:11:56.9477494Z 
2022-05-18T04:11:56.9477597Z OK
2022-05-18T04:11:56.9477953Z 
2022-05-18T04:11:56.9478115Z Generating XML reports...
2022-05-18T04:11:56.9521909Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041152.xml
2022-05-18T04:11:58.2326858Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:11:58.2336990Z 
2022-05-18T04:11:58.2337320Z Running tests...
2022-05-18T04:11:58.2337813Z ----------------------------------------------------------------------
2022-05-18T04:11:58.2348269Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:11:59.9171379Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:11:59.9600749Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4909
2022-05-18T04:11:59.9718355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4910
2022-05-18T04:12:00.8984738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:00.9341335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:02.2824905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3muvu96a
2022-05-18T04:12:02.2842725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3muvu96a/_remote_module_non_scriptable.py
2022-05-18T04:12:02.2843349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnojhyirh
2022-05-18T04:12:02.2843908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnojhyirh/_remote_module_non_scriptable.py
2022-05-18T04:12:02.4926281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.4926859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.5297483Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.5298046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.5485484Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:12:02.5486433Z   warnings.warn(
2022-05-18T04:12:02.5487491Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:12:02.5488235Z   warnings.warn(
2022-05-18T04:12:02.5614999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.5615530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.5868669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.5869511Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.6239954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.6240485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.6554460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.6554969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:02.9809060Z ok (4.747s)
2022-05-18T04:12:02.9809678Z 
2022-05-18T04:12:02.9810149Z ----------------------------------------------------------------------
2022-05-18T04:12:02.9810515Z Ran 1 test in 4.747s
2022-05-18T04:12:02.9810661Z 
2022-05-18T04:12:02.9810767Z OK
2022-05-18T04:12:02.9810912Z 
2022-05-18T04:12:02.9811059Z Generating XML reports...
2022-05-18T04:12:02.9855875Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041158.xml
2022-05-18T04:12:04.2940308Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:12:04.2950808Z 
2022-05-18T04:12:04.2951188Z Running tests...
2022-05-18T04:12:04.2951684Z ----------------------------------------------------------------------
2022-05-18T04:12:04.2959134Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:12:05.9410204Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:05.9844751Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5124
2022-05-18T04:12:05.9959556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5125
2022-05-18T04:12:06.9436674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:06.9828308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:08.3397707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpij2femc2
2022-05-18T04:12:08.3398372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpij2femc2/_remote_module_non_scriptable.py
2022-05-18T04:12:08.3647387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_vk5qfan
2022-05-18T04:12:08.3647965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_vk5qfan/_remote_module_non_scriptable.py
2022-05-18T04:12:08.5824850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:08.5825424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:08.6168715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:08.6169285Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:08.9048434Z ok (4.609s)
2022-05-18T04:12:08.9048683Z 
2022-05-18T04:12:08.9049123Z ----------------------------------------------------------------------
2022-05-18T04:12:08.9049496Z Ran 1 test in 4.610s
2022-05-18T04:12:08.9049647Z 
2022-05-18T04:12:08.9049746Z OK
2022-05-18T04:12:08.9049881Z 
2022-05-18T04:12:08.9050021Z Generating XML reports...
2022-05-18T04:12:08.9097484Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041204.xml
2022-05-18T04:12:10.1623095Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:12:10.1634823Z 
2022-05-18T04:12:10.1635297Z Running tests...
2022-05-18T04:12:10.1636194Z ----------------------------------------------------------------------
2022-05-18T04:12:10.1645050Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:12:11.8151802Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:11.8584496Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5339
2022-05-18T04:12:11.8702291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5340
2022-05-18T04:12:12.8555363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:12.8706596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:14.2253660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6dce20k2
2022-05-18T04:12:14.2254348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6dce20k2/_remote_module_non_scriptable.py
2022-05-18T04:12:14.2466155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphngl48yt
2022-05-18T04:12:14.2466964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphngl48yt/_remote_module_non_scriptable.py
2022-05-18T04:12:14.4656685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:14.4657286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:14.5031210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:14.5031765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:14.8787432Z ok (4.715s)
2022-05-18T04:12:14.8787733Z 
2022-05-18T04:12:14.8788176Z ----------------------------------------------------------------------
2022-05-18T04:12:14.8788551Z Ran 1 test in 4.715s
2022-05-18T04:12:14.8788723Z 
2022-05-18T04:12:14.8788824Z OK
2022-05-18T04:12:14.8788966Z 
2022-05-18T04:12:14.8789086Z Generating XML reports...
2022-05-18T04:12:14.8832468Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041210.xml
2022-05-18T04:12:16.1446497Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:12:16.1457197Z 
2022-05-18T04:12:16.1457596Z Running tests...
2022-05-18T04:12:16.1458087Z ----------------------------------------------------------------------
2022-05-18T04:12:16.1470848Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:12:17.8048239Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:17.8463411Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5554
2022-05-18T04:12:17.8577776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5555
2022-05-18T04:12:18.8365820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:18.8464599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:20.1824766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3tlexui
2022-05-18T04:12:20.1825393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3tlexui/_remote_module_non_scriptable.py
2022-05-18T04:12:20.2060063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68_zqy50
2022-05-18T04:12:20.2060653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68_zqy50/_remote_module_non_scriptable.py
2022-05-18T04:12:20.4174819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:20.4175442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:20.4466237Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:12:20.4468369Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:12:20.4879051Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:20.4879577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:20.8665742Z ok (4.720s)
2022-05-18T04:12:20.8665970Z 
2022-05-18T04:12:20.8666394Z ----------------------------------------------------------------------
2022-05-18T04:12:20.8666742Z Ran 1 test in 4.721s
2022-05-18T04:12:20.8666909Z 
2022-05-18T04:12:20.8666986Z OK
2022-05-18T04:12:20.8667120Z 
2022-05-18T04:12:20.8667258Z Generating XML reports...
2022-05-18T04:12:20.8712286Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041216.xml
2022-05-18T04:12:22.1324336Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:12:22.1333125Z 
2022-05-18T04:12:22.1333544Z Running tests...
2022-05-18T04:12:22.1335079Z ----------------------------------------------------------------------
2022-05-18T04:12:22.1345869Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:12:23.7964898Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:23.8402510Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5769
2022-05-18T04:12:23.8519794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5770
2022-05-18T04:12:24.8390380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:24.8659790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:26.2038634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikylj4yg
2022-05-18T04:12:26.2039282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikylj4yg/_remote_module_non_scriptable.py
2022-05-18T04:12:26.2363978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7dd39g0
2022-05-18T04:12:26.2364578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7dd39g0/_remote_module_non_scriptable.py
2022-05-18T04:12:26.8609535Z ok (4.727s)
2022-05-18T04:12:26.8609761Z 
2022-05-18T04:12:26.8610479Z ----------------------------------------------------------------------
2022-05-18T04:12:26.8610856Z Ran 1 test in 4.728s
2022-05-18T04:12:26.8611027Z 
2022-05-18T04:12:26.8611122Z OK
2022-05-18T04:12:26.8611248Z 
2022-05-18T04:12:26.8611380Z Generating XML reports...
2022-05-18T04:12:26.8658499Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041222.xml
2022-05-18T04:12:28.1280967Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:12:28.1289789Z 
2022-05-18T04:12:28.1290202Z Running tests...
2022-05-18T04:12:28.1290697Z ----------------------------------------------------------------------
2022-05-18T04:12:28.1298389Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:12:29.7610334Z Checkpointing should work with static graph in the case of checkpointing ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:29.8036766Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5984
2022-05-18T04:12:29.8139462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5985
2022-05-18T04:12:30.7146816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:30.7635388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:32.1069011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm83j3taj
2022-05-18T04:12:32.1069940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm83j3taj/_remote_module_non_scriptable.py
2022-05-18T04:12:32.1319512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9d9bn1b
2022-05-18T04:12:32.1320105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9d9bn1b/_remote_module_non_scriptable.py
2022-05-18T04:12:32.3524719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:32.3525309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:32.3885124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:32.3885643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:32.7224583Z ok (4.593s)
2022-05-18T04:12:32.7224928Z 
2022-05-18T04:12:32.7225662Z ----------------------------------------------------------------------
2022-05-18T04:12:32.7226153Z Ran 1 test in 4.593s
2022-05-18T04:12:32.7226326Z 
2022-05-18T04:12:32.7226410Z OK
2022-05-18T04:12:32.7226552Z 
2022-05-18T04:12:32.7227972Z Generating XML reports...
2022-05-18T04:12:32.7273387Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041228.xml
2022-05-18T04:12:33.9960782Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:12:33.9971259Z 
2022-05-18T04:12:33.9971675Z Running tests...
2022-05-18T04:12:33.9972167Z ----------------------------------------------------------------------
2022-05-18T04:12:33.9984158Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:12:35.6506330Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:35.6948102Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6199
2022-05-18T04:12:35.7069055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6200
2022-05-18T04:12:36.6567860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:36.6835383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:38.0406935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplw4vpw_x
2022-05-18T04:12:38.0407574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplw4vpw_x/_remote_module_non_scriptable.py
2022-05-18T04:12:38.0645187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnrvuwzi
2022-05-18T04:12:38.0645783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnrvuwzi/_remote_module_non_scriptable.py
2022-05-18T04:12:38.2648300Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:12:38.2657170Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:12:38.2971710Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:12:38.2972540Z   warnings.warn(
2022-05-18T04:12:38.2973640Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:12:38.2974384Z   warnings.warn(
2022-05-18T04:12:38.3099038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:38.3099599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:38.3703135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:38.3703676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:38.7155945Z ok (4.718s)
2022-05-18T04:12:38.7156231Z 
2022-05-18T04:12:38.7156679Z ----------------------------------------------------------------------
2022-05-18T04:12:38.7157041Z Ran 1 test in 4.718s
2022-05-18T04:12:38.7157203Z 
2022-05-18T04:12:38.7157303Z OK
2022-05-18T04:12:38.7157440Z 
2022-05-18T04:12:38.7157587Z Generating XML reports...
2022-05-18T04:12:38.7201065Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041233.xml
2022-05-18T04:12:39.9993984Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:12:40.0004056Z 
2022-05-18T04:12:40.0004327Z Running tests...
2022-05-18T04:12:40.0004842Z ----------------------------------------------------------------------
2022-05-18T04:12:40.0016579Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:12:41.6747268Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:41.7186080Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6414
2022-05-18T04:12:41.7305394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6415
2022-05-18T04:12:42.6739282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:42.6767627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:44.0295468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfvd1nrww
2022-05-18T04:12:44.0296196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfvd1nrww/_remote_module_non_scriptable.py
2022-05-18T04:12:44.0328805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0gzhq6d
2022-05-18T04:12:44.0329410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0gzhq6d/_remote_module_non_scriptable.py
2022-05-18T04:12:44.2473835Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:12:44.2475117Z   warnings.warn(
2022-05-18T04:12:44.2476323Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:12:44.2477081Z   warnings.warn(
2022-05-18T04:12:44.2613297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:44.2613868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:44.3069541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:44.3070087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:44.6391307Z ok (4.638s)
2022-05-18T04:12:44.6391535Z 
2022-05-18T04:12:44.6391982Z ----------------------------------------------------------------------
2022-05-18T04:12:44.6392328Z Ran 1 test in 4.639s
2022-05-18T04:12:44.6392494Z 
2022-05-18T04:12:44.6392591Z OK
2022-05-18T04:12:44.6392731Z 
2022-05-18T04:12:44.6392851Z Generating XML reports...
2022-05-18T04:12:44.6438866Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041239.xml
2022-05-18T04:12:45.9261486Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:12:45.9272324Z 
2022-05-18T04:12:45.9272695Z Running tests...
2022-05-18T04:12:45.9273186Z ----------------------------------------------------------------------
2022-05-18T04:12:45.9285990Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:12:47.5907680Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:47.6345600Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6629
2022-05-18T04:12:47.6464687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6630
2022-05-18T04:12:48.5882702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:48.6263453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:49.9601502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwtsqhvq
2022-05-18T04:12:49.9602130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwtsqhvq/_remote_module_non_scriptable.py
2022-05-18T04:12:50.0121072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp41cxh4vi
2022-05-18T04:12:50.0121730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp41cxh4vi/_remote_module_non_scriptable.py
2022-05-18T04:12:50.2218751Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:50.2219359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:50.2625136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:50.2625695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:50.6555467Z ok (4.728s)
2022-05-18T04:12:50.6555725Z 
2022-05-18T04:12:50.6556178Z ----------------------------------------------------------------------
2022-05-18T04:12:50.6556537Z Ran 1 test in 4.728s
2022-05-18T04:12:50.6556712Z 
2022-05-18T04:12:50.6557204Z OK
2022-05-18T04:12:50.6557352Z 
2022-05-18T04:12:50.6557507Z Generating XML reports...
2022-05-18T04:12:50.6602250Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041245.xml
2022-05-18T04:12:51.9441193Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:12:51.9452033Z 
2022-05-18T04:12:51.9452418Z Running tests...
2022-05-18T04:12:51.9453227Z ----------------------------------------------------------------------
2022-05-18T04:12:51.9465050Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:12:53.6016831Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:53.6448761Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6844
2022-05-18T04:12:53.6554559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6845
2022-05-18T04:12:54.5568250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:54.5666965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:55.9000227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9m3lemfi
2022-05-18T04:12:55.9000857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9m3lemfi/_remote_module_non_scriptable.py
2022-05-18T04:12:55.9184255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpso9_etk_
2022-05-18T04:12:55.9184849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpso9_etk_/_remote_module_non_scriptable.py
2022-05-18T04:12:56.1244147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:56.1244738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:56.1607463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:56.1608028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:56.1860489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:56.1861032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:56.2212405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:56.2213087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:12:56.5640607Z ok (4.619s)
2022-05-18T04:12:56.5640854Z 
2022-05-18T04:12:56.5641296Z ----------------------------------------------------------------------
2022-05-18T04:12:56.5641653Z Ran 1 test in 4.619s
2022-05-18T04:12:56.5641828Z 
2022-05-18T04:12:56.5641926Z OK
2022-05-18T04:12:56.5642190Z 
2022-05-18T04:12:56.5642312Z Generating XML reports...
2022-05-18T04:12:56.5688226Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041251.xml
2022-05-18T04:12:57.8647896Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:12:57.8658047Z 
2022-05-18T04:12:57.8658232Z Running tests...
2022-05-18T04:12:57.8658832Z ----------------------------------------------------------------------
2022-05-18T04:12:57.8668282Z   test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-05-18T04:12:59.5439995Z This unit test verifies whether the Future object is passed properly. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:12:59.5867082Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7059
2022-05-18T04:12:59.5985336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7060
2022-05-18T04:13:00.5553031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:00.5616284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:00.5896139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5h3yy9vg
2022-05-18T04:13:00.5896714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5h3yy9vg/_remote_module_non_scriptable.py
2022-05-18T04:13:00.5902597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qtvpdjz
2022-05-18T04:13:00.5903559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qtvpdjz/_remote_module_non_scriptable.py
2022-05-18T04:13:00.8030450Z ok (2.937s)
2022-05-18T04:13:00.8031070Z 
2022-05-18T04:13:00.8031539Z ----------------------------------------------------------------------
2022-05-18T04:13:00.8031917Z Ran 1 test in 2.937s
2022-05-18T04:13:00.8032084Z 
2022-05-18T04:13:00.8032189Z OK
2022-05-18T04:13:00.8032331Z 
2022-05-18T04:13:00.8032450Z Generating XML reports...
2022-05-18T04:13:00.8074165Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041257.xml
2022-05-18T04:13:02.0400741Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:02.0413060Z 
2022-05-18T04:13:02.0413530Z Running tests...
2022-05-18T04:13:02.0414396Z ----------------------------------------------------------------------
2022-05-18T04:13:02.0424146Z   test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-05-18T04:13:03.7420412Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:03.7852984Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7272
2022-05-18T04:13:03.7973612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7273
2022-05-18T04:13:04.7822237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:04.8031654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:06.1333746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsqbqeuu
2022-05-18T04:13:06.1334388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsqbqeuu/_remote_module_non_scriptable.py
2022-05-18T04:13:06.1683793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59u_r3fh
2022-05-18T04:13:06.1684377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59u_r3fh/_remote_module_non_scriptable.py
2022-05-18T04:13:06.5054275Z ok (4.464s)
2022-05-18T04:13:06.5054547Z 
2022-05-18T04:13:06.5056023Z ----------------------------------------------------------------------
2022-05-18T04:13:06.5056466Z Ran 1 test in 4.464s
2022-05-18T04:13:06.5056638Z 
2022-05-18T04:13:06.5056751Z OK
2022-05-18T04:13:06.5056894Z 
2022-05-18T04:13:06.5057019Z Generating XML reports...
2022-05-18T04:13:06.5098924Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041302.xml
2022-05-18T04:13:07.8038307Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:07.8047858Z 
2022-05-18T04:13:07.8048109Z Running tests...
2022-05-18T04:13:07.8048598Z ----------------------------------------------------------------------
2022-05-18T04:13:07.8059165Z   test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-05-18T04:13:09.4777105Z DDP communication hook can only be registered once. This test validates whether ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:09.5208026Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7487
2022-05-18T04:13:09.5315535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7488
2022-05-18T04:13:10.4794793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:10.5235389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:10.5477771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5td4h801
2022-05-18T04:13:10.5478624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5td4h801/_remote_module_non_scriptable.py
2022-05-18T04:13:10.5479351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd01gdkxw
2022-05-18T04:13:10.5481176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd01gdkxw/_remote_module_non_scriptable.py
2022-05-18T04:13:10.7361233Z ok (2.931s)
2022-05-18T04:13:10.7361903Z 
2022-05-18T04:13:10.7362369Z ----------------------------------------------------------------------
2022-05-18T04:13:10.7362759Z Ran 1 test in 2.931s
2022-05-18T04:13:10.7362930Z 
2022-05-18T04:13:10.7363026Z OK
2022-05-18T04:13:10.7363143Z 
2022-05-18T04:13:10.7363289Z Generating XML reports...
2022-05-18T04:13:10.7407289Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041307.xml
2022-05-18T04:13:12.0047034Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:12.0055991Z 
2022-05-18T04:13:12.0056281Z Running tests...
2022-05-18T04:13:12.0056771Z ----------------------------------------------------------------------
2022-05-18T04:13:12.0069623Z   test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-05-18T04:13:13.6047208Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:13.6465757Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7692
2022-05-18T04:13:13.6583269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7693
2022-05-18T04:13:14.6000401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:14.6023208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:14.6350979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xzxsqu2
2022-05-18T04:13:14.6351738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xzxsqu2/_remote_module_non_scriptable.py
2022-05-18T04:13:14.6352306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvq_faj5p
2022-05-18T04:13:14.6356580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvq_faj5p/_remote_module_non_scriptable.py
2022-05-18T04:13:14.8627877Z ok (2.857s)
2022-05-18T04:13:14.8628181Z 
2022-05-18T04:13:14.8628644Z ----------------------------------------------------------------------
2022-05-18T04:13:14.8628999Z Ran 1 test in 2.857s
2022-05-18T04:13:14.8629173Z 
2022-05-18T04:13:14.8629273Z OK
2022-05-18T04:13:14.8629393Z 
2022-05-18T04:13:14.8629539Z Generating XML reports...
2022-05-18T04:13:14.8674033Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041312.xml
2022-05-18T04:13:16.0876642Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:16.0886005Z 
2022-05-18T04:13:16.0886302Z Running tests...
2022-05-18T04:13:16.0886813Z ----------------------------------------------------------------------
2022-05-18T04:13:16.0898181Z   test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-05-18T04:13:17.7423652Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:17.7860333Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7967
2022-05-18T04:13:17.7970905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7968
2022-05-18T04:13:18.7563973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:18.7598848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:18.7902295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpas8ekk_y
2022-05-18T04:13:18.7903394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpas8ekk_y/_remote_module_non_scriptable.py
2022-05-18T04:13:18.7904494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3a220zay
2022-05-18T04:13:18.7906790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3a220zay/_remote_module_non_scriptable.py
2022-05-18T04:13:19.0017727Z ok (2.913s)
2022-05-18T04:13:19.0018000Z 
2022-05-18T04:13:19.0019039Z ----------------------------------------------------------------------
2022-05-18T04:13:19.0019431Z Ran 1 test in 2.913s
2022-05-18T04:13:19.0019599Z 
2022-05-18T04:13:19.0019692Z OK
2022-05-18T04:13:19.0019829Z 
2022-05-18T04:13:19.0019950Z Generating XML reports...
2022-05-18T04:13:19.0065286Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041316.xml
2022-05-18T04:13:20.2529715Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:20.2539713Z 
2022-05-18T04:13:20.2539928Z Running tests...
2022-05-18T04:13:20.2540712Z ----------------------------------------------------------------------
2022-05-18T04:13:20.2555320Z   test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-05-18T04:13:21.9377888Z This test checks whether return annotation checked properly if defined. It also ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:21.9821032Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8172
2022-05-18T04:13:21.9944215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8173
2022-05-18T04:13:22.9424019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:22.9443384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:22.9766192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyeitqeig
2022-05-18T04:13:22.9766771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphinag5zf
2022-05-18T04:13:22.9767347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyeitqeig/_remote_module_non_scriptable.py
2022-05-18T04:13:22.9768280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphinag5zf/_remote_module_non_scriptable.py
2022-05-18T04:13:23.1989000Z ok (2.945s)
2022-05-18T04:13:23.1989278Z 
2022-05-18T04:13:23.1989736Z ----------------------------------------------------------------------
2022-05-18T04:13:23.1990110Z Ran 1 test in 2.945s
2022-05-18T04:13:23.1990284Z 
2022-05-18T04:13:23.1990385Z OK
2022-05-18T04:13:23.1990503Z 
2022-05-18T04:13:23.1990644Z Generating XML reports...
2022-05-18T04:13:23.2035403Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041320.xml
2022-05-18T04:13:24.4389809Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:24.4399509Z 
2022-05-18T04:13:24.4399703Z Running tests...
2022-05-18T04:13:24.4400196Z ----------------------------------------------------------------------
2022-05-18T04:13:24.4420096Z   test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-05-18T04:13:26.0891269Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:26.1322898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8385
2022-05-18T04:13:26.1442235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8386
2022-05-18T04:13:27.0319067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:27.0789377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:27.1137274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfxsm2a5
2022-05-18T04:13:27.1137913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2x8fk3m
2022-05-18T04:13:27.1138458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfxsm2a5/_remote_module_non_scriptable.py
2022-05-18T04:13:27.1139039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2x8fk3m/_remote_module_non_scriptable.py
2022-05-18T04:13:27.1289863Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:13:28.7521076Z ok (4.312s)
2022-05-18T04:13:28.7521477Z 
2022-05-18T04:13:28.7521878Z ----------------------------------------------------------------------
2022-05-18T04:13:28.7522254Z Ran 1 test in 4.312s
2022-05-18T04:13:28.7522424Z 
2022-05-18T04:13:28.7522531Z OK
2022-05-18T04:13:28.7522672Z 
2022-05-18T04:13:28.7522811Z Generating XML reports...
2022-05-18T04:13:28.7567880Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041324.xml
2022-05-18T04:13:30.0406881Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:30.0418032Z 
2022-05-18T04:13:30.0418401Z Running tests...
2022-05-18T04:13:30.0418862Z ----------------------------------------------------------------------
2022-05-18T04:13:31.6870933Z   test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:31.7292871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8600
2022-05-18T04:13:31.7408164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8601
2022-05-18T04:13:32.7034709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:32.7095020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:32.7378724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5yk91eex
2022-05-18T04:13:32.7379348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5yk91eex/_remote_module_non_scriptable.py
2022-05-18T04:13:32.7387215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1t3uvy33
2022-05-18T04:13:32.7388169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1t3uvy33/_remote_module_non_scriptable.py
2022-05-18T04:13:34.4490438Z ok (4.407s)
2022-05-18T04:13:34.4490739Z 
2022-05-18T04:13:34.4491186Z ----------------------------------------------------------------------
2022-05-18T04:13:34.4491545Z Ran 1 test in 4.407s
2022-05-18T04:13:34.4491849Z 
2022-05-18T04:13:34.4491950Z OK
2022-05-18T04:13:34.4492067Z 
2022-05-18T04:13:34.4492215Z Generating XML reports...
2022-05-18T04:13:34.4546120Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041330.xml
2022-05-18T04:13:35.7486343Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:35.7496491Z 
2022-05-18T04:13:35.7496912Z Running tests...
2022-05-18T04:13:35.7497394Z ----------------------------------------------------------------------
2022-05-18T04:13:37.4112051Z   test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:37.4544628Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8815
2022-05-18T04:13:37.4660864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8816
2022-05-18T04:13:38.4106293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:38.4395686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:38.4640319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskezhi08
2022-05-18T04:13:38.4640941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskezhi08/_remote_module_non_scriptable.py
2022-05-18T04:13:38.4647633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptuafm0zc
2022-05-18T04:13:38.4648274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptuafm0zc/_remote_module_non_scriptable.py
2022-05-18T04:13:40.0739057Z ok (4.324s)
2022-05-18T04:13:40.0739299Z 
2022-05-18T04:13:40.0739729Z ----------------------------------------------------------------------
2022-05-18T04:13:40.0740078Z Ran 1 test in 4.324s
2022-05-18T04:13:40.0740245Z 
2022-05-18T04:13:40.0740345Z OK
2022-05-18T04:13:40.0740481Z 
2022-05-18T04:13:40.0740618Z Generating XML reports...
2022-05-18T04:13:40.0783478Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041335.xml
2022-05-18T04:13:41.3158377Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:41.3170294Z 
2022-05-18T04:13:41.3170683Z Running tests...
2022-05-18T04:13:41.3171199Z ----------------------------------------------------------------------
2022-05-18T04:13:42.9641978Z   test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:43.0076421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9030
2022-05-18T04:13:43.0193928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9031
2022-05-18T04:13:43.9284921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:43.9470399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:43.9817279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yvsuzua
2022-05-18T04:13:43.9818399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yvsuzua/_remote_module_non_scriptable.py
2022-05-18T04:13:43.9819459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxjxy8g7
2022-05-18T04:13:43.9825652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxjxy8g7/_remote_module_non_scriptable.py
2022-05-18T04:13:43.9978047Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:13:43.9979648Z   warnings.warn(
2022-05-18T04:13:43.9982356Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:13:43.9983739Z   warnings.warn(
2022-05-18T04:13:45.6276140Z ok (4.310s)
2022-05-18T04:13:45.6276369Z 
2022-05-18T04:13:45.6276797Z ----------------------------------------------------------------------
2022-05-18T04:13:45.6277135Z Ran 1 test in 4.311s
2022-05-18T04:13:45.6277289Z 
2022-05-18T04:13:45.6277386Z OK
2022-05-18T04:13:45.6277913Z 
2022-05-18T04:13:45.6278051Z Generating XML reports...
2022-05-18T04:13:45.6320709Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041341.xml
2022-05-18T04:13:46.9098734Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:46.9108963Z 
2022-05-18T04:13:46.9109361Z Running tests...
2022-05-18T04:13:46.9109827Z ----------------------------------------------------------------------
2022-05-18T04:13:48.5429498Z   test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:48.5876649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9245
2022-05-18T04:13:48.5995303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9246
2022-05-18T04:13:49.5524274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:49.5705163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:50.9014957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppszg_gog
2022-05-18T04:13:50.9015610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppszg_gog/_remote_module_non_scriptable.py
2022-05-18T04:13:50.9314515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjs0y64nt
2022-05-18T04:13:50.9315137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjs0y64nt/_remote_module_non_scriptable.py
2022-05-18T04:13:51.1392227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:51.1392798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:51.5082025Z ok (4.597s)
2022-05-18T04:13:51.5082265Z 
2022-05-18T04:13:51.5082719Z ----------------------------------------------------------------------
2022-05-18T04:13:51.5083099Z Ran 1 test in 4.597s
2022-05-18T04:13:51.5083259Z 
2022-05-18T04:13:51.5083365Z OK
2022-05-18T04:13:51.5083504Z 
2022-05-18T04:13:51.5083645Z Generating XML reports...
2022-05-18T04:13:51.5126941Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041346.xml
2022-05-18T04:13:52.7785136Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:52.7796620Z 
2022-05-18T04:13:52.7797036Z Running tests...
2022-05-18T04:13:52.7797533Z ----------------------------------------------------------------------
2022-05-18T04:13:54.4432953Z   test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:13:54.4860172Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9460
2022-05-18T04:13:54.4965672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9461
2022-05-18T04:13:55.4506609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:55.4547262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:56.7854707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilj7_wk2
2022-05-18T04:13:56.7855350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilj7_wk2/_remote_module_non_scriptable.py
2022-05-18T04:13:56.8213216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6txjudgc
2022-05-18T04:13:56.8215163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6txjudgc/_remote_module_non_scriptable.py
2022-05-18T04:13:57.0256264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:57.0256900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:13:57.4051328Z ok (4.625s)
2022-05-18T04:13:57.4051777Z 
2022-05-18T04:13:57.4052200Z ----------------------------------------------------------------------
2022-05-18T04:13:57.4052559Z Ran 1 test in 4.625s
2022-05-18T04:13:57.4052731Z 
2022-05-18T04:13:57.4052829Z OK
2022-05-18T04:13:57.4052970Z 
2022-05-18T04:13:57.4053146Z Generating XML reports...
2022-05-18T04:13:57.4097606Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041352.xml
2022-05-18T04:13:58.7127495Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:58.7137192Z 
2022-05-18T04:13:58.7137796Z Running tests...
2022-05-18T04:13:58.7138651Z ----------------------------------------------------------------------
2022-05-18T04:14:00.3576638Z   test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:00.4010116Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9675
2022-05-18T04:14:00.4125615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9676
2022-05-18T04:14:01.3554553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:01.3795592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:03.7420580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps712_4_f
2022-05-18T04:14:03.7421300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps712_4_f/_remote_module_non_scriptable.py
2022-05-18T04:14:03.7793562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8v407pm2
2022-05-18T04:14:03.7794167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8v407pm2/_remote_module_non_scriptable.py
2022-05-18T04:14:04.1113573Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:14:04.1114261Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:14:04.1121549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:04.1122140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:04.5238192Z ok (5.810s)
2022-05-18T04:14:04.5238676Z 
2022-05-18T04:14:04.5239424Z ----------------------------------------------------------------------
2022-05-18T04:14:04.5240070Z Ran 1 test in 5.810s
2022-05-18T04:14:04.5240366Z 
2022-05-18T04:14:04.5240521Z OK
2022-05-18T04:14:04.5240755Z 
2022-05-18T04:14:04.5242545Z Generating XML reports...
2022-05-18T04:14:04.5285817Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041358.xml
2022-05-18T04:14:05.7938368Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:05.7957218Z 
2022-05-18T04:14:05.7960952Z Running tests...
2022-05-18T04:14:05.7964566Z ----------------------------------------------------------------------
2022-05-18T04:14:07.4307198Z   test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:07.4739498Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9892
2022-05-18T04:14:07.4856549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9893
2022-05-18T04:14:08.4332570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:08.4360579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:08.5900835Z skip: Need at least 8 CUDA devices (2.795s)
2022-05-18T04:14:08.5901335Z 
2022-05-18T04:14:08.5902536Z ----------------------------------------------------------------------
2022-05-18T04:14:08.5903183Z Ran 1 test in 2.795s
2022-05-18T04:14:08.5903491Z 
2022-05-18T04:14:08.5904141Z OK (skipped=1)
2022-05-18T04:14:08.5904407Z 
2022-05-18T04:14:08.5904654Z Generating XML reports...
2022-05-18T04:14:08.5948831Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041405.xml
2022-05-18T04:14:09.8377835Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:09.8387749Z 
2022-05-18T04:14:09.8388280Z Running tests...
2022-05-18T04:14:09.8388832Z ----------------------------------------------------------------------
2022-05-18T04:14:11.4666456Z   test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:11.5089990Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10091
2022-05-18T04:14:11.5206935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10092
2022-05-18T04:14:12.4770886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:12.4897367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:12.5248386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2kapa1yz
2022-05-18T04:14:12.5248991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2kapa1yz/_remote_module_non_scriptable.py
2022-05-18T04:14:12.5249533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptkgzfs3y
2022-05-18T04:14:12.5254700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptkgzfs3y/_remote_module_non_scriptable.py
2022-05-18T04:14:12.5458838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:12.5459390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:12.7249405Z ok (2.886s)
2022-05-18T04:14:12.7249699Z 
2022-05-18T04:14:12.7250134Z ----------------------------------------------------------------------
2022-05-18T04:14:12.7250493Z Ran 1 test in 2.886s
2022-05-18T04:14:12.7250668Z 
2022-05-18T04:14:12.7250766Z OK
2022-05-18T04:14:12.7250883Z 
2022-05-18T04:14:12.7251022Z Generating XML reports...
2022-05-18T04:14:12.7294924Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041409.xml
2022-05-18T04:14:13.9572606Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:13.9582151Z 
2022-05-18T04:14:13.9582573Z Running tests...
2022-05-18T04:14:13.9583668Z ----------------------------------------------------------------------
2022-05-18T04:14:15.5890004Z   test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:15.6318041Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10304
2022-05-18T04:14:15.6437188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10305
2022-05-18T04:14:16.6056878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:16.6070289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:16.6427059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwz9grahm
2022-05-18T04:14:16.6427720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskgpyvyh
2022-05-18T04:14:16.6428292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwz9grahm/_remote_module_non_scriptable.py
2022-05-18T04:14:16.6429441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskgpyvyh/_remote_module_non_scriptable.py
2022-05-18T04:14:16.6645888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:16.6646504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:16.8482142Z ok (2.890s)
2022-05-18T04:14:16.8482821Z 
2022-05-18T04:14:16.8483320Z ----------------------------------------------------------------------
2022-05-18T04:14:16.8483768Z Ran 1 test in 2.890s
2022-05-18T04:14:16.8483982Z 
2022-05-18T04:14:16.8484117Z OK
2022-05-18T04:14:16.8484238Z 
2022-05-18T04:14:16.8484601Z Generating XML reports...
2022-05-18T04:14:16.8529500Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041413.xml
2022-05-18T04:14:18.1089435Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:18.1098750Z 
2022-05-18T04:14:18.1099106Z Running tests...
2022-05-18T04:14:18.1100133Z ----------------------------------------------------------------------
2022-05-18T04:14:18.1117785Z   test_ignored_output (__main__.DistributedDataParallelTest)
2022-05-18T04:14:19.7657663Z Test that the output of a model can be ignored and that there is no ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:19.8090671Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10517
2022-05-18T04:14:19.8209214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10518
2022-05-18T04:14:20.7623849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:20.7938010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:20.8276114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwi31wmvk
2022-05-18T04:14:20.8276752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwi31wmvk/_remote_module_non_scriptable.py
2022-05-18T04:14:20.8277392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfv2rdkp0
2022-05-18T04:14:20.8281010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfv2rdkp0/_remote_module_non_scriptable.py
2022-05-18T04:14:20.8532801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:20.8533842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:21.0257002Z ok (2.916s)
2022-05-18T04:14:21.0257304Z 
2022-05-18T04:14:21.0257753Z ----------------------------------------------------------------------
2022-05-18T04:14:21.0258206Z Ran 1 test in 2.916s
2022-05-18T04:14:21.0258580Z 
2022-05-18T04:14:21.0258721Z OK
2022-05-18T04:14:21.0258925Z 
2022-05-18T04:14:21.0259110Z Generating XML reports...
2022-05-18T04:14:21.0303423Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041418.xml
2022-05-18T04:14:22.2745770Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:22.2773791Z 
2022-05-18T04:14:22.2774062Z Running tests...
2022-05-18T04:14:22.2774835Z ----------------------------------------------------------------------
2022-05-18T04:14:22.2776162Z   test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-05-18T04:14:23.9173013Z Test that the output of a model can be ignored and that there is no ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:23.9613443Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10792
2022-05-18T04:14:23.9733894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10793
2022-05-18T04:14:24.9358801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:24.9531836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:24.9876071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4lmah56
2022-05-18T04:14:24.9877226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4lmah56/_remote_module_non_scriptable.py
2022-05-18T04:14:24.9878283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqns6xh3w
2022-05-18T04:14:24.9883168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqns6xh3w/_remote_module_non_scriptable.py
2022-05-18T04:14:25.2784678Z ok (3.002s)
2022-05-18T04:14:25.2784987Z 
2022-05-18T04:14:25.2785439Z ----------------------------------------------------------------------
2022-05-18T04:14:25.2785810Z Ran 1 test in 3.003s
2022-05-18T04:14:25.2785995Z 
2022-05-18T04:14:25.2786096Z OK
2022-05-18T04:14:25.2786504Z 
2022-05-18T04:14:25.2786648Z Generating XML reports...
2022-05-18T04:14:25.2833591Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041422.xml
2022-05-18T04:14:26.5816156Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:26.5826897Z 
2022-05-18T04:14:26.5827228Z Running tests...
2022-05-18T04:14:26.5827700Z ----------------------------------------------------------------------
2022-05-18T04:14:28.2504918Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:28.2948059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11067
2022-05-18T04:14:28.3069752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11068
2022-05-18T04:14:29.2748822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:29.2758002Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:29.2759191Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:29.2760298Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:29.2761396Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:29.2762477Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:29.2763558Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:29.3021310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:29.3028919Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:29.3030451Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:29.3031639Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:29.3032751Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:29.3033846Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:29.3034939Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:29.5119423Z ok (2.929s)
2022-05-18T04:14:29.5119850Z 
2022-05-18T04:14:29.5120700Z ----------------------------------------------------------------------
2022-05-18T04:14:29.5121080Z Ran 1 test in 2.929s
2022-05-18T04:14:29.5121341Z 
2022-05-18T04:14:29.5121493Z OK
2022-05-18T04:14:29.5121773Z 
2022-05-18T04:14:29.5122052Z Generating XML reports...
2022-05-18T04:14:29.5166143Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041426.xml
2022-05-18T04:14:30.7510514Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:30.7520779Z 
2022-05-18T04:14:30.7521195Z Running tests...
2022-05-18T04:14:30.7521697Z ----------------------------------------------------------------------
2022-05-18T04:14:32.4249168Z   test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:32.4688523Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11266
2022-05-18T04:14:32.4811252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11267
2022-05-18T04:14:33.4199864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:33.4399587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:33.4513585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:14:33.4514125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:14:33.4514973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:33.4515704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:14:34.7730784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_5fg03f
2022-05-18T04:14:34.7731398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_5fg03f/_remote_module_non_scriptable.py
2022-05-18T04:14:34.7994450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjl6qg1j3
2022-05-18T04:14:34.7995054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjl6qg1j3/_remote_module_non_scriptable.py
2022-05-18T04:14:35.0032197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:35.0032779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:36.0413794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:36.0414748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:36.3921123Z ok (5.640s)
2022-05-18T04:14:36.3921309Z 
2022-05-18T04:14:36.3921740Z ----------------------------------------------------------------------
2022-05-18T04:14:36.3922112Z Ran 1 test in 5.640s
2022-05-18T04:14:36.3922284Z 
2022-05-18T04:14:36.3922396Z OK
2022-05-18T04:14:36.3922553Z 
2022-05-18T04:14:36.3922669Z Generating XML reports...
2022-05-18T04:14:36.3967780Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041430.xml
2022-05-18T04:14:37.6378987Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:37.6390025Z 
2022-05-18T04:14:37.6390304Z Running tests...
2022-05-18T04:14:37.6390788Z ----------------------------------------------------------------------
2022-05-18T04:14:39.2862578Z   test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:39.3306448Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11482
2022-05-18T04:14:39.3418259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11483
2022-05-18T04:14:40.2994867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:40.3142254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:40.3448281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nl4cef_
2022-05-18T04:14:40.3448878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7wqftq7
2022-05-18T04:14:40.3449420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nl4cef_/_remote_module_non_scriptable.py
2022-05-18T04:14:40.3452840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7wqftq7/_remote_module_non_scriptable.py
2022-05-18T04:14:40.5464470Z ok (2.907s)
2022-05-18T04:14:40.5464712Z 
2022-05-18T04:14:40.5465168Z ----------------------------------------------------------------------
2022-05-18T04:14:40.5465497Z Ran 1 test in 2.908s
2022-05-18T04:14:40.5465666Z 
2022-05-18T04:14:40.5465761Z OK
2022-05-18T04:14:40.5465897Z 
2022-05-18T04:14:40.5466035Z Generating XML reports...
2022-05-18T04:14:40.5510110Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041437.xml
2022-05-18T04:14:41.8149022Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:41.8159373Z 
2022-05-18T04:14:41.8159654Z Running tests...
2022-05-18T04:14:41.8160126Z ----------------------------------------------------------------------
2022-05-18T04:14:43.4781092Z   test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:43.5219316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11757
2022-05-18T04:14:43.5340918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11758
2022-05-18T04:14:44.5009623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:44.5304330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:44.5566922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp141jyr41
2022-05-18T04:14:44.5567733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoq4nrizg
2022-05-18T04:14:44.5568291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp141jyr41/_remote_module_non_scriptable.py
2022-05-18T04:14:44.5568828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoq4nrizg/_remote_module_non_scriptable.py
2022-05-18T04:14:44.7386259Z ok (2.922s)
2022-05-18T04:14:44.7386492Z 
2022-05-18T04:14:44.7386917Z ----------------------------------------------------------------------
2022-05-18T04:14:44.7387699Z Ran 1 test in 2.923s
2022-05-18T04:14:44.7387873Z 
2022-05-18T04:14:44.7387973Z OK
2022-05-18T04:14:44.7388109Z 
2022-05-18T04:14:44.7388248Z Generating XML reports...
2022-05-18T04:14:44.7431766Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041441.xml
2022-05-18T04:14:45.9795386Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:45.9805271Z 
2022-05-18T04:14:45.9805789Z Running tests...
2022-05-18T04:14:45.9806277Z ----------------------------------------------------------------------
2022-05-18T04:14:47.6572092Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:47.7015563Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12032
2022-05-18T04:14:47.7132736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12033
2022-05-18T04:14:48.6511655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:48.6673413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:50.0260622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbe4h65em
2022-05-18T04:14:50.0261298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbe4h65em/_remote_module_non_scriptable.py
2022-05-18T04:14:50.0304836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl08z_vss
2022-05-18T04:14:50.0305733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl08z_vss/_remote_module_non_scriptable.py
2022-05-18T04:14:50.8032413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:50.8032984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:51.1230613Z ok (5.142s)
2022-05-18T04:14:51.1230860Z 
2022-05-18T04:14:51.1231320Z ----------------------------------------------------------------------
2022-05-18T04:14:51.1231703Z Ran 1 test in 5.143s
2022-05-18T04:14:51.1231853Z 
2022-05-18T04:14:51.1231949Z OK
2022-05-18T04:14:51.1232085Z 
2022-05-18T04:14:51.1232224Z Generating XML reports...
2022-05-18T04:14:51.1277696Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041445.xml
2022-05-18T04:14:52.3933171Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:52.3954042Z 
2022-05-18T04:14:52.3954703Z Running tests...
2022-05-18T04:14:52.3955575Z ----------------------------------------------------------------------
2022-05-18T04:14:54.0519487Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:14:54.0955961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12247
2022-05-18T04:14:54.1076264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12248
2022-05-18T04:14:55.0480899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:55.0505943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:56.3665417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_plojtr
2022-05-18T04:14:56.3666035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_plojtr/_remote_module_non_scriptable.py
2022-05-18T04:14:56.3858083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6ybehwr
2022-05-18T04:14:56.3858675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6ybehwr/_remote_module_non_scriptable.py
2022-05-18T04:14:56.9798014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:56.9798751Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:57.3169482Z ok (4.922s)
2022-05-18T04:14:57.3169673Z 
2022-05-18T04:14:57.3170104Z ----------------------------------------------------------------------
2022-05-18T04:14:57.3170454Z Ran 1 test in 4.922s
2022-05-18T04:14:57.3170619Z 
2022-05-18T04:14:57.3170694Z OK
2022-05-18T04:14:57.3171388Z 
2022-05-18T04:14:57.3171555Z Generating XML reports...
2022-05-18T04:14:57.3216172Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041452.xml
2022-05-18T04:14:58.5906259Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:58.5916415Z 
2022-05-18T04:14:58.5916603Z Running tests...
2022-05-18T04:14:58.5917088Z ----------------------------------------------------------------------
2022-05-18T04:15:00.2448630Z   test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:00.2890973Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12462
2022-05-18T04:15:00.3011695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12463
2022-05-18T04:15:00.3136018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12464
2022-05-18T04:15:00.3246119Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12465
2022-05-18T04:15:01.3157493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:01.3395995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:01.3613615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:01.3807246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:01.6296592Z ok (3.038s)
2022-05-18T04:15:01.6296820Z 
2022-05-18T04:15:01.6297257Z ----------------------------------------------------------------------
2022-05-18T04:15:01.6297635Z Ran 1 test in 3.038s
2022-05-18T04:15:01.6297782Z 
2022-05-18T04:15:01.6297880Z OK
2022-05-18T04:15:01.6298016Z 
2022-05-18T04:15:01.6298156Z Generating XML reports...
2022-05-18T04:15:01.6344777Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041458.xml
2022-05-18T04:15:02.9813069Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:02.9823191Z 
2022-05-18T04:15:02.9823349Z Running tests...
2022-05-18T04:15:02.9823862Z ----------------------------------------------------------------------
2022-05-18T04:15:04.6357120Z   test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:04.6787780Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12805
2022-05-18T04:15:04.6910442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12806
2022-05-18T04:15:04.7029712Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12807
2022-05-18T04:15:04.7151220Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12808
2022-05-18T04:15:05.7029630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:05.7286766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:05.7359596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:05.7813502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:07.7241900Z ok (4.741s)
2022-05-18T04:15:07.7242238Z 
2022-05-18T04:15:07.7242794Z ----------------------------------------------------------------------
2022-05-18T04:15:07.7243194Z Ran 1 test in 4.742s
2022-05-18T04:15:07.7243369Z 
2022-05-18T04:15:07.7243463Z OK
2022-05-18T04:15:07.7243580Z 
2022-05-18T04:15:07.7244110Z Generating XML reports...
2022-05-18T04:15:07.7284924Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041502.xml
2022-05-18T04:15:08.9935831Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:08.9946588Z 
2022-05-18T04:15:08.9946979Z Running tests...
2022-05-18T04:15:08.9947476Z ----------------------------------------------------------------------
2022-05-18T04:15:10.6520842Z   test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:10.6966142Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13152
2022-05-18T04:15:10.7078713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13153
2022-05-18T04:15:10.7185266Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13154
2022-05-18T04:15:10.7309467Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13155
2022-05-18T04:15:11.6617590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:11.7170441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:11.7225444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:11.7291693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:12.0360788Z ok (3.041s)
2022-05-18T04:15:12.0361018Z 
2022-05-18T04:15:12.0361451Z ----------------------------------------------------------------------
2022-05-18T04:15:12.0361906Z Ran 1 test in 3.041s
2022-05-18T04:15:12.0362067Z 
2022-05-18T04:15:12.0362166Z OK
2022-05-18T04:15:12.0362303Z 
2022-05-18T04:15:12.0362440Z Generating XML reports...
2022-05-18T04:15:12.0406560Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041508.xml
2022-05-18T04:15:13.3218960Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:13.3229466Z 
2022-05-18T04:15:13.3229688Z Running tests...
2022-05-18T04:15:13.3230180Z ----------------------------------------------------------------------
2022-05-18T04:15:14.9758422Z   test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:15.0201273Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13495
2022-05-18T04:15:15.0322598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13496
2022-05-18T04:15:15.0447422Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13497
2022-05-18T04:15:15.0558115Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13498
2022-05-18T04:15:16.0303664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:16.0330235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:16.0385454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:16.0719384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:16.0932024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:15:16.1034856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:15:16.1035474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:15:16.1035994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:15:16.1036850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:15:16.1037749Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:15:16.1038461Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:15:16.1137427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:15:16.3609585Z ok (3.038s)
2022-05-18T04:15:16.3609778Z 
2022-05-18T04:15:16.3610207Z ----------------------------------------------------------------------
2022-05-18T04:15:16.3610761Z Ran 1 test in 3.038s
2022-05-18T04:15:16.3610935Z 
2022-05-18T04:15:16.3611034Z OK
2022-05-18T04:15:16.3611174Z 
2022-05-18T04:15:16.3611314Z Generating XML reports...
2022-05-18T04:15:16.3655018Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041513.xml
2022-05-18T04:15:17.6295367Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:17.6305735Z 
2022-05-18T04:15:17.6305920Z Running tests...
2022-05-18T04:15:17.6306716Z ----------------------------------------------------------------------
2022-05-18T04:15:19.2759312Z   test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:19.3194517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13838
2022-05-18T04:15:19.3317005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13839
2022-05-18T04:15:19.3437964Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13840
2022-05-18T04:15:19.3559429Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13841
2022-05-18T04:15:20.3433942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:20.3637711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:20.3874413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:20.4277795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:20.6611166Z ok (3.030s)
2022-05-18T04:15:20.6611400Z 
2022-05-18T04:15:20.6611866Z ----------------------------------------------------------------------
2022-05-18T04:15:20.6612230Z Ran 1 test in 3.031s
2022-05-18T04:15:20.6612379Z 
2022-05-18T04:15:20.6612475Z OK
2022-05-18T04:15:20.6612610Z 
2022-05-18T04:15:20.6612751Z Generating XML reports...
2022-05-18T04:15:20.6657492Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041517.xml
2022-05-18T04:15:21.9361223Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:21.9370570Z 
2022-05-18T04:15:21.9370823Z Running tests...
2022-05-18T04:15:21.9371310Z ----------------------------------------------------------------------
2022-05-18T04:15:23.5952363Z   test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:23.6400861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14181
2022-05-18T04:15:23.6523498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14182
2022-05-18T04:15:23.6651532Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14183
2022-05-18T04:15:23.6763433Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14184
2022-05-18T04:15:24.6875573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:24.6915159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:24.6925244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:24.6961980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:24.9814839Z ok (3.044s)
2022-05-18T04:15:24.9815075Z 
2022-05-18T04:15:24.9815516Z ----------------------------------------------------------------------
2022-05-18T04:15:24.9815843Z Ran 1 test in 3.044s
2022-05-18T04:15:24.9816010Z 
2022-05-18T04:15:24.9816113Z OK
2022-05-18T04:15:24.9816268Z 
2022-05-18T04:15:24.9816407Z Generating XML reports...
2022-05-18T04:15:24.9859622Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041521.xml
2022-05-18T04:15:26.2217835Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:26.2230320Z 
2022-05-18T04:15:26.2231121Z Running tests...
2022-05-18T04:15:26.2232031Z ----------------------------------------------------------------------
2022-05-18T04:15:27.8848889Z   test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:27.9302517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14524
2022-05-18T04:15:27.9425799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14525
2022-05-18T04:15:27.9552450Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14526
2022-05-18T04:15:27.9676829Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14527
2022-05-18T04:15:28.9246497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:28.9824147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:28.9931004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:29.0165772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:29.6736615Z ok (3.450s)
2022-05-18T04:15:29.6736956Z 
2022-05-18T04:15:29.6737406Z ----------------------------------------------------------------------
2022-05-18T04:15:29.6737773Z Ran 1 test in 3.451s
2022-05-18T04:15:29.6737939Z 
2022-05-18T04:15:29.6738036Z OK
2022-05-18T04:15:29.6738173Z 
2022-05-18T04:15:29.6738312Z Generating XML reports...
2022-05-18T04:15:29.6782104Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041526.xml
2022-05-18T04:15:30.9554657Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:30.9563890Z 
2022-05-18T04:15:30.9564081Z Running tests...
2022-05-18T04:15:30.9564875Z ----------------------------------------------------------------------
2022-05-18T04:15:32.5787016Z   test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:32.6231441Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14891
2022-05-18T04:15:32.6352817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14892
2022-05-18T04:15:32.6474844Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14893
2022-05-18T04:15:32.6584199Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14894
2022-05-18T04:15:33.6013342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:33.6699785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:33.7146810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:33.7206609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:37.1709789Z ok (6.214s)
2022-05-18T04:15:37.1710016Z 
2022-05-18T04:15:37.1710452Z ----------------------------------------------------------------------
2022-05-18T04:15:37.1710818Z Ran 1 test in 6.215s
2022-05-18T04:15:37.1710990Z 
2022-05-18T04:15:37.1711069Z OK
2022-05-18T04:15:37.1711206Z 
2022-05-18T04:15:37.1711708Z Generating XML reports...
2022-05-18T04:15:37.1754729Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041530.xml
2022-05-18T04:15:38.4617173Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:38.4628347Z 
2022-05-18T04:15:38.4628708Z Running tests...
2022-05-18T04:15:38.4629231Z ----------------------------------------------------------------------
2022-05-18T04:15:40.1358010Z   test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:40.1806730Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15262
2022-05-18T04:15:40.1932762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15263
2022-05-18T04:15:40.2058640Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15264
2022-05-18T04:15:40.2183674Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15265
2022-05-18T04:15:41.2487779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:41.2508262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:41.2519255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:41.2585362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:41.5237821Z ok (3.061s)
2022-05-18T04:15:41.5238120Z 
2022-05-18T04:15:41.5238719Z ----------------------------------------------------------------------
2022-05-18T04:15:41.5239072Z Ran 1 test in 3.061s
2022-05-18T04:15:41.5239239Z 
2022-05-18T04:15:41.5239335Z OK
2022-05-18T04:15:41.5239471Z 
2022-05-18T04:15:41.5239587Z Generating XML reports...
2022-05-18T04:15:41.5283177Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041538.xml
2022-05-18T04:15:42.8110125Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:42.8120717Z 
2022-05-18T04:15:42.8121175Z Running tests...
2022-05-18T04:15:42.8121669Z ----------------------------------------------------------------------
2022-05-18T04:15:44.4795058Z   test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:44.5246212Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15605
2022-05-18T04:15:44.5368240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15606
2022-05-18T04:15:44.5480872Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15607
2022-05-18T04:15:44.5605695Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15608
2022-05-18T04:15:45.5023949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:45.5064497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:45.5464558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:45.5505748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:47.4698863Z ok (4.658s)
2022-05-18T04:15:47.4699498Z 
2022-05-18T04:15:47.4699917Z ----------------------------------------------------------------------
2022-05-18T04:15:47.4700295Z Ran 1 test in 4.658s
2022-05-18T04:15:47.4700471Z 
2022-05-18T04:15:47.4700571Z OK
2022-05-18T04:15:47.4700716Z 
2022-05-18T04:15:47.4700861Z Generating XML reports...
2022-05-18T04:15:47.4744329Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041542.xml
2022-05-18T04:15:48.7975928Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:48.7987304Z 
2022-05-18T04:15:48.7987506Z Running tests...
2022-05-18T04:15:48.7988331Z ----------------------------------------------------------------------
2022-05-18T04:15:50.4361964Z   test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:50.4808107Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15952
2022-05-18T04:15:50.4930132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15953
2022-05-18T04:15:50.5056375Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15954
2022-05-18T04:15:50.5181020Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15955
2022-05-18T04:15:51.5219086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:51.5423494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:51.5650707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:51.6227852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:53.5277210Z ok (4.729s)
2022-05-18T04:15:53.5277447Z 
2022-05-18T04:15:53.5277874Z ----------------------------------------------------------------------
2022-05-18T04:15:53.5278237Z Ran 1 test in 4.729s
2022-05-18T04:15:53.5278432Z 
2022-05-18T04:15:53.5278532Z OK
2022-05-18T04:15:53.5280731Z 
2022-05-18T04:15:53.5281049Z Generating XML reports...
2022-05-18T04:15:53.5323850Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041548.xml
2022-05-18T04:15:54.8043430Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:54.8052160Z 
2022-05-18T04:15:54.8052592Z Running tests...
2022-05-18T04:15:54.8053061Z ----------------------------------------------------------------------
2022-05-18T04:15:56.4087743Z   test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:15:56.4530257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16299
2022-05-18T04:15:56.4653119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16300
2022-05-18T04:15:56.4782914Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16301
2022-05-18T04:15:56.4901244Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16302
2022-05-18T04:15:57.5076064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:57.5250491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:57.5479977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:57.5544394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:57.7952952Z ok (2.990s)
2022-05-18T04:15:57.7953267Z 
2022-05-18T04:15:57.7953870Z ----------------------------------------------------------------------
2022-05-18T04:15:57.7954240Z Ran 1 test in 2.990s
2022-05-18T04:15:57.7954410Z 
2022-05-18T04:15:57.7954511Z OK
2022-05-18T04:15:57.7954663Z 
2022-05-18T04:15:57.7954801Z Generating XML reports...
2022-05-18T04:15:57.7998276Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041554.xml
2022-05-18T04:15:59.0571612Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:59.0582109Z 
2022-05-18T04:15:59.0582403Z Running tests...
2022-05-18T04:15:59.0583269Z ----------------------------------------------------------------------
2022-05-18T04:16:00.7214510Z   test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:00.7665094Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16642
2022-05-18T04:16:00.7778158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16643
2022-05-18T04:16:00.7899493Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16644
2022-05-18T04:16:00.8023121Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16645
2022-05-18T04:16:01.7472116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:01.7987467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:01.7998104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:01.8227663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:02.1077264Z ok (3.049s)
2022-05-18T04:16:02.1077562Z 
2022-05-18T04:16:02.1078013Z ----------------------------------------------------------------------
2022-05-18T04:16:02.1078374Z Ran 1 test in 3.049s
2022-05-18T04:16:02.1078552Z 
2022-05-18T04:16:02.1078672Z OK
2022-05-18T04:16:02.1078788Z 
2022-05-18T04:16:02.1078933Z Generating XML reports...
2022-05-18T04:16:02.1122452Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041559.xml
2022-05-18T04:16:03.3887329Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:03.3897446Z 
2022-05-18T04:16:03.3897934Z Running tests...
2022-05-18T04:16:03.3898441Z ----------------------------------------------------------------------
2022-05-18T04:16:05.0732693Z   test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:05.1180822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16985
2022-05-18T04:16:05.1307211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16986
2022-05-18T04:16:05.1429791Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16987
2022-05-18T04:16:05.1553464Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16988
2022-05-18T04:16:06.1386430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:06.1427372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:06.1520517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:06.1771685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:06.1881521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:16:06.1946824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:16:06.1947399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:16:06.1947946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:16:06.1948807Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:16:06.1949529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:16:06.1984853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:16:06.2049522Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:16:06.4605465Z ok (3.071s)
2022-05-18T04:16:06.4605703Z 
2022-05-18T04:16:06.4606275Z ----------------------------------------------------------------------
2022-05-18T04:16:06.4606612Z Ran 1 test in 3.071s
2022-05-18T04:16:06.4606791Z 
2022-05-18T04:16:06.4606892Z OK
2022-05-18T04:16:06.4607035Z 
2022-05-18T04:16:06.4607814Z Generating XML reports...
2022-05-18T04:16:06.4652174Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041603.xml
2022-05-18T04:16:07.7481175Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:07.7490880Z 
2022-05-18T04:16:07.7491110Z Running tests...
2022-05-18T04:16:07.7491628Z ----------------------------------------------------------------------
2022-05-18T04:16:09.3545245Z   test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:09.3988613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17328
2022-05-18T04:16:09.4100417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17329
2022-05-18T04:16:09.4224336Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17330
2022-05-18T04:16:09.4348484Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17331
2022-05-18T04:16:10.3810933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:10.4007591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:10.4053397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:10.4384016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:10.7400541Z ok (2.991s)
2022-05-18T04:16:10.7400766Z 
2022-05-18T04:16:10.7401219Z ----------------------------------------------------------------------
2022-05-18T04:16:10.7401769Z Ran 1 test in 2.991s
2022-05-18T04:16:10.7401948Z 
2022-05-18T04:16:10.7402057Z OK
2022-05-18T04:16:10.7402202Z 
2022-05-18T04:16:10.7402346Z Generating XML reports...
2022-05-18T04:16:10.7446458Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041607.xml
2022-05-18T04:16:12.0029943Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:12.0039578Z 
2022-05-18T04:16:12.0039918Z Running tests...
2022-05-18T04:16:12.0040420Z ----------------------------------------------------------------------
2022-05-18T04:16:13.6473130Z   test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:13.6927244Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17671
2022-05-18T04:16:13.7051674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17672
2022-05-18T04:16:13.7162681Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17673
2022-05-18T04:16:13.7282515Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17674
2022-05-18T04:16:14.6582838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:14.6987265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:14.7017624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:14.7037075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:15.0333229Z ok (3.029s)
2022-05-18T04:16:15.0334263Z 
2022-05-18T04:16:15.0334898Z ----------------------------------------------------------------------
2022-05-18T04:16:15.0335241Z Ran 1 test in 3.029s
2022-05-18T04:16:15.0335416Z 
2022-05-18T04:16:15.0335516Z OK
2022-05-18T04:16:15.0335656Z 
2022-05-18T04:16:15.0335800Z Generating XML reports...
2022-05-18T04:16:15.0378967Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041612.xml
2022-05-18T04:16:16.3200002Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:16.3209949Z 
2022-05-18T04:16:16.3210446Z Running tests...
2022-05-18T04:16:16.3211341Z ----------------------------------------------------------------------
2022-05-18T04:16:17.9291393Z   test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:17.9725986Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18014
2022-05-18T04:16:17.9835961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18015
2022-05-18T04:16:17.9960131Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18016
2022-05-18T04:16:18.0068425Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18017
2022-05-18T04:16:18.9524390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:18.9907866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:19.0246070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:19.0367614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:20.9160806Z ok (4.595s)
2022-05-18T04:16:20.9161047Z 
2022-05-18T04:16:20.9161506Z ----------------------------------------------------------------------
2022-05-18T04:16:20.9161865Z Ran 1 test in 4.595s
2022-05-18T04:16:20.9162056Z 
2022-05-18T04:16:20.9162157Z OK
2022-05-18T04:16:20.9162300Z 
2022-05-18T04:16:20.9162444Z Generating XML reports...
2022-05-18T04:16:20.9206680Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041616.xml
2022-05-18T04:16:22.2034739Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:22.2044110Z 
2022-05-18T04:16:22.2044439Z Running tests...
2022-05-18T04:16:22.2044911Z ----------------------------------------------------------------------
2022-05-18T04:16:23.8518152Z   test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:23.8939677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18361
2022-05-18T04:16:23.9056891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18362
2022-05-18T04:16:23.9173749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18363
2022-05-18T04:16:23.9288936Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18364
2022-05-18T04:16:24.9624418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:24.9725855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:24.9737012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:24.9802012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:25.5343556Z ok (3.330s)
2022-05-18T04:16:25.5343982Z 
2022-05-18T04:16:25.5344701Z ----------------------------------------------------------------------
2022-05-18T04:16:25.5345300Z Ran 1 test in 3.330s
2022-05-18T04:16:25.5345596Z 
2022-05-18T04:16:25.5345756Z OK
2022-05-18T04:16:25.5345988Z 
2022-05-18T04:16:25.5346245Z Generating XML reports...
2022-05-18T04:16:25.5389775Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041622.xml
2022-05-18T04:16:26.7820983Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:26.7830623Z 
2022-05-18T04:16:26.7830891Z Running tests...
2022-05-18T04:16:26.7831761Z ----------------------------------------------------------------------
2022-05-18T04:16:28.3824664Z   test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:28.4235834Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18728
2022-05-18T04:16:28.4352553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18729
2022-05-18T04:16:28.4465042Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18730
2022-05-18T04:16:28.4580731Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18731
2022-05-18T04:16:29.4433979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:29.4668960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:29.4680869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:29.4940163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:29.9633191Z ok (3.180s)
2022-05-18T04:16:29.9633433Z 
2022-05-18T04:16:29.9633872Z ----------------------------------------------------------------------
2022-05-18T04:16:29.9634201Z Ran 1 test in 3.180s
2022-05-18T04:16:29.9634378Z 
2022-05-18T04:16:29.9634495Z OK
2022-05-18T04:16:29.9634632Z 
2022-05-18T04:16:29.9634775Z Generating XML reports...
2022-05-18T04:16:29.9677632Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041626.xml
2022-05-18T04:16:31.2558719Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:31.2569863Z 
2022-05-18T04:16:31.2570163Z Running tests...
2022-05-18T04:16:31.2570631Z ----------------------------------------------------------------------
2022-05-18T04:16:32.9132435Z   test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:32.9572906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19095
2022-05-18T04:16:32.9693847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19096
2022-05-18T04:16:32.9801453Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19097
2022-05-18T04:16:32.9906906Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19098
2022-05-18T04:16:33.9048006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:33.9261765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:33.9996395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:34.0025272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:36.3006314Z ok (5.043s)
2022-05-18T04:16:36.3006570Z 
2022-05-18T04:16:36.3007005Z ----------------------------------------------------------------------
2022-05-18T04:16:36.3007350Z Ran 1 test in 5.044s
2022-05-18T04:16:36.3007521Z 
2022-05-18T04:16:36.3007619Z OK
2022-05-18T04:16:36.3010240Z 
2022-05-18T04:16:36.3010620Z Generating XML reports...
2022-05-18T04:16:36.3052009Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041631.xml
2022-05-18T04:16:37.5781420Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:37.5790986Z 
2022-05-18T04:16:37.5791314Z Running tests...
2022-05-18T04:16:37.5791963Z ----------------------------------------------------------------------
2022-05-18T04:16:39.2391650Z   test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:39.2829176Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19466
2022-05-18T04:16:39.2950794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19467
2022-05-18T04:16:39.3074679Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19468
2022-05-18T04:16:39.3185317Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19469
2022-05-18T04:16:40.2806509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:40.3134177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:40.3183876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:40.3211512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:40.6237235Z ok (3.044s)
2022-05-18T04:16:40.6237485Z 
2022-05-18T04:16:40.6237937Z ----------------------------------------------------------------------
2022-05-18T04:16:40.6238297Z Ran 1 test in 3.045s
2022-05-18T04:16:40.6238450Z 
2022-05-18T04:16:40.6238553Z OK
2022-05-18T04:16:40.6238696Z 
2022-05-18T04:16:40.6238838Z Generating XML reports...
2022-05-18T04:16:40.6283071Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041637.xml
2022-05-18T04:16:41.8721411Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:41.8730651Z 
2022-05-18T04:16:41.8730920Z Running tests...
2022-05-18T04:16:41.8731450Z ----------------------------------------------------------------------
2022-05-18T04:16:43.4904280Z   test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:43.5341266Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19809
2022-05-18T04:16:43.5456511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19810
2022-05-18T04:16:43.5575678Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19811
2022-05-18T04:16:43.5684423Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19812
2022-05-18T04:16:44.4970233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:44.5240956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:44.5465113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:44.5493291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:44.7733114Z ok (2.900s)
2022-05-18T04:16:44.7733481Z 
2022-05-18T04:16:44.7733934Z ----------------------------------------------------------------------
2022-05-18T04:16:44.7734688Z Ran 1 test in 2.900s
2022-05-18T04:16:44.7734904Z 
2022-05-18T04:16:44.7734996Z OK
2022-05-18T04:16:44.7735145Z 
2022-05-18T04:16:44.7735285Z Generating XML reports...
2022-05-18T04:16:44.7778708Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041641.xml
2022-05-18T04:16:46.0380114Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:46.0390965Z 
2022-05-18T04:16:46.0391504Z Running tests...
2022-05-18T04:16:46.0392119Z ----------------------------------------------------------------------
2022-05-18T04:16:47.6983352Z   test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:47.7430055Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20152
2022-05-18T04:16:47.7555186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20153
2022-05-18T04:16:47.7681310Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20154
2022-05-18T04:16:47.7793865Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20155
2022-05-18T04:16:48.7175211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:48.7519711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:48.7943568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:48.7955747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:50.6887722Z ok (4.649s)
2022-05-18T04:16:50.6887993Z 
2022-05-18T04:16:50.6888447Z ----------------------------------------------------------------------
2022-05-18T04:16:50.6888799Z Ran 1 test in 4.650s
2022-05-18T04:16:50.6888947Z 
2022-05-18T04:16:50.6889044Z OK
2022-05-18T04:16:50.6889182Z 
2022-05-18T04:16:50.6889422Z Generating XML reports...
2022-05-18T04:16:50.6932921Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041646.xml
2022-05-18T04:16:51.9595872Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:51.9605787Z 
2022-05-18T04:16:51.9606086Z Running tests...
2022-05-18T04:16:51.9606576Z ----------------------------------------------------------------------
2022-05-18T04:16:53.6163181Z   test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:53.6621619Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20499
2022-05-18T04:16:53.6748183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20500
2022-05-18T04:16:53.6873258Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20501
2022-05-18T04:16:53.6998542Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20502
2022-05-18T04:16:54.6400701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:54.7047016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:54.7235073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:54.7287442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:55.0050316Z ok (3.044s)
2022-05-18T04:16:55.0050648Z 
2022-05-18T04:16:55.0051101Z ----------------------------------------------------------------------
2022-05-18T04:16:55.0051483Z Ran 1 test in 3.044s
2022-05-18T04:16:55.0051655Z 
2022-05-18T04:16:55.0051750Z OK
2022-05-18T04:16:55.0051902Z 
2022-05-18T04:16:55.0052034Z Generating XML reports...
2022-05-18T04:16:55.0096393Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041651.xml
2022-05-18T04:16:56.2932401Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:56.2942323Z 
2022-05-18T04:16:56.2942595Z Running tests...
2022-05-18T04:16:56.2943663Z ----------------------------------------------------------------------
2022-05-18T04:16:57.9491736Z   test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:16:57.9935294Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20842
2022-05-18T04:16:58.0046850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20843
2022-05-18T04:16:58.0165260Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20844
2022-05-18T04:16:58.0285705Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20845
2022-05-18T04:16:58.9562849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:58.9650975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:58.9672660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:58.9908547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:59.4338107Z ok (3.139s)
2022-05-18T04:16:59.4338331Z 
2022-05-18T04:16:59.4338858Z ----------------------------------------------------------------------
2022-05-18T04:16:59.4339209Z Ran 1 test in 3.140s
2022-05-18T04:16:59.4339357Z 
2022-05-18T04:16:59.4339461Z OK
2022-05-18T04:16:59.4339597Z 
2022-05-18T04:16:59.4339738Z Generating XML reports...
2022-05-18T04:16:59.4385162Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041656.xml
2022-05-18T04:17:00.6907861Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:17:00.6917603Z 
2022-05-18T04:17:00.6917915Z Running tests...
2022-05-18T04:17:00.6918428Z ----------------------------------------------------------------------
2022-05-18T04:17:02.3194519Z   test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:02.3624851Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21209
2022-05-18T04:17:02.3739160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21210
2022-05-18T04:17:02.3853120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21211
2022-05-18T04:17:02.3967896Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21212
2022-05-18T04:17:03.4135929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:17:03.4467182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:03.4467687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:03.5853024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:17:05.8066955Z ok (5.115s)
2022-05-18T04:17:05.8067183Z 
2022-05-18T04:17:05.8067610Z ----------------------------------------------------------------------
2022-05-18T04:17:05.8067985Z Ran 1 test in 5.115s
2022-05-18T04:17:05.8068154Z 
2022-05-18T04:17:05.8068251Z OK
2022-05-18T04:17:05.8068389Z 
2022-05-18T04:17:05.8068510Z Generating XML reports...
2022-05-18T04:17:05.8112282Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041700.xml
2022-05-18T04:17:07.0630829Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:17:07.0640797Z 
2022-05-18T04:17:07.0641114Z Running tests...
2022-05-18T04:17:07.0641604Z ----------------------------------------------------------------------
2022-05-18T04:17:08.6794253Z   test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:08.7239554Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21580
2022-05-18T04:17:08.7365133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21581
2022-05-18T04:17:08.7485699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21582
2022-05-18T04:17:08.7594783Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21583
2022-05-18T04:17:09.7449156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:09.7739706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:17:09.7749457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:17:09.7860673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:10.0645881Z ok (3.000s)
2022-05-18T04:17:10.0646131Z 
2022-05-18T04:17:10.0646982Z ----------------------------------------------------------------------
2022-05-18T04:17:10.0647362Z Ran 1 test in 3.001s
2022-05-18T04:17:10.0647537Z 
2022-05-18T04:17:10.0647641Z OK
2022-05-18T04:17:10.0647756Z 
2022-05-18T04:17:10.0647904Z Generating XML reports...
2022-05-18T04:17:10.0692017Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041707.xml
2022-05-18T04:17:11.3402781Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:17:11.3413236Z 
2022-05-18T04:17:11.3413633Z Running tests...
2022-05-18T04:17:11.3414467Z ----------------------------------------------------------------------
2022-05-18T04:17:13.0029976Z   test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:13.0477168Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21923
2022-05-18T04:17:13.0602108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21924
2022-05-18T04:17:13.0711976Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21925
2022-05-18T04:17:13.0833982Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21926
2022-05-18T04:17:14.0127049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:14.0665565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:17:14.0757836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:17:14.0806466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:14.3885777Z ok (3.047s)
2022-05-18T04:17:14.3886017Z 
2022-05-18T04:17:14.3887023Z ----------------------------------------------------------------------
2022-05-18T04:17:14.3887411Z Ran 1 test in 3.047s
2022-05-18T04:17:14.3887593Z 
2022-05-18T04:17:14.3887689Z OK
2022-05-18T04:17:14.3887833Z 
2022-05-18T04:17:14.3887983Z Generating XML reports...
2022-05-18T04:17:14.3932373Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041711.xml
2022-05-18T04:17:15.6764473Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:17:15.6773399Z 
2022-05-18T04:17:15.6773741Z Running tests...
2022-05-18T04:17:15.6774238Z ----------------------------------------------------------------------
2022-05-18T04:17:17.3331767Z   test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:17.3782571Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22266
2022-05-18T04:17:17.3903490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22267
2022-05-18T04:17:17.4014375Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22268
2022-05-18T04:17:17.4136950Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22269
2022-05-18T04:17:18.3559950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:17:18.3575890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:18.3589515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:18.3636618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:17:20.3229513Z ok (4.645s)
2022-05-18T04:17:20.3229772Z 
2022-05-18T04:17:20.3230249Z ----------------------------------------------------------------------
2022-05-18T04:17:20.3230587Z Ran 1 test in 4.646s
2022-05-18T04:17:20.3230761Z 
2022-05-18T04:17:20.3230866Z OK
2022-05-18T04:17:20.3231309Z 
2022-05-18T04:17:20.3231475Z Generating XML reports...
2022-05-18T04:17:20.3277495Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041715.xml
2022-05-18T04:17:21.6281216Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:17:21.6292218Z 
2022-05-18T04:17:21.6292409Z Running tests...
2022-05-18T04:17:21.6292950Z ----------------------------------------------------------------------
2022-05-18T04:17:23.2896869Z   test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:23.3340004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22613
2022-05-18T04:17:23.3460853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22614
2022-05-18T04:17:23.3585519Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22615
2022-05-18T04:17:23.3694518Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22616
2022-05-18T04:17:24.3285250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:24.3456624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:17:24.3597604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:24.3855798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:17:24.6745741Z ok (3.045s)
2022-05-18T04:17:24.6745984Z 
2022-05-18T04:17:24.6746452Z ----------------------------------------------------------------------
2022-05-18T04:17:24.6746799Z Ran 1 test in 3.045s
2022-05-18T04:17:24.6746969Z 
2022-05-18T04:17:24.6747073Z OK
2022-05-18T04:17:24.6747217Z 
2022-05-18T04:17:24.6747386Z Generating XML reports...
2022-05-18T04:17:24.6792853Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041721.xml
2022-05-18T04:17:25.9478659Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:17:25.9488468Z 
2022-05-18T04:17:25.9488814Z Running tests...
2022-05-18T04:17:25.9489299Z ----------------------------------------------------------------------
2022-05-18T04:17:27.5495959Z   test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:27.5921858Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22956
2022-05-18T04:17:27.6030723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22957
2022-05-18T04:17:27.6148245Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22958
2022-05-18T04:17:27.6260437Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22959
2022-05-18T04:17:28.5880495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:28.5961375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:17:28.6180756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:17:28.6276293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:28.9310074Z ok (2.982s)
2022-05-18T04:17:28.9310338Z 
2022-05-18T04:17:28.9310811Z ----------------------------------------------------------------------
2022-05-18T04:17:28.9311175Z Ran 1 test in 2.982s
2022-05-18T04:17:28.9311342Z 
2022-05-18T04:17:28.9311441Z OK
2022-05-18T04:17:28.9311581Z 
2022-05-18T04:17:28.9311733Z Generating XML reports...
2022-05-18T04:17:28.9359001Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041725.xml
2022-05-18T04:17:30.1751427Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:17:30.1763112Z 
2022-05-18T04:17:30.1763768Z Running tests...
2022-05-18T04:17:30.1764628Z ----------------------------------------------------------------------
2022-05-18T04:17:31.8316015Z   test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:31.8763923Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23299
2022-05-18T04:17:31.8886684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23300
2022-05-18T04:17:31.8997062Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23301
2022-05-18T04:17:31.9116405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23302
2022-05-18T04:17:32.8416242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:17:32.8417237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:17:32.8653740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:32.8928434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:33.7179082Z ok (3.541s)
2022-05-18T04:17:33.7179353Z 
2022-05-18T04:17:33.7179808Z ----------------------------------------------------------------------
2022-05-18T04:17:33.7180216Z Ran 1 test in 3.542s
2022-05-18T04:17:33.7180364Z 
2022-05-18T04:17:33.7180490Z OK
2022-05-18T04:17:33.7180698Z 
2022-05-18T04:17:33.7180844Z Generating XML reports...
2022-05-18T04:17:33.7223739Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041730.xml
2022-05-18T04:17:34.9449748Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:17:34.9458926Z 
2022-05-18T04:17:34.9459250Z Running tests...
2022-05-18T04:17:34.9459775Z ----------------------------------------------------------------------
2022-05-18T04:17:36.5653746Z   test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:36.6073567Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23666
2022-05-18T04:17:36.6188797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23667
2022-05-18T04:17:36.6301480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23668
2022-05-18T04:17:36.6413588Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23669
2022-05-18T04:17:37.6570385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:37.6571566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:37.6681112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:17:37.7148010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:17:41.3536288Z ok (6.408s)
2022-05-18T04:17:41.3536532Z 
2022-05-18T04:17:41.3536975Z ----------------------------------------------------------------------
2022-05-18T04:17:41.3537324Z Ran 1 test in 6.408s
2022-05-18T04:17:41.3537536Z 
2022-05-18T04:17:41.3537613Z OK
2022-05-18T04:17:41.3537751Z 
2022-05-18T04:17:41.3537890Z Generating XML reports...
2022-05-18T04:17:41.3581169Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041734.xml
2022-05-18T04:17:42.6257968Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:17:42.6268431Z 
2022-05-18T04:17:42.6268845Z Running tests...
2022-05-18T04:17:42.6269346Z ----------------------------------------------------------------------
2022-05-18T04:17:44.2522433Z   test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:44.2947550Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24037
2022-05-18T04:17:44.3070593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24038
2022-05-18T04:17:44.3191223Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24039
2022-05-18T04:17:44.3311600Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24040
2022-05-18T04:17:45.3279478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:45.3475355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:17:45.3525195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:45.3913265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:17:45.7366530Z ok (3.110s)
2022-05-18T04:17:45.7367106Z 
2022-05-18T04:17:45.7368099Z ----------------------------------------------------------------------
2022-05-18T04:17:45.7368506Z Ran 1 test in 3.110s
2022-05-18T04:17:45.7368662Z 
2022-05-18T04:17:45.7368765Z OK
2022-05-18T04:17:45.7368909Z 
2022-05-18T04:17:45.7369050Z Generating XML reports...
2022-05-18T04:17:45.7411276Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041742.xml
2022-05-18T04:17:47.0196594Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:17:47.0207416Z 
2022-05-18T04:17:47.0207859Z Running tests...
2022-05-18T04:17:47.0208354Z ----------------------------------------------------------------------
2022-05-18T04:17:48.6647801Z   test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:48.7087752Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24384
2022-05-18T04:17:48.7211924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24385
2022-05-18T04:17:48.7335240Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24386
2022-05-18T04:17:48.7457599Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24387
2022-05-18T04:17:49.7647254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:49.7647827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:49.7834791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:17:49.8195910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:17:50.1509509Z ok (3.130s)
2022-05-18T04:17:50.1509767Z 
2022-05-18T04:17:50.1510217Z ----------------------------------------------------------------------
2022-05-18T04:17:50.1510558Z Ran 1 test in 3.130s
2022-05-18T04:17:50.1510738Z 
2022-05-18T04:17:50.1510840Z OK
2022-05-18T04:17:50.1511004Z 
2022-05-18T04:17:50.1513419Z Generating XML reports...
2022-05-18T04:17:50.1555276Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041747.xml
2022-05-18T04:17:51.4157168Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:17:51.4166628Z 
2022-05-18T04:17:51.4166818Z Running tests...
2022-05-18T04:17:51.4167316Z ----------------------------------------------------------------------
2022-05-18T04:17:53.0779524Z   test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:53.1224224Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24727
2022-05-18T04:17:53.1345391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24728
2022-05-18T04:17:53.1470128Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24729
2022-05-18T04:17:53.1581776Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24730
2022-05-18T04:17:54.0977670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:17:54.1020761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:54.1315363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:17:54.1617373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:56.1674901Z ok (4.750s)
2022-05-18T04:17:56.1675131Z 
2022-05-18T04:17:56.1675556Z ----------------------------------------------------------------------
2022-05-18T04:17:56.1675884Z Ran 1 test in 4.751s
2022-05-18T04:17:56.1676064Z 
2022-05-18T04:17:56.1676165Z OK
2022-05-18T04:17:56.1676306Z 
2022-05-18T04:17:56.1676448Z Generating XML reports...
2022-05-18T04:17:56.1720047Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041751.xml
2022-05-18T04:17:57.4124015Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:17:57.4134190Z 
2022-05-18T04:17:57.4134618Z Running tests...
2022-05-18T04:17:57.4135112Z ----------------------------------------------------------------------
2022-05-18T04:17:59.0637387Z   test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:17:59.1088192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25074
2022-05-18T04:17:59.1212620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25075
2022-05-18T04:17:59.1322962Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25076
2022-05-18T04:17:59.1429852Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25077
2022-05-18T04:18:00.0889403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:00.1069603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:00.1500673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:18:00.1593163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:18:00.4484363Z ok (3.035s)
2022-05-18T04:18:00.4484761Z 
2022-05-18T04:18:00.4485388Z ----------------------------------------------------------------------
2022-05-18T04:18:00.4485733Z Ran 1 test in 3.035s
2022-05-18T04:18:00.4485904Z 
2022-05-18T04:18:00.4486006Z OK
2022-05-18T04:18:00.4486147Z 
2022-05-18T04:18:00.4486292Z Generating XML reports...
2022-05-18T04:18:00.4529003Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041757.xml
2022-05-18T04:18:01.7217488Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:01.7227650Z 
2022-05-18T04:18:01.7227918Z Running tests...
2022-05-18T04:18:01.7228461Z ----------------------------------------------------------------------
2022-05-18T04:18:03.3976624Z   test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:03.4419918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25417
2022-05-18T04:18:03.4531916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25418
2022-05-18T04:18:03.4651554Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25419
2022-05-18T04:18:03.4771933Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25420
2022-05-18T04:18:04.4686545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:04.4687455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:04.4801705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:18:04.5186841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:18:04.9826777Z ok (3.260s)
2022-05-18T04:18:04.9826998Z 
2022-05-18T04:18:04.9827430Z ----------------------------------------------------------------------
2022-05-18T04:18:04.9828181Z Ran 1 test in 3.260s
2022-05-18T04:18:04.9828349Z 
2022-05-18T04:18:04.9828444Z OK
2022-05-18T04:18:04.9828579Z 
2022-05-18T04:18:04.9828718Z Generating XML reports...
2022-05-18T04:18:04.9871367Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041801.xml
2022-05-18T04:18:06.2401812Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:06.2412135Z 
2022-05-18T04:18:06.2412334Z Running tests...
2022-05-18T04:18:06.2413200Z ----------------------------------------------------------------------
2022-05-18T04:18:07.8834444Z   test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:07.9266059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25784
2022-05-18T04:18:07.9377372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25785
2022-05-18T04:18:07.9483454Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25786
2022-05-18T04:18:07.9601090Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25787
2022-05-18T04:18:08.9043830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:18:08.9053633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:08.9273347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:08.9384542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:18:11.6705252Z ok (5.429s)
2022-05-18T04:18:11.6705493Z 
2022-05-18T04:18:11.6706039Z ----------------------------------------------------------------------
2022-05-18T04:18:11.6706511Z Ran 1 test in 5.429s
2022-05-18T04:18:11.6706658Z 
2022-05-18T04:18:11.6706752Z OK
2022-05-18T04:18:11.6706890Z 
2022-05-18T04:18:11.6707031Z Generating XML reports...
2022-05-18T04:18:11.6751112Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041806.xml
2022-05-18T04:18:12.9750685Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:12.9762013Z 
2022-05-18T04:18:12.9762446Z Running tests...
2022-05-18T04:18:12.9762944Z ----------------------------------------------------------------------
2022-05-18T04:18:14.6183976Z   test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:14.6640388Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26155
2022-05-18T04:18:14.6765431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26156
2022-05-18T04:18:14.6894625Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26157
2022-05-18T04:18:14.7008409Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26158
2022-05-18T04:18:15.6522572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:15.6558982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:15.6960824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:18:15.6970957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:18:16.0062777Z ok (3.030s)
2022-05-18T04:18:16.0063155Z 
2022-05-18T04:18:16.0063620Z ----------------------------------------------------------------------
2022-05-18T04:18:16.0063977Z Ran 1 test in 3.030s
2022-05-18T04:18:16.0064164Z 
2022-05-18T04:18:16.0064245Z OK
2022-05-18T04:18:16.0064390Z 
2022-05-18T04:18:16.0064525Z Generating XML reports...
2022-05-18T04:18:16.0107071Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041812.xml
2022-05-18T04:18:17.2751403Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:17.2762749Z 
2022-05-18T04:18:17.2762947Z Running tests...
2022-05-18T04:18:17.2763427Z ----------------------------------------------------------------------
2022-05-18T04:18:18.9263137Z   test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:18.9697932Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26510
2022-05-18T04:18:18.9819719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26511
2022-05-18T04:18:18.9945485Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26512
2022-05-18T04:18:19.0054829Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26513
2022-05-18T04:18:19.9695893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:19.9733184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:19.9821384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:18:19.9952520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:18:20.5111445Z ok (3.235s)
2022-05-18T04:18:20.5111674Z 
2022-05-18T04:18:20.5112119Z ----------------------------------------------------------------------
2022-05-18T04:18:20.5112486Z Ran 1 test in 3.235s
2022-05-18T04:18:20.5112653Z 
2022-05-18T04:18:20.5112730Z OK
2022-05-18T04:18:20.5112868Z 
2022-05-18T04:18:20.5113009Z Generating XML reports...
2022-05-18T04:18:20.5155150Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041817.xml
2022-05-18T04:18:21.7600789Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:21.7611396Z 
2022-05-18T04:18:21.7611602Z Running tests...
2022-05-18T04:18:21.7612093Z ----------------------------------------------------------------------
2022-05-18T04:18:23.4082369Z   test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:23.4530888Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26889
2022-05-18T04:18:23.4653865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26890
2022-05-18T04:18:23.4765215Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26891
2022-05-18T04:18:23.4886057Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26892
2022-05-18T04:18:24.4994009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:24.5216028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:18:24.5224651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:18:24.5393503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:24.7940291Z ok (3.033s)
2022-05-18T04:18:24.7940565Z 
2022-05-18T04:18:24.7940993Z ----------------------------------------------------------------------
2022-05-18T04:18:24.7941363Z Ran 1 test in 3.033s
2022-05-18T04:18:24.7941539Z 
2022-05-18T04:18:24.7941633Z OK
2022-05-18T04:18:24.7944710Z 
2022-05-18T04:18:24.7945012Z Generating XML reports...
2022-05-18T04:18:24.7984036Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041821.xml
2022-05-18T04:18:26.0568659Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:26.0578405Z 
2022-05-18T04:18:26.0579236Z Running tests...
2022-05-18T04:18:26.0579707Z ----------------------------------------------------------------------
2022-05-18T04:18:27.7105718Z   test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:27.7556788Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27232
2022-05-18T04:18:27.7678701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27233
2022-05-18T04:18:27.7790467Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27234
2022-05-18T04:18:27.7912363Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27235
2022-05-18T04:18:28.7268858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:28.7344079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:28.7671122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:18:28.7705883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:18:30.7002712Z ok (4.642s)
2022-05-18T04:18:30.7002944Z 
2022-05-18T04:18:30.7003373Z ----------------------------------------------------------------------
2022-05-18T04:18:30.7003766Z Ran 1 test in 4.643s
2022-05-18T04:18:30.7003936Z 
2022-05-18T04:18:30.7004036Z OK
2022-05-18T04:18:30.7004171Z 
2022-05-18T04:18:30.7004313Z Generating XML reports...
2022-05-18T04:18:30.7051333Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041826.xml
2022-05-18T04:18:31.9926449Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:31.9936505Z 
2022-05-18T04:18:31.9936917Z Running tests...
2022-05-18T04:18:31.9937444Z ----------------------------------------------------------------------
2022-05-18T04:18:33.6388076Z   test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:33.6823650Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27579
2022-05-18T04:18:33.6944942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27580
2022-05-18T04:18:33.7074576Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27581
2022-05-18T04:18:33.7185507Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27582
2022-05-18T04:18:34.7257341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:18:34.7288991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:34.7312474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:34.7338820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:18:35.0236693Z ok (3.030s)
2022-05-18T04:18:35.0236963Z 
2022-05-18T04:18:35.0237392Z ----------------------------------------------------------------------
2022-05-18T04:18:35.0237765Z Ran 1 test in 3.030s
2022-05-18T04:18:35.0237913Z 
2022-05-18T04:18:35.0238014Z OK
2022-05-18T04:18:35.0238150Z 
2022-05-18T04:18:35.0238320Z Generating XML reports...
2022-05-18T04:18:35.0281754Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041831.xml
2022-05-18T04:18:36.2955135Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:36.2965564Z 
2022-05-18T04:18:36.2965761Z Running tests...
2022-05-18T04:18:36.2966563Z ----------------------------------------------------------------------
2022-05-18T04:18:37.9262855Z   test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:37.9695122Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27922
2022-05-18T04:18:37.9813454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27923
2022-05-18T04:18:37.9936705Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27924
2022-05-18T04:18:38.0045223Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27925
2022-05-18T04:18:38.9251467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:38.9449896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:39.0049974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:18:39.0075713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:18:39.9106900Z ok (3.614s)
2022-05-18T04:18:39.9107178Z 
2022-05-18T04:18:39.9107806Z ----------------------------------------------------------------------
2022-05-18T04:18:39.9108677Z Ran 1 test in 3.614s
2022-05-18T04:18:39.9108883Z 
2022-05-18T04:18:39.9108985Z OK
2022-05-18T04:18:39.9109134Z 
2022-05-18T04:18:39.9109252Z Generating XML reports...
2022-05-18T04:18:39.9151879Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041836.xml
2022-05-18T04:18:41.1621818Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:41.1633213Z 
2022-05-18T04:18:41.1633674Z Running tests...
2022-05-18T04:18:41.1639899Z ----------------------------------------------------------------------
2022-05-18T04:18:41.1641083Z   test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip: Test is flaky, see https://github.com/pytorch/pytorch/issues/15963 (0.001s)
2022-05-18T04:18:41.1641522Z 
2022-05-18T04:18:41.1641807Z ----------------------------------------------------------------------
2022-05-18T04:18:41.1642127Z Ran 1 test in 0.001s
2022-05-18T04:18:41.1642286Z 
2022-05-18T04:18:41.1642421Z OK (skipped=1)
2022-05-18T04:18:41.1642579Z 
2022-05-18T04:18:41.1642705Z Generating XML reports...
2022-05-18T04:18:41.1678482Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041841.xml
2022-05-18T04:18:42.2577911Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:42.2587937Z 
2022-05-18T04:18:42.2588132Z Running tests...
2022-05-18T04:18:42.2588624Z ----------------------------------------------------------------------
2022-05-18T04:18:43.9201871Z   test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:43.9638549Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28354
2022-05-18T04:18:43.9753137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28355
2022-05-18T04:18:43.9856564Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28356
2022-05-18T04:18:43.9971637Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28357
2022-05-18T04:18:45.0208594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:45.0218955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:18:45.0260928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:18:45.0292348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:45.3021016Z ok (3.043s)
2022-05-18T04:18:45.3021212Z 
2022-05-18T04:18:45.3021652Z ----------------------------------------------------------------------
2022-05-18T04:18:45.3022387Z Ran 1 test in 3.043s
2022-05-18T04:18:45.3022540Z 
2022-05-18T04:18:45.3022643Z OK
2022-05-18T04:18:45.3022780Z 
2022-05-18T04:18:45.3022918Z Generating XML reports...
2022-05-18T04:18:45.3068771Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041842.xml
2022-05-18T04:18:46.5500188Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:46.5509670Z 
2022-05-18T04:18:46.5510072Z Running tests...
2022-05-18T04:18:46.5510694Z ----------------------------------------------------------------------
2022-05-18T04:18:46.5513192Z   test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip: intermittent failures on Windows, in CI (0.000s)
2022-05-18T04:18:46.5513519Z 
2022-05-18T04:18:46.5513826Z ----------------------------------------------------------------------
2022-05-18T04:18:46.5514146Z Ran 1 test in 0.001s
2022-05-18T04:18:46.5514322Z 
2022-05-18T04:18:46.5514440Z OK (skipped=1)
2022-05-18T04:18:46.5514602Z 
2022-05-18T04:18:46.5514739Z Generating XML reports...
2022-05-18T04:18:46.5549234Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041846.xml
2022-05-18T04:18:47.6735051Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:47.6746152Z 
2022-05-18T04:18:47.6746458Z Running tests...
2022-05-18T04:18:47.6746961Z ----------------------------------------------------------------------
2022-05-18T04:18:49.3477864Z   test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:49.3917487Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28762
2022-05-18T04:18:49.4027114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28763
2022-05-18T04:18:49.4146531Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28764
2022-05-18T04:18:49.4269860Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28765
2022-05-18T04:18:50.3580545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:50.3630193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:50.3941228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:18:50.4082988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:18:52.5367064Z ok (4.862s)
2022-05-18T04:18:52.5367306Z 
2022-05-18T04:18:52.5367750Z ----------------------------------------------------------------------
2022-05-18T04:18:52.5368112Z Ran 1 test in 4.862s
2022-05-18T04:18:52.5368263Z 
2022-05-18T04:18:52.5368364Z OK
2022-05-18T04:18:52.5368506Z 
2022-05-18T04:18:52.5369905Z Generating XML reports...
2022-05-18T04:18:52.5412860Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041847.xml
2022-05-18T04:18:53.7915661Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:53.7925726Z 
2022-05-18T04:18:53.7926016Z Running tests...
2022-05-18T04:18:53.7927138Z ----------------------------------------------------------------------
2022-05-18T04:18:55.4195331Z   test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:18:55.4625943Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29481
2022-05-18T04:18:55.4744601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29482
2022-05-18T04:18:55.4863883Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29483
2022-05-18T04:18:55.4985009Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29484
2022-05-18T04:18:56.4632092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:56.4903080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:56.5469726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:18:56.5480738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:18:56.8035439Z ok (3.011s)
2022-05-18T04:18:56.8035665Z 
2022-05-18T04:18:56.8036259Z ----------------------------------------------------------------------
2022-05-18T04:18:56.8036626Z Ran 1 test in 3.011s
2022-05-18T04:18:56.8037086Z 
2022-05-18T04:18:56.8037196Z OK
2022-05-18T04:18:56.8037340Z 
2022-05-18T04:18:56.8037454Z Generating XML reports...
2022-05-18T04:18:56.8079943Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041853.xml
2022-05-18T04:18:58.0531631Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:58.0542016Z 
2022-05-18T04:18:58.0542344Z Running tests...
2022-05-18T04:18:58.0543316Z ----------------------------------------------------------------------
2022-05-18T04:18:58.0645615Z   test_forward_backward (__main__.ReducerTest) ... ok (0.010s)
2022-05-18T04:18:58.0662350Z 
2022-05-18T04:18:58.0663616Z ----------------------------------------------------------------------
2022-05-18T04:18:58.0663988Z Ran 1 test in 0.012s
2022-05-18T04:18:58.0664158Z 
2022-05-18T04:18:58.0664260Z OK
2022-05-18T04:18:58.0664406Z 
2022-05-18T04:18:58.0664544Z Generating XML reports...
2022-05-18T04:18:58.0700588Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041858.xml
2022-05-18T04:18:59.1268093Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:18:59.1279050Z 
2022-05-18T04:18:59.1279411Z Running tests...
2022-05-18T04:18:59.1279905Z ----------------------------------------------------------------------
2022-05-18T04:18:59.1419108Z   test_forward_backward_optimizer (__main__.ReducerTest) ... [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:18:59.1442911Z ok (0.016s)
2022-05-18T04:18:59.1503517Z 
2022-05-18T04:18:59.1504117Z ----------------------------------------------------------------------
2022-05-18T04:18:59.1504789Z Ran 1 test in 0.022s
2022-05-18T04:18:59.1505143Z 
2022-05-18T04:18:59.1505333Z OK
2022-05-18T04:18:59.1505597Z 
2022-05-18T04:18:59.1505792Z Generating XML reports...
2022-05-18T04:18:59.1539603Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041859.xml
2022-05-18T04:19:00.2145812Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:19:00.2156366Z 
2022-05-18T04:19:00.2156768Z Running tests...
2022-05-18T04:19:00.2157246Z ----------------------------------------------------------------------
2022-05-18T04:19:00.2267298Z   test_forward_backward_unused_parameters (__main__.ReducerTest) ... ok (0.011s)
2022-05-18T04:19:00.2278352Z 
2022-05-18T04:19:00.2278743Z ----------------------------------------------------------------------
2022-05-18T04:19:00.2279196Z Ran 1 test in 0.012s
2022-05-18T04:19:00.2279496Z 
2022-05-18T04:19:00.2279630Z OK
2022-05-18T04:19:00.2279784Z 
2022-05-18T04:19:00.2279925Z Generating XML reports...
2022-05-18T04:19:00.2314572Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041900.xml
2022-05-18T04:19:01.3309483Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:19:01.3320700Z 
2022-05-18T04:19:01.3321124Z Running tests...
2022-05-18T04:19:01.3321610Z ----------------------------------------------------------------------
2022-05-18T04:19:01.3387022Z   test_multi_dtype_multi_bucket (__main__.ReducerTest) ... ok (0.007s)
2022-05-18T04:19:01.3440886Z 
2022-05-18T04:19:01.3441515Z ----------------------------------------------------------------------
2022-05-18T04:19:01.3442052Z Ran 1 test in 0.012s
2022-05-18T04:19:01.3442211Z 
2022-05-18T04:19:01.3442313Z OK
2022-05-18T04:19:01.3442719Z 
2022-05-18T04:19:01.3442856Z Generating XML reports...
2022-05-18T04:19:01.3479665Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041901.xml
2022-05-18T04:19:02.4295726Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:19:02.4308792Z 
2022-05-18T04:19:02.4309195Z Running tests...
2022-05-18T04:19:02.4309735Z ----------------------------------------------------------------------
2022-05-18T04:19:02.4406145Z   test_multi_dtype_single_bucket (__main__.ReducerTest) ... ok (0.010s)
2022-05-18T04:19:02.4429795Z 
2022-05-18T04:19:02.4430550Z ----------------------------------------------------------------------
2022-05-18T04:19:02.4430966Z Ran 1 test in 0.012s
2022-05-18T04:19:02.4431141Z 
2022-05-18T04:19:02.4431240Z OK
2022-05-18T04:19:02.4431377Z 
2022-05-18T04:19:02.4431510Z Generating XML reports...
2022-05-18T04:19:02.4465249Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041902.xml
2022-05-18T04:19:03.5545259Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:19:03.5556038Z 
2022-05-18T04:19:03.5556418Z Running tests...
2022-05-18T04:19:03.5556887Z ----------------------------------------------------------------------
2022-05-18T04:19:03.5618111Z   test_single_dtype_single_bucket (__main__.ReducerTest) ... ok (0.006s)
2022-05-18T04:19:03.5673480Z 
2022-05-18T04:19:03.5674146Z ----------------------------------------------------------------------
2022-05-18T04:19:03.5674692Z Ran 1 test in 0.012s
2022-05-18T04:19:03.5674845Z 
2022-05-18T04:19:03.5674957Z OK
2022-05-18T04:19:03.5675097Z 
2022-05-18T04:19:03.5675228Z Generating XML reports...
2022-05-18T04:19:03.5712564Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041903.xml
2022-05-18T04:19:04.6882020Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:19:04.6891423Z 
2022-05-18T04:19:04.6891816Z Running tests...
2022-05-18T04:19:04.6892314Z ----------------------------------------------------------------------
2022-05-18T04:19:06.3295471Z   test_logging_init (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:06.3483050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:06.3483927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T04:19:06.3579884Z ok (1.669s)
2022-05-18T04:19:06.3580074Z 
2022-05-18T04:19:06.3580413Z ----------------------------------------------------------------------
2022-05-18T04:19:06.3580761Z Ran 1 test in 1.669s
2022-05-18T04:19:06.3580929Z 
2022-05-18T04:19:06.3581015Z OK
2022-05-18T04:19:06.3581154Z 
2022-05-18T04:19:06.3581285Z Generating XML reports...
2022-05-18T04:19:06.3620070Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220518041904.xml
2022-05-18T04:19:07.6065402Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:19:07.6076022Z 
2022-05-18T04:19:07.6076181Z Running tests...
2022-05-18T04:19:07.6076682Z ----------------------------------------------------------------------
2022-05-18T04:19:09.2643123Z   test_default_store_timeout_gloo (__main__.TimeoutTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:09.2810486Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74714 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (1.673s)
2022-05-18T04:19:09.2811086Z 
2022-05-18T04:19:09.2811370Z ----------------------------------------------------------------------
2022-05-18T04:19:09.2811694Z Ran 1 test in 1.673s
2022-05-18T04:19:09.2812137Z 
2022-05-18T04:19:09.2812249Z OK (skipped=1)
2022-05-18T04:19:09.2812406Z 
2022-05-18T04:19:09.2812533Z Generating XML reports...
2022-05-18T04:19:09.2845250Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220518041907.xml
2022-05-18T04:19:11.1862282Z 
2022-05-18T04:19:11.1863136Z real	8m20.333s
2022-05-18T04:19:11.1863474Z user	16m43.173s
2022-05-18T04:19:11.1863733Z sys	24m2.682s
2022-05-18T04:19:11.1864336Z + python test/run_test.py --verbose -i distributed/test_c10d_nccl
2022-05-18T04:19:20.6468282Z Ignoring disabled issues:  []
2022-05-18T04:19:20.6599778Z /var/lib/jenkins/workspace/test/run_test.py:894: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-05-18T04:19:20.6600367Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) == "11.6":
2022-05-18T04:19:20.6600735Z Selected tests:
2022-05-18T04:19:20.6601002Z  distributed/test_c10d_nccl
2022-05-18T04:19:20.6712324Z Prioritized test from test file changes.
2022-05-18T04:19:20.6713757Z reordering tests for PR:
2022-05-18T04:19:20.6714130Z prioritized: []
2022-05-18T04:19:20.6714763Z the rest: ['distributed/test_c10d_nccl']
2022-05-18T04:19:20.6715082Z 
2022-05-18T04:19:20.6721028Z Running distributed/test_c10d_nccl ... [2022-05-18 04:19:20.671700]
2022-05-18T04:19:20.6721732Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:19:20.671779]
2022-05-18T04:19:21.6857244Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_all_reduce_coalesced_nccl>, <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>]>
2022-05-18T04:19:21.6860746Z test_all_reduce_coalesced_nccl (__main__.CommTest)
2022-05-18T04:19:21.6861360Z test_broadcast_coalesced_nccl (__main__.CommTest)
2022-05-18T04:19:21.6862651Z test_nccl_barrier (__main__.CommTest)
2022-05-18T04:19:21.6863108Z test_nccl_barrier_device_ids (__main__.CommTest)
2022-05-18T04:19:21.6863463Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest)
2022-05-18T04:19:21.6863855Z test_nccl_barrier_timeout (__main__.CommTest)
2022-05-18T04:19:21.6864213Z test_nccl_barrier_timeout_new_group (__main__.CommTest)
2022-05-18T04:19:21.6864704Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest)
2022-05-18T04:19:21.6865397Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest)
2022-05-18T04:19:21.6866047Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest)
2022-05-18T04:19:21.6866698Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest)
2022-05-18T04:19:21.6867298Z test_pass_nccl_options_high_priority_stream (__main__.CommTest)
2022-05-18T04:19:21.6867991Z test_sequence_num_incremented_nccl_default (__main__.CommTest)
2022-05-18T04:19:21.6868685Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest)
2022-05-18T04:19:21.6869186Z test_sequence_num_set_default_pg_nccl (__main__.CommTest)
2022-05-18T04:19:21.6869553Z test_sequence_num_set_nccl_new_group (__main__.CommTest)
2022-05-18T04:19:21.6885562Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module>, <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-05-18T04:19:21.6901560Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6902916Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6904066Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6904704Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6905173Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6905622Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6906037Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6906499Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6906984Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6907452Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6907953Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6908505Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6909027Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6909546Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6910059Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6910553Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6911022Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6911535Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6912060Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6912577Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6913210Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6913703Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6914175Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6914673Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6915137Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6915691Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6916159Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6916579Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6917014Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6917466Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6917902Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6918339Z test_failure_recovery (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6918794Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6919258Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6919735Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6920239Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6920765Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6921258Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6921694Z test_fp16 (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6922103Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6922524Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6922950Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6923409Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6923887Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6924304Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6924760Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6925241Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6925724Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6926233Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6926706Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6927141Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6927579Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6928068Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6928570Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6929047Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6929523Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6929935Z test_no_grad (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6930454Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6930856Z test_pass_default_pg (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6931268Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6931721Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6932151Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6932582Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-05-18T04:19:21.6932960Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:19:21.6934297Z <unittest.suite.TestSuite tests=[<__main__.NcclErrorHandlingTest testMethod=test_invalid_nccl_blocking_wait_env>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]>
2022-05-18T04:19:21.6935578Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest)
2022-05-18T04:19:21.6935990Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest)
2022-05-18T04:19:21.6936376Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest)
2022-05-18T04:19:21.6936779Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest)
2022-05-18T04:19:21.6937185Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest)
2022-05-18T04:19:21.6937572Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest)
2022-05-18T04:19:21.6937969Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest)
2022-05-18T04:19:21.6938365Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest)
2022-05-18T04:19:21.6938728Z test_nccl_timeout (__main__.NcclErrorHandlingTest)
2022-05-18T04:19:21.6939168Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLNoGPUTest testMethod=test_init_no_gpus>]>
2022-05-18T04:19:21.6939624Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest)
2022-05-18T04:19:21.6941501Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>]>
2022-05-18T04:19:21.6944056Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6944433Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6944797Z test_allgather_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6945156Z test_allreduce_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6945484Z test_barrier (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6945828Z test_broadcast_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6946272Z test_empty_tensors (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6946611Z test_gather_checks (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6946955Z test_gather_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6947453Z test_gather_stress (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6947773Z test_reduce_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6948310Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6948696Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6949172Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6949527Z test_scatter_checks (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6949876Z test_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6950233Z test_scatter_stress (__main__.ProcessGroupNCCLTest)
2022-05-18T04:19:21.6950647Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_common_errors>]>
2022-05-18T04:19:21.6951214Z test_common_errors (__main__.RendezvousEnvTest)
2022-05-18T04:19:21.6951526Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:19:21.6951915Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_nccl>]>
2022-05-18T04:19:21.6952501Z test_default_store_timeout_nccl (__main__.TimeoutTest)
2022-05-18T04:19:22.6301402Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:19:22.6314167Z 
2022-05-18T04:19:22.6315276Z Running tests...
2022-05-18T04:19:22.6316210Z ----------------------------------------------------------------------
2022-05-18T04:19:24.3053443Z   test_all_reduce_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:24.3494694Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30609
2022-05-18T04:19:24.3612817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30610
2022-05-18T04:19:25.3320171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:25.3627909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:27.0742812Z ok (4.438s)
2022-05-18T04:19:27.0743066Z 
2022-05-18T04:19:27.0743547Z ----------------------------------------------------------------------
2022-05-18T04:19:27.0743880Z Ran 1 test in 4.438s
2022-05-18T04:19:27.0744049Z 
2022-05-18T04:19:27.0744148Z OK
2022-05-18T04:19:27.0744284Z 
2022-05-18T04:19:27.0744424Z Generating XML reports...
2022-05-18T04:19:27.0745039Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041922.xml
2022-05-18T04:19:28.3807492Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:19:28.3817359Z 
2022-05-18T04:19:28.3817816Z Running tests...
2022-05-18T04:19:28.3818417Z ----------------------------------------------------------------------
2022-05-18T04:19:30.0757362Z   test_broadcast_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:30.1186645Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30821
2022-05-18T04:19:30.1312357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30822
2022-05-18T04:19:31.0998159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:31.1004314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:32.8392081Z ok (4.457s)
2022-05-18T04:19:32.8392302Z 
2022-05-18T04:19:32.8392722Z ----------------------------------------------------------------------
2022-05-18T04:19:32.8393091Z Ran 1 test in 4.457s
2022-05-18T04:19:32.8393260Z 
2022-05-18T04:19:32.8393356Z OK
2022-05-18T04:19:32.8393491Z 
2022-05-18T04:19:32.8393628Z Generating XML reports...
2022-05-18T04:19:32.8438201Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041928.xml
2022-05-18T04:19:34.1306692Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:19:34.1317708Z 
2022-05-18T04:19:34.1317954Z Running tests...
2022-05-18T04:19:34.1318456Z ----------------------------------------------------------------------
2022-05-18T04:19:35.7836415Z   test_nccl_barrier (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:35.8267081Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31033
2022-05-18T04:19:35.8387792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31034
2022-05-18T04:19:36.7733825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:36.7734392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:36.7782947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:36.7784198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:36.7785517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:36.7838265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:38.1890253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:19:38.1890878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:19:38.1891772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:19:38.1990548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:19:38.2251883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:19:38.2252468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:19:38.2253312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:19:38.2355208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-05-18T04:19:38.2355814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:19:38.2520361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:19:38.2521086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:19:38.2564219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-05-18T04:19:38.5468171Z ok (4.415s)
2022-05-18T04:19:38.5468417Z 
2022-05-18T04:19:38.5468828Z ----------------------------------------------------------------------
2022-05-18T04:19:38.5469180Z Ran 1 test in 4.415s
2022-05-18T04:19:38.5469347Z 
2022-05-18T04:19:38.5469443Z OK
2022-05-18T04:19:38.5469580Z 
2022-05-18T04:19:38.5469722Z Generating XML reports...
2022-05-18T04:19:38.5514879Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041934.xml
2022-05-18T04:19:39.8118853Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:19:39.8129236Z 
2022-05-18T04:19:39.8129667Z Running tests...
2022-05-18T04:19:39.8130436Z ----------------------------------------------------------------------
2022-05-18T04:19:41.4340631Z   test_nccl_barrier_device_ids (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:41.4756537Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31268
2022-05-18T04:19:41.4868770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31269
2022-05-18T04:19:42.4590758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:42.4591337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:42.4932411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:42.4933567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:42.4934445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:42.5003015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:44.1947063Z ok (4.381s)
2022-05-18T04:19:44.1947321Z 
2022-05-18T04:19:44.1947777Z ----------------------------------------------------------------------
2022-05-18T04:19:44.1948135Z Ran 1 test in 4.382s
2022-05-18T04:19:44.1948312Z 
2022-05-18T04:19:44.1948420Z OK
2022-05-18T04:19:44.1948563Z 
2022-05-18T04:19:44.1948713Z Generating XML reports...
2022-05-18T04:19:44.1992298Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041939.xml
2022-05-18T04:19:45.4626046Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:19:45.4635390Z 
2022-05-18T04:19:45.4635780Z Running tests...
2022-05-18T04:19:45.4636290Z ----------------------------------------------------------------------
2022-05-18T04:19:47.1078738Z   test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:47.1499672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31480
2022-05-18T04:19:47.1611657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31481
2022-05-18T04:19:48.1050409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:48.1051021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:48.1108197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:48.1110118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:48.1111039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:48.1155675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:19:48.2651177Z ok (2.801s)
2022-05-18T04:19:48.2651623Z 
2022-05-18T04:19:48.2652737Z ----------------------------------------------------------------------
2022-05-18T04:19:48.2653136Z Ran 1 test in 2.802s
2022-05-18T04:19:48.2653315Z 
2022-05-18T04:19:48.2653422Z OK
2022-05-18T04:19:48.2653569Z 
2022-05-18T04:19:48.2653683Z Generating XML reports...
2022-05-18T04:19:48.2698382Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041945.xml
2022-05-18T04:19:49.5303565Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:19:49.5314831Z 
2022-05-18T04:19:49.5315126Z Running tests...
2022-05-18T04:19:49.5315636Z ----------------------------------------------------------------------
2022-05-18T04:19:51.2063411Z   test_nccl_barrier_timeout (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:19:51.2508651Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31683
2022-05-18T04:19:51.2628345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31684
2022-05-18T04:19:52.2155415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:52.2497065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:02.4874937Z ok (12.956s)
2022-05-18T04:20:02.4875146Z 
2022-05-18T04:20:02.4875591Z ----------------------------------------------------------------------
2022-05-18T04:20:02.4875952Z Ran 1 test in 12.956s
2022-05-18T04:20:02.4876117Z 
2022-05-18T04:20:02.4876213Z OK
2022-05-18T04:20:02.4876763Z 
2022-05-18T04:20:02.4876898Z Generating XML reports...
2022-05-18T04:20:02.4920793Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041949.xml
2022-05-18T04:20:03.7891645Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:20:03.7902858Z 
2022-05-18T04:20:03.7903257Z Running tests...
2022-05-18T04:20:03.7903753Z ----------------------------------------------------------------------
2022-05-18T04:20:05.4579560Z   test_nccl_barrier_timeout_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:05.5016967Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31886
2022-05-18T04:20:05.5137089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31887
2022-05-18T04:20:06.4738409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:06.4758670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:09.1330351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:09.1350849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:09.1351701Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:09.1431886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:10.1737557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:20:11.4290989Z ok (7.639s)
2022-05-18T04:20:11.4291349Z 
2022-05-18T04:20:11.4292200Z ----------------------------------------------------------------------
2022-05-18T04:20:11.4292722Z Ran 1 test in 7.639s
2022-05-18T04:20:11.4292892Z 
2022-05-18T04:20:11.4292967Z OK
2022-05-18T04:20:11.4293102Z 
2022-05-18T04:20:11.4293244Z Generating XML reports...
2022-05-18T04:20:11.4337494Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042003.xml
2022-05-18T04:20:12.7021602Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:20:12.7031617Z 
2022-05-18T04:20:12.7031966Z Running tests...
2022-05-18T04:20:12.7032492Z ----------------------------------------------------------------------
2022-05-18T04:20:14.3347665Z   test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:14.3778976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32110
2022-05-18T04:20:14.3893370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32111
2022-05-18T04:20:15.3137049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:15.3531325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:18.0081058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:18.0100805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:18.0101693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:18.0182548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:19.0220101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:20:20.3036191Z ok (7.600s)
2022-05-18T04:20:20.3036488Z 
2022-05-18T04:20:20.3036937Z ----------------------------------------------------------------------
2022-05-18T04:20:20.3037289Z Ran 1 test in 7.600s
2022-05-18T04:20:20.3037468Z 
2022-05-18T04:20:20.3037564Z OK
2022-05-18T04:20:20.3037685Z 
2022-05-18T04:20:20.3038213Z Generating XML reports...
2022-05-18T04:20:20.3080928Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042012.xml
2022-05-18T04:20:21.6055860Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:20:21.6066172Z 
2022-05-18T04:20:21.6066568Z Running tests...
2022-05-18T04:20:21.6067053Z ----------------------------------------------------------------------
2022-05-18T04:20:23.2649994Z   test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:23.3092138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32325
2022-05-18T04:20:23.3201923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32326
2022-05-18T04:20:24.2764371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:24.3215373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:24.3378441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:24.3379203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:24.3380055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:24.3387937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:24.3481968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:24.3482568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:24.3483242Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:20:24.3491928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:20:26.1284774Z ok (4.522s)
2022-05-18T04:20:26.1285037Z 
2022-05-18T04:20:26.1285468Z ----------------------------------------------------------------------
2022-05-18T04:20:26.1285827Z Ran 1 test in 4.522s
2022-05-18T04:20:26.1286005Z 
2022-05-18T04:20:26.1286111Z OK
2022-05-18T04:20:26.1286255Z 
2022-05-18T04:20:26.1286399Z Generating XML reports...
2022-05-18T04:20:26.1330894Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042021.xml
2022-05-18T04:20:27.4315044Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:20:27.4324651Z 
2022-05-18T04:20:27.4324963Z Running tests...
2022-05-18T04:20:27.4325459Z ----------------------------------------------------------------------
2022-05-18T04:20:29.0928310Z   test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:29.1348226Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32552
2022-05-18T04:20:29.1465120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32553
2022-05-18T04:20:30.0902583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:30.0903689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:30.1166715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:30.1168454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:30.1169296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:30.1170026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:30.1212078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:30.1212663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:30.1213331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:20:30.1273276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:20:31.8545863Z ok (4.422s)
2022-05-18T04:20:31.8546071Z 
2022-05-18T04:20:31.8546516Z ----------------------------------------------------------------------
2022-05-18T04:20:31.8546902Z Ran 1 test in 4.422s
2022-05-18T04:20:31.8547072Z 
2022-05-18T04:20:31.8547168Z OK
2022-05-18T04:20:31.8547305Z 
2022-05-18T04:20:31.8547441Z Generating XML reports...
2022-05-18T04:20:31.8591246Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042027.xml
2022-05-18T04:20:33.0959737Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:20:33.0970007Z 
2022-05-18T04:20:33.0970338Z Running tests...
2022-05-18T04:20:33.0970836Z ----------------------------------------------------------------------
2022-05-18T04:20:34.7377338Z   test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:34.7827175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32770
2022-05-18T04:20:34.7946403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32771
2022-05-18T04:20:35.7613605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:35.7614729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:35.7758573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:35.7759688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:35.7760661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:35.7761223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:35.7822522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:35.7823444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:35.7824126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:20:35.7867182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:20:37.6032888Z ok (4.506s)
2022-05-18T04:20:37.6033265Z 
2022-05-18T04:20:37.6034070Z ----------------------------------------------------------------------
2022-05-18T04:20:37.6034629Z Ran 1 test in 4.506s
2022-05-18T04:20:37.6034796Z 
2022-05-18T04:20:37.6034892Z OK
2022-05-18T04:20:37.6035387Z 
2022-05-18T04:20:37.6035532Z Generating XML reports...
2022-05-18T04:20:37.6078135Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042033.xml
2022-05-18T04:20:38.8977344Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:20:38.8987555Z 
2022-05-18T04:20:38.8987747Z Running tests...
2022-05-18T04:20:38.8988245Z ----------------------------------------------------------------------
2022-05-18T04:20:40.5316687Z   test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:40.5760077Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32988
2022-05-18T04:20:40.5865019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32989
2022-05-18T04:20:41.5432158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:41.5437649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:41.5631739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:41.5632286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:41.5633159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:41.5634870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:41.5637264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:41.5641923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:41.5642708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:20:41.5741530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:20:43.2944910Z ok (4.395s)
2022-05-18T04:20:43.2945423Z 
2022-05-18T04:20:43.2946100Z ----------------------------------------------------------------------
2022-05-18T04:20:43.2946429Z Ran 1 test in 4.396s
2022-05-18T04:20:43.2946600Z 
2022-05-18T04:20:43.2946705Z OK
2022-05-18T04:20:43.2946842Z 
2022-05-18T04:20:43.2946979Z Generating XML reports...
2022-05-18T04:20:43.2991007Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042038.xml
2022-05-18T04:20:44.5963668Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:20:44.5974249Z 
2022-05-18T04:20:44.5974573Z Running tests...
2022-05-18T04:20:44.5975042Z ----------------------------------------------------------------------
2022-05-18T04:20:46.2917946Z   test_sequence_num_incremented_nccl_default (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:46.3365447Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33204
2022-05-18T04:20:46.3476403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33205
2022-05-18T04:20:47.3027132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:47.3038207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:47.3052161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:47.3063090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:47.3064142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:47.3156590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:47.3366349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:47.3366903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:47.3368048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:20:47.3469920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:20:49.0557551Z ok (4.458s)
2022-05-18T04:20:49.0557792Z 
2022-05-18T04:20:49.0558232Z ----------------------------------------------------------------------
2022-05-18T04:20:49.0558578Z Ran 1 test in 4.458s
2022-05-18T04:20:49.0558743Z 
2022-05-18T04:20:49.0558836Z OK
2022-05-18T04:20:49.0558968Z 
2022-05-18T04:20:49.0559086Z Generating XML reports...
2022-05-18T04:20:49.0605157Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042044.xml
2022-05-18T04:20:50.3614599Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:20:50.3624559Z 
2022-05-18T04:20:50.3624721Z Running tests...
2022-05-18T04:20:50.3626315Z ----------------------------------------------------------------------
2022-05-18T04:20:52.0123198Z   test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:52.0556158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33422
2022-05-18T04:20:52.0675855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33423
2022-05-18T04:20:52.9819991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:52.9967805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:53.1722040Z ok (2.809s)
2022-05-18T04:20:53.1722296Z 
2022-05-18T04:20:53.1722751Z ----------------------------------------------------------------------
2022-05-18T04:20:53.1723108Z Ran 1 test in 2.810s
2022-05-18T04:20:53.1723278Z 
2022-05-18T04:20:53.1723386Z OK
2022-05-18T04:20:53.1725882Z 
2022-05-18T04:20:53.1726241Z Generating XML reports...
2022-05-18T04:20:53.1768552Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042050.xml
2022-05-18T04:20:54.4248846Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:20:54.4263012Z 
2022-05-18T04:20:54.4263935Z Running tests...
2022-05-18T04:20:54.4264640Z ----------------------------------------------------------------------
2022-05-18T04:20:56.0694223Z   test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:20:56.1125673Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33621
2022-05-18T04:20:56.1245333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33622
2022-05-18T04:20:57.0534641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:57.0556024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:57.0880919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:57.0903553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:57.0904451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:57.0964987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:20:58.7324533Z ok (4.306s)
2022-05-18T04:20:58.7324882Z 
2022-05-18T04:20:58.7326249Z ----------------------------------------------------------------------
2022-05-18T04:20:58.7326653Z Ran 1 test in 4.306s
2022-05-18T04:20:58.7326825Z 
2022-05-18T04:20:58.7326902Z OK
2022-05-18T04:20:58.7327043Z 
2022-05-18T04:20:58.7327187Z Generating XML reports...
2022-05-18T04:20:58.7370877Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042054.xml
2022-05-18T04:21:00.0062435Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:21:00.0072315Z 
2022-05-18T04:21:00.0072891Z Running tests...
2022-05-18T04:21:00.0073756Z ----------------------------------------------------------------------
2022-05-18T04:21:01.6645168Z   test_sequence_num_set_nccl_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:01.7080191Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33833
2022-05-18T04:21:01.7200582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33834
2022-05-18T04:21:02.6619968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:02.6634557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:02.6639846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:02.6654048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:02.6654958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:02.6655544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:21:02.6743104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:21:02.6743708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:21:02.6744387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:21:02.6761010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:21:04.3279990Z ok (4.320s)
2022-05-18T04:21:04.3280256Z 
2022-05-18T04:21:04.3280715Z ----------------------------------------------------------------------
2022-05-18T04:21:04.3281064Z Ran 1 test in 4.321s
2022-05-18T04:21:04.3281242Z 
2022-05-18T04:21:04.3281340Z OK
2022-05-18T04:21:04.3281460Z 
2022-05-18T04:21:04.3281618Z Generating XML reports...
2022-05-18T04:21:04.3325821Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042059.xml
2022-05-18T04:21:05.6204019Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:21:05.6214528Z 
2022-05-18T04:21:05.6214836Z Running tests...
2022-05-18T04:21:05.6215320Z ----------------------------------------------------------------------
2022-05-18T04:21:07.2794534Z   test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:07.3212814Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34049
2022-05-18T04:21:07.3333797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34050
2022-05-18T04:21:08.2910896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:08.2979829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:09.6089679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphib0518b
2022-05-18T04:21:09.6090316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphib0518b/_remote_module_non_scriptable.py
2022-05-18T04:21:09.6113950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5pl0rpjf
2022-05-18T04:21:09.6114649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5pl0rpjf/_remote_module_non_scriptable.py
2022-05-18T04:21:09.9079831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:09.9080409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:10.2418573Z ok (4.620s)
2022-05-18T04:21:10.2418868Z 
2022-05-18T04:21:10.2419595Z ----------------------------------------------------------------------
2022-05-18T04:21:10.2419952Z Ran 1 test in 4.620s
2022-05-18T04:21:10.2420464Z 
2022-05-18T04:21:10.2420647Z OK
2022-05-18T04:21:10.2420880Z 
2022-05-18T04:21:10.2421036Z Generating XML reports...
2022-05-18T04:21:10.2462736Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042105.xml
2022-05-18T04:21:11.5229555Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:21:11.5240786Z 
2022-05-18T04:21:11.5241222Z Running tests...
2022-05-18T04:21:11.5241705Z ----------------------------------------------------------------------
2022-05-18T04:21:13.1866099Z   test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:13.2313574Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34269
2022-05-18T04:21:13.2423137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34270
2022-05-18T04:21:14.2072681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:14.2285560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:15.5538676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4mbmupa
2022-05-18T04:21:15.5539334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4mbmupa/_remote_module_non_scriptable.py
2022-05-18T04:21:15.5550078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8p2acj6p
2022-05-18T04:21:15.5550686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8p2acj6p/_remote_module_non_scriptable.py
2022-05-18T04:21:15.8537402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:15.8538011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:16.1511392Z ok (4.627s)
2022-05-18T04:21:16.1511659Z 
2022-05-18T04:21:16.1512618Z ----------------------------------------------------------------------
2022-05-18T04:21:16.1512961Z Ran 1 test in 4.627s
2022-05-18T04:21:16.1513140Z 
2022-05-18T04:21:16.1513254Z OK
2022-05-18T04:21:16.1513397Z 
2022-05-18T04:21:16.1513552Z Generating XML reports...
2022-05-18T04:21:16.1558157Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042111.xml
2022-05-18T04:21:17.4525823Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:21:17.4536476Z 
2022-05-18T04:21:17.4536764Z Running tests...
2022-05-18T04:21:17.4537262Z ----------------------------------------------------------------------
2022-05-18T04:21:19.1113593Z   test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:19.1547755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34489
2022-05-18T04:21:19.1666874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34490
2022-05-18T04:21:20.1362094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:20.1660281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:21.4425478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_fe8cfsl
2022-05-18T04:21:21.4426372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_fe8cfsl/_remote_module_non_scriptable.py
2022-05-18T04:21:21.4815133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyuvzlm6
2022-05-18T04:21:21.4815852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyuvzlm6/_remote_module_non_scriptable.py
2022-05-18T04:21:21.7547987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:21.7548918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:22.0752772Z ok (4.621s)
2022-05-18T04:21:22.0753010Z 
2022-05-18T04:21:22.0753475Z ----------------------------------------------------------------------
2022-05-18T04:21:22.0753857Z Ran 1 test in 4.622s
2022-05-18T04:21:22.0754037Z 
2022-05-18T04:21:22.0754136Z OK
2022-05-18T04:21:22.0754270Z 
2022-05-18T04:21:22.0754431Z Generating XML reports...
2022-05-18T04:21:22.0798325Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042117.xml
2022-05-18T04:21:23.3404692Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:21:23.3415235Z 
2022-05-18T04:21:23.3415507Z Running tests...
2022-05-18T04:21:23.3415993Z ----------------------------------------------------------------------
2022-05-18T04:21:24.9706188Z   test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:25.0136223Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34709
2022-05-18T04:21:25.0260165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34710
2022-05-18T04:21:25.9602940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:26.0060097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:27.2760083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpppqlcomq
2022-05-18T04:21:27.2947954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpppqlcomq/_remote_module_non_scriptable.py
2022-05-18T04:21:27.2948574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5r4ustuf
2022-05-18T04:21:27.2949146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5r4ustuf/_remote_module_non_scriptable.py
2022-05-18T04:21:27.5705349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:27.5705938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:27.9346842Z ok (4.593s)
2022-05-18T04:21:27.9347043Z 
2022-05-18T04:21:27.9347491Z ----------------------------------------------------------------------
2022-05-18T04:21:27.9347984Z Ran 1 test in 4.593s
2022-05-18T04:21:27.9348135Z 
2022-05-18T04:21:27.9348415Z OK
2022-05-18T04:21:27.9348596Z 
2022-05-18T04:21:27.9348742Z Generating XML reports...
2022-05-18T04:21:27.9393933Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042123.xml
2022-05-18T04:21:29.1877098Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:21:29.1888654Z 
2022-05-18T04:21:29.1889122Z Running tests...
2022-05-18T04:21:29.1890026Z ----------------------------------------------------------------------
2022-05-18T04:21:29.1896918Z   test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s)
2022-05-18T04:21:29.1897597Z 
2022-05-18T04:21:29.1898166Z ----------------------------------------------------------------------
2022-05-18T04:21:29.1898810Z Ran 1 test in 0.001s
2022-05-18T04:21:29.1899124Z 
2022-05-18T04:21:29.1899867Z OK (skipped=1)
2022-05-18T04:21:29.1900143Z 
2022-05-18T04:21:29.1900374Z Generating XML reports...
2022-05-18T04:21:29.1936650Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042129.xml
2022-05-18T04:21:30.2472603Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:21:30.2485136Z 
2022-05-18T04:21:30.2485731Z Running tests...
2022-05-18T04:21:30.2486233Z ----------------------------------------------------------------------
2022-05-18T04:21:30.2491570Z   test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s)
2022-05-18T04:21:30.2492051Z 
2022-05-18T04:21:30.2492376Z ----------------------------------------------------------------------
2022-05-18T04:21:30.2492716Z Ran 1 test in 0.001s
2022-05-18T04:21:30.2492882Z 
2022-05-18T04:21:30.2492994Z OK (skipped=1)
2022-05-18T04:21:30.2493131Z 
2022-05-18T04:21:30.2493271Z Generating XML reports...
2022-05-18T04:21:30.2530726Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042130.xml
2022-05-18T04:21:31.3423271Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:21:31.3434122Z 
2022-05-18T04:21:31.3434424Z Running tests...
2022-05-18T04:21:31.3434906Z ----------------------------------------------------------------------
2022-05-18T04:21:32.9950144Z   test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:33.0391566Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35059
2022-05-18T04:21:33.0511435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35060
2022-05-18T04:21:34.0297195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:34.0341478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:35.3687133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp187baw2r
2022-05-18T04:21:35.3687762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp187baw2r/_remote_module_non_scriptable.py
2022-05-18T04:21:35.4029514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphn_do_37
2022-05-18T04:21:35.4030125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphn_do_37/_remote_module_non_scriptable.py
2022-05-18T04:21:35.8595778Z ok (4.516s)
2022-05-18T04:21:35.8595979Z 
2022-05-18T04:21:35.8596443Z ----------------------------------------------------------------------
2022-05-18T04:21:35.8596769Z Ran 1 test in 4.516s
2022-05-18T04:21:35.8596936Z 
2022-05-18T04:21:35.8597031Z OK
2022-05-18T04:21:35.8597169Z 
2022-05-18T04:21:35.8597302Z Generating XML reports...
2022-05-18T04:21:35.8641079Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042131.xml
2022-05-18T04:21:37.1182725Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:21:37.1192978Z 
2022-05-18T04:21:37.1193477Z Running tests...
2022-05-18T04:21:37.1194275Z ----------------------------------------------------------------------
2022-05-18T04:21:38.7290555Z   test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:38.7713790Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35279
2022-05-18T04:21:38.7833040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35280
2022-05-18T04:21:39.7673856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:39.7688315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:41.0718676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgazcmkne
2022-05-18T04:21:41.0719697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgazcmkne/_remote_module_non_scriptable.py
2022-05-18T04:21:41.0918181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcb6kqq23
2022-05-18T04:21:41.0918793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcb6kqq23/_remote_module_non_scriptable.py
2022-05-18T04:21:41.4913295Z ok (4.372s)
2022-05-18T04:21:41.4913532Z 
2022-05-18T04:21:41.4913963Z ----------------------------------------------------------------------
2022-05-18T04:21:41.4914316Z Ran 1 test in 4.372s
2022-05-18T04:21:41.4914878Z 
2022-05-18T04:21:41.4914969Z OK
2022-05-18T04:21:41.4915110Z 
2022-05-18T04:21:41.4915247Z Generating XML reports...
2022-05-18T04:21:41.4958450Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042137.xml
2022-05-18T04:21:42.7496882Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:21:42.7505648Z 
2022-05-18T04:21:42.7505994Z Running tests...
2022-05-18T04:21:42.7506494Z ----------------------------------------------------------------------
2022-05-18T04:21:42.7514078Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-05-18T04:21:44.3577058Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:44.3998793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35499
2022-05-18T04:21:44.4111834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35500
2022-05-18T04:21:45.3874881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:45.3903995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:46.7379864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5845lyz8
2022-05-18T04:21:46.7380566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5845lyz8/_remote_module_non_scriptable.py
2022-05-18T04:21:46.7402054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppd5w0uyv
2022-05-18T04:21:46.7402656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppd5w0uyv/_remote_module_non_scriptable.py
2022-05-18T04:21:47.4199752Z ok (4.669s)
2022-05-18T04:21:47.4200031Z 
2022-05-18T04:21:47.4200462Z ----------------------------------------------------------------------
2022-05-18T04:21:47.4200803Z Ran 1 test in 4.669s
2022-05-18T04:21:47.4200980Z 
2022-05-18T04:21:47.4201098Z OK
2022-05-18T04:21:47.4201220Z 
2022-05-18T04:21:47.4201361Z Generating XML reports...
2022-05-18T04:21:47.4247820Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042142.xml
2022-05-18T04:21:48.6900065Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:21:48.6911812Z 
2022-05-18T04:21:48.6912090Z Running tests...
2022-05-18T04:21:48.6912573Z ----------------------------------------------------------------------
2022-05-18T04:21:48.6921857Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:21:50.3466250Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:50.3915976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35719
2022-05-18T04:21:50.4044255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35720
2022-05-18T04:21:51.3794847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:51.3867371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:52.7143613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz1wgbgyf
2022-05-18T04:21:52.7144483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz1wgbgyf/_remote_module_non_scriptable.py
2022-05-18T04:21:52.7154670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp60iyfqc
2022-05-18T04:21:52.7155271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp60iyfqc/_remote_module_non_scriptable.py
2022-05-18T04:21:53.3134489Z ok (4.622s)
2022-05-18T04:21:53.3134888Z 
2022-05-18T04:21:53.3135334Z ----------------------------------------------------------------------
2022-05-18T04:21:53.3135684Z Ran 1 test in 4.622s
2022-05-18T04:21:53.3135863Z 
2022-05-18T04:21:53.3136320Z OK
2022-05-18T04:21:53.3136454Z 
2022-05-18T04:21:53.3136601Z Generating XML reports...
2022-05-18T04:21:53.3179963Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042148.xml
2022-05-18T04:21:54.5876991Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:21:54.5889727Z 
2022-05-18T04:21:54.5889919Z Running tests...
2022-05-18T04:21:54.5890405Z ----------------------------------------------------------------------
2022-05-18T04:21:54.5902025Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:21:56.2233219Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:21:56.2672576Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35939
2022-05-18T04:21:56.2783757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35940
2022-05-18T04:21:57.2367636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:57.2994805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:58.5542474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxddcf046
2022-05-18T04:21:58.5543412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxddcf046/_remote_module_non_scriptable.py
2022-05-18T04:21:58.6203514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptupywrdy
2022-05-18T04:21:58.6204126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptupywrdy/_remote_module_non_scriptable.py
2022-05-18T04:21:58.9041016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:58.9041610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:58.9361639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:58.9362185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:58.9525319Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:21:58.9526284Z   warnings.warn(
2022-05-18T04:21:58.9527383Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:21:58.9528127Z   warnings.warn(
2022-05-18T04:21:58.9643244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:58.9643795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:58.9869556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:58.9870101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:59.0183014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:59.0183575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:59.0460465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:59.0461015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:21:59.3875631Z ok (4.798s)
2022-05-18T04:21:59.3875946Z 
2022-05-18T04:21:59.3876387Z ----------------------------------------------------------------------
2022-05-18T04:21:59.3876934Z Ran 1 test in 4.799s
2022-05-18T04:21:59.3877134Z 
2022-05-18T04:21:59.3877244Z OK
2022-05-18T04:21:59.3877379Z 
2022-05-18T04:21:59.3877511Z Generating XML reports...
2022-05-18T04:21:59.3921775Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042154.xml
2022-05-18T04:22:00.6768254Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:22:00.6778569Z 
2022-05-18T04:22:00.6778757Z Running tests...
2022-05-18T04:22:00.6779233Z ----------------------------------------------------------------------
2022-05-18T04:22:00.6790266Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:22:02.3521567Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:02.3959902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36159
2022-05-18T04:22:02.4069725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36160
2022-05-18T04:22:03.4065577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:03.4066185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:04.7500694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlf3q_9z
2022-05-18T04:22:04.7501361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlf3q_9z/_remote_module_non_scriptable.py
2022-05-18T04:22:04.7617581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1c8ox4_f
2022-05-18T04:22:04.7618199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1c8ox4_f/_remote_module_non_scriptable.py
2022-05-18T04:22:05.0534312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:05.0534869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:05.0865652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:05.0866209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:05.1032703Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:22:05.1033503Z   warnings.warn(
2022-05-18T04:22:05.1034563Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:22:05.1035309Z   warnings.warn(
2022-05-18T04:22:05.1148786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:05.1149330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:05.1378042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:05.1378587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:05.1708186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:05.1709179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:05.1985219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:05.1986016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:05.5157786Z ok (4.838s)
2022-05-18T04:22:05.5158027Z 
2022-05-18T04:22:05.5158487Z ----------------------------------------------------------------------
2022-05-18T04:22:05.5158853Z Ran 1 test in 4.838s
2022-05-18T04:22:05.5159021Z 
2022-05-18T04:22:05.5159120Z OK
2022-05-18T04:22:05.5159262Z 
2022-05-18T04:22:05.5159378Z Generating XML reports...
2022-05-18T04:22:05.5201907Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042200.xml
2022-05-18T04:22:06.7864615Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:22:06.7876144Z 
2022-05-18T04:22:06.7876435Z Running tests...
2022-05-18T04:22:06.7876938Z ----------------------------------------------------------------------
2022-05-18T04:22:06.7885306Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:22:08.4493307Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:08.4936229Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36379
2022-05-18T04:22:08.5059652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36380
2022-05-18T04:22:09.4592902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:09.4803275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:10.7790608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw597oenq
2022-05-18T04:22:10.7791240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw597oenq/_remote_module_non_scriptable.py
2022-05-18T04:22:10.7843745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ofi8ln4
2022-05-18T04:22:10.7844324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ofi8ln4/_remote_module_non_scriptable.py
2022-05-18T04:22:11.0807506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:11.0808101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:11.1114570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:11.1115138Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:11.4146540Z ok (4.627s)
2022-05-18T04:22:11.4146758Z 
2022-05-18T04:22:11.4147196Z ----------------------------------------------------------------------
2022-05-18T04:22:11.4147554Z Ran 1 test in 4.627s
2022-05-18T04:22:11.4147731Z 
2022-05-18T04:22:11.4147845Z OK
2022-05-18T04:22:11.4147962Z 
2022-05-18T04:22:11.4148117Z Generating XML reports...
2022-05-18T04:22:11.4192869Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042206.xml
2022-05-18T04:22:12.7051079Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:22:12.7061804Z 
2022-05-18T04:22:12.7062165Z Running tests...
2022-05-18T04:22:12.7063133Z ----------------------------------------------------------------------
2022-05-18T04:22:12.7071576Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:22:14.3604395Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:14.4048102Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36599
2022-05-18T04:22:14.4171169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36600
2022-05-18T04:22:15.3586258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:15.3754683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:16.6831130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2s1m27m
2022-05-18T04:22:16.6831787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2s1m27m/_remote_module_non_scriptable.py
2022-05-18T04:22:16.6905616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwbuxlw6
2022-05-18T04:22:16.6906225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwbuxlw6/_remote_module_non_scriptable.py
2022-05-18T04:22:16.9877197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:16.9877775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:17.0219251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:17.0219779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:17.3259929Z ok (4.619s)
2022-05-18T04:22:17.3260140Z 
2022-05-18T04:22:17.3260592Z ----------------------------------------------------------------------
2022-05-18T04:22:17.3260968Z Ran 1 test in 4.620s
2022-05-18T04:22:17.3261136Z 
2022-05-18T04:22:17.3261232Z OK
2022-05-18T04:22:17.3261371Z 
2022-05-18T04:22:17.3305996Z Generating XML reports...
2022-05-18T04:22:17.3306730Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042212.xml
2022-05-18T04:22:18.6160420Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:22:18.6170325Z 
2022-05-18T04:22:18.6170766Z Running tests...
2022-05-18T04:22:18.6171738Z ----------------------------------------------------------------------
2022-05-18T04:22:18.6183335Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:22:20.2717240Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:20.3163826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36819
2022-05-18T04:22:20.3286275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36820
2022-05-18T04:22:21.3328287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:21.3537842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:22.6393365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dqpdhqp
2022-05-18T04:22:22.6394079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dqpdhqp/_remote_module_non_scriptable.py
2022-05-18T04:22:22.7023248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pvkieo8
2022-05-18T04:22:22.7024082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pvkieo8/_remote_module_non_scriptable.py
2022-05-18T04:22:22.9935099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:22.9935718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:23.0198468Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:22:23.0200362Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:22:23.0571460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:23.0572005Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:23.4378883Z ok (4.820s)
2022-05-18T04:22:23.4379335Z 
2022-05-18T04:22:23.4380048Z ----------------------------------------------------------------------
2022-05-18T04:22:23.4380804Z Ran 1 test in 4.821s
2022-05-18T04:22:23.4381098Z 
2022-05-18T04:22:23.4381201Z OK
2022-05-18T04:22:23.4381346Z 
2022-05-18T04:22:23.4381513Z Generating XML reports...
2022-05-18T04:22:23.4425930Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042218.xml
2022-05-18T04:22:24.7209963Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:22:24.7219991Z 
2022-05-18T04:22:24.7220235Z Running tests...
2022-05-18T04:22:24.7221266Z ----------------------------------------------------------------------
2022-05-18T04:22:24.7233662Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:22:26.3916000Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:26.4346006Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37039
2022-05-18T04:22:26.4457741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37040
2022-05-18T04:22:27.4498032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:27.4507084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:28.7856387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp099a1s_n
2022-05-18T04:22:28.7857040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp099a1s_n/_remote_module_non_scriptable.py
2022-05-18T04:22:28.7972279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0g3hcjk
2022-05-18T04:22:28.7972964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0g3hcjk/_remote_module_non_scriptable.py
2022-05-18T04:22:29.4548213Z ok (4.732s)
2022-05-18T04:22:29.4548419Z 
2022-05-18T04:22:29.4548825Z ----------------------------------------------------------------------
2022-05-18T04:22:29.4549198Z Ran 1 test in 4.733s
2022-05-18T04:22:29.4549370Z 
2022-05-18T04:22:29.4549473Z OK
2022-05-18T04:22:29.4549613Z 
2022-05-18T04:22:29.4549763Z Generating XML reports...
2022-05-18T04:22:29.4592874Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042224.xml
2022-05-18T04:22:30.7341513Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:22:30.7352692Z 
2022-05-18T04:22:30.7353008Z Running tests...
2022-05-18T04:22:30.7353479Z ----------------------------------------------------------------------
2022-05-18T04:22:30.7360847Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:22:32.4161490Z Checkpointing should work with static graph in the case of checkpointing ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:32.4600514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37259
2022-05-18T04:22:32.4720578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37260
2022-05-18T04:22:33.4204393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:33.4545742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:34.7102748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufzsakfg
2022-05-18T04:22:34.7103459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufzsakfg/_remote_module_non_scriptable.py
2022-05-18T04:22:34.7435234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrexxzs_
2022-05-18T04:22:34.7436167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrexxzs_/_remote_module_non_scriptable.py
2022-05-18T04:22:35.0378105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:35.0379914Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:35.0684663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:35.0685211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:35.3805439Z ok (4.645s)
2022-05-18T04:22:35.3805658Z 
2022-05-18T04:22:35.3806082Z ----------------------------------------------------------------------
2022-05-18T04:22:35.3806409Z Ran 1 test in 4.645s
2022-05-18T04:22:35.3806603Z 
2022-05-18T04:22:35.3806703Z OK
2022-05-18T04:22:35.3806843Z 
2022-05-18T04:22:35.3807583Z Generating XML reports...
2022-05-18T04:22:35.3850242Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042230.xml
2022-05-18T04:22:36.6461216Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:22:36.6471178Z 
2022-05-18T04:22:36.6471673Z Running tests...
2022-05-18T04:22:36.6472293Z ----------------------------------------------------------------------
2022-05-18T04:22:36.6482284Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:22:38.2603650Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:38.3021236Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37479
2022-05-18T04:22:38.3136031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37480
2022-05-18T04:22:39.2484180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:39.2514873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:40.5764537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoqfpd90i
2022-05-18T04:22:40.5765183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoqfpd90i/_remote_module_non_scriptable.py
2022-05-18T04:22:40.5823861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp9lts1te
2022-05-18T04:22:40.5824472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp9lts1te/_remote_module_non_scriptable.py
2022-05-18T04:22:40.8467324Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:22:40.8488370Z [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:22:40.8846771Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:22:40.8847579Z   warnings.warn(
2022-05-18T04:22:40.8848673Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:22:40.8849422Z   warnings.warn(
2022-05-18T04:22:40.8958718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:40.8960027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:40.9497736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:40.9498257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:41.3220899Z ok (4.675s)
2022-05-18T04:22:41.3221100Z 
2022-05-18T04:22:41.3222181Z ----------------------------------------------------------------------
2022-05-18T04:22:41.3222583Z Ran 1 test in 4.675s
2022-05-18T04:22:41.3222754Z 
2022-05-18T04:22:41.3222858Z OK
2022-05-18T04:22:41.3223006Z 
2022-05-18T04:22:41.3223140Z Generating XML reports...
2022-05-18T04:22:41.3268058Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042236.xml
2022-05-18T04:22:42.5960534Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:22:42.5970567Z 
2022-05-18T04:22:42.5970749Z Running tests...
2022-05-18T04:22:42.5971612Z ----------------------------------------------------------------------
2022-05-18T04:22:42.5983040Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:22:44.2690424Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:44.3118418Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37699
2022-05-18T04:22:44.3231943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37700
2022-05-18T04:22:45.2856473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:45.3144801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:46.5985801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hyghz1g
2022-05-18T04:22:46.5986428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hyghz1g/_remote_module_non_scriptable.py
2022-05-18T04:22:46.6160925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqeumf1iq
2022-05-18T04:22:46.6161519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqeumf1iq/_remote_module_non_scriptable.py
2022-05-18T04:22:46.9048132Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:22:46.9049338Z   warnings.warn(
2022-05-18T04:22:46.9050440Z /opt/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:1736: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-05-18T04:22:46.9051200Z   warnings.warn(
2022-05-18T04:22:46.9352525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:46.9356859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:46.9770942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:46.9772738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:47.3319220Z ok (4.734s)
2022-05-18T04:22:47.3319484Z 
2022-05-18T04:22:47.3319951Z ----------------------------------------------------------------------
2022-05-18T04:22:47.3320300Z Ran 1 test in 4.735s
2022-05-18T04:22:47.3320458Z 
2022-05-18T04:22:47.3320556Z OK
2022-05-18T04:22:47.3320692Z 
2022-05-18T04:22:47.3320829Z Generating XML reports...
2022-05-18T04:22:47.3365322Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042242.xml
2022-05-18T04:22:48.6072188Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:22:48.6082054Z 
2022-05-18T04:22:48.6082374Z Running tests...
2022-05-18T04:22:48.6082860Z ----------------------------------------------------------------------
2022-05-18T04:22:48.6096064Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:22:50.2880942Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:50.3329667Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37919
2022-05-18T04:22:50.3453194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37920
2022-05-18T04:22:51.2801609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:51.3219774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:52.5887858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmf3hkqp
2022-05-18T04:22:52.5888559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmf3hkqp/_remote_module_non_scriptable.py
2022-05-18T04:22:52.6276171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6ca43xi
2022-05-18T04:22:52.6276783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6ca43xi/_remote_module_non_scriptable.py
2022-05-18T04:22:52.9044513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:52.9069540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:52.9430780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:52.9431322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:53.2543440Z ok (4.646s)
2022-05-18T04:22:53.2543728Z 
2022-05-18T04:22:53.2544146Z ----------------------------------------------------------------------
2022-05-18T04:22:53.2544498Z Ran 1 test in 4.646s
2022-05-18T04:22:53.2544670Z 
2022-05-18T04:22:53.2544771Z OK
2022-05-18T04:22:53.2544914Z 
2022-05-18T04:22:53.2545055Z Generating XML reports...
2022-05-18T04:22:53.2590244Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042248.xml
2022-05-18T04:22:54.5677154Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:22:54.5694597Z 
2022-05-18T04:22:54.5694777Z Running tests...
2022-05-18T04:22:54.5695296Z ----------------------------------------------------------------------
2022-05-18T04:22:54.5706701Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:22:56.2324911Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:22:56.2763676Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38139
2022-05-18T04:22:56.2887108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38140
2022-05-18T04:22:57.2419021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:57.2799129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:58.5910815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvuluy594
2022-05-18T04:22:58.5911459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvuluy594/_remote_module_non_scriptable.py
2022-05-18T04:22:58.5922044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4mci_ci
2022-05-18T04:22:58.5923415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4mci_ci/_remote_module_non_scriptable.py
2022-05-18T04:22:58.8780745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:58.8811280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:58.9125408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:58.9129089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:58.9342146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:58.9343063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:58.9649135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:58.9651820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:22:59.2976302Z ok (4.728s)
2022-05-18T04:22:59.2976610Z 
2022-05-18T04:22:59.2977089Z ----------------------------------------------------------------------
2022-05-18T04:22:59.2977452Z Ran 1 test in 4.729s
2022-05-18T04:22:59.2977626Z 
2022-05-18T04:22:59.2977727Z OK
2022-05-18T04:22:59.2977848Z 
2022-05-18T04:22:59.2977999Z Generating XML reports...
2022-05-18T04:22:59.3023216Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042254.xml
2022-05-18T04:23:00.5558043Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:23:00.5569169Z 
2022-05-18T04:23:00.5570102Z Running tests...
2022-05-18T04:23:00.5570592Z ----------------------------------------------------------------------
2022-05-18T04:23:02.2396531Z   test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:02.2828265Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38359
2022-05-18T04:23:02.2950925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38360
2022-05-18T04:23:03.2785954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:03.2832977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:04.6152133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ahwfw44
2022-05-18T04:23:04.6152775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ahwfw44/_remote_module_non_scriptable.py
2022-05-18T04:23:04.6349818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6nq4yrj
2022-05-18T04:23:04.6350433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6nq4yrj/_remote_module_non_scriptable.py
2022-05-18T04:23:05.0033172Z ok (4.446s)
2022-05-18T04:23:05.0033544Z 
2022-05-18T04:23:05.0034371Z ----------------------------------------------------------------------
2022-05-18T04:23:05.0034823Z Ran 1 test in 4.446s
2022-05-18T04:23:05.0034997Z 
2022-05-18T04:23:05.0035096Z OK
2022-05-18T04:23:05.0035233Z 
2022-05-18T04:23:05.0035350Z Generating XML reports...
2022-05-18T04:23:05.0078351Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042300.xml
2022-05-18T04:23:06.2820430Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:23:06.2831294Z 
2022-05-18T04:23:06.2831752Z Running tests...
2022-05-18T04:23:06.2832274Z ----------------------------------------------------------------------
2022-05-18T04:23:07.9529490Z   test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:07.9966627Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38579
2022-05-18T04:23:08.0087707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38580
2022-05-18T04:23:08.9771016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:08.9805275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:10.2825078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gk2qouz
2022-05-18T04:23:10.2825713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gk2qouz/_remote_module_non_scriptable.py
2022-05-18T04:23:10.2872116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatnpwvl6
2022-05-18T04:23:10.2872796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatnpwvl6/_remote_module_non_scriptable.py
2022-05-18T04:23:10.7170826Z ok (4.434s)
2022-05-18T04:23:10.7171066Z 
2022-05-18T04:23:10.7171503Z ----------------------------------------------------------------------
2022-05-18T04:23:10.7171877Z Ran 1 test in 4.434s
2022-05-18T04:23:10.7172050Z 
2022-05-18T04:23:10.7172143Z OK
2022-05-18T04:23:10.7172284Z 
2022-05-18T04:23:10.7172423Z Generating XML reports...
2022-05-18T04:23:10.7217047Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042306.xml
2022-05-18T04:23:12.0014679Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:23:12.0025479Z 
2022-05-18T04:23:12.0025784Z Running tests...
2022-05-18T04:23:12.0026278Z ----------------------------------------------------------------------
2022-05-18T04:23:13.6459388Z   test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:13.6894022Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38799
2022-05-18T04:23:13.7003851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38800
2022-05-18T04:23:14.6778807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:14.7240544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:15.9508611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1s8m7mb
2022-05-18T04:23:15.9509243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1s8m7mb/_remote_module_non_scriptable.py
2022-05-18T04:23:15.9949708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5ig9iof
2022-05-18T04:23:15.9950338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5ig9iof/_remote_module_non_scriptable.py
2022-05-18T04:23:16.4083776Z ok (4.405s)
2022-05-18T04:23:16.4084159Z 
2022-05-18T04:23:16.4084632Z ----------------------------------------------------------------------
2022-05-18T04:23:16.4085015Z Ran 1 test in 4.406s
2022-05-18T04:23:16.4085189Z 
2022-05-18T04:23:16.4085293Z OK
2022-05-18T04:23:16.4085436Z 
2022-05-18T04:23:16.4086798Z Generating XML reports...
2022-05-18T04:23:16.4131633Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042311.xml
2022-05-18T04:23:17.6793132Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:23:17.6803515Z 
2022-05-18T04:23:17.6803808Z Running tests...
2022-05-18T04:23:17.6804279Z ----------------------------------------------------------------------
2022-05-18T04:23:17.6818037Z   test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:23:19.3341830Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:19.3785303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39019
2022-05-18T04:23:19.3905743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39020
2022-05-18T04:23:20.3272901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:20.3507764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:21.6379637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33y1bdpx
2022-05-18T04:23:21.6380272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33y1bdpx/_remote_module_non_scriptable.py
2022-05-18T04:23:21.6420001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprxlq6f0r
2022-05-18T04:23:21.6420628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprxlq6f0r/_remote_module_non_scriptable.py
2022-05-18T04:23:21.9987846Z ok (4.318s)
2022-05-18T04:23:21.9988078Z 
2022-05-18T04:23:21.9988515Z ----------------------------------------------------------------------
2022-05-18T04:23:21.9988868Z Ran 1 test in 4.318s
2022-05-18T04:23:21.9989050Z 
2022-05-18T04:23:21.9989153Z OK
2022-05-18T04:23:21.9989300Z 
2022-05-18T04:23:21.9989414Z Generating XML reports...
2022-05-18T04:23:22.0032724Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042317.xml
2022-05-18T04:23:23.2903178Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:23:23.2914534Z 
2022-05-18T04:23:23.2914687Z Running tests...
2022-05-18T04:23:23.2915177Z ----------------------------------------------------------------------
2022-05-18T04:23:23.2924100Z   test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:23:24.9346195Z This unit test verifies whether the Future object is passed properly using nccl backend. ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:24.9794409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39239
2022-05-18T04:23:24.9916836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39240
2022-05-18T04:23:25.9330622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:25.9656291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:27.2326033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rimlqux
2022-05-18T04:23:27.2326685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rimlqux/_remote_module_non_scriptable.py
2022-05-18T04:23:27.2775684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplxf_3pcj
2022-05-18T04:23:27.2776288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplxf_3pcj/_remote_module_non_scriptable.py
2022-05-18T04:23:27.7005437Z ok (4.409s)
2022-05-18T04:23:27.7005677Z 
2022-05-18T04:23:27.7006135Z ----------------------------------------------------------------------
2022-05-18T04:23:27.7006487Z Ran 1 test in 4.409s
2022-05-18T04:23:27.7006654Z 
2022-05-18T04:23:27.7006752Z OK
2022-05-18T04:23:27.7006889Z 
2022-05-18T04:23:27.7007030Z Generating XML reports...
2022-05-18T04:23:27.7053235Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042323.xml
2022-05-18T04:23:28.9690324Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:23:28.9701543Z 
2022-05-18T04:23:28.9701823Z Running tests...
2022-05-18T04:23:28.9702728Z ----------------------------------------------------------------------
2022-05-18T04:23:30.6213217Z   test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:30.6668253Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39459
2022-05-18T04:23:30.6790356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39460
2022-05-18T04:23:31.6754786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:31.6860117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:34.4900229Z ok (5.519s)
2022-05-18T04:23:34.4900483Z 
2022-05-18T04:23:34.4901054Z ----------------------------------------------------------------------
2022-05-18T04:23:34.4901412Z Ran 1 test in 5.520s
2022-05-18T04:23:34.4901582Z 
2022-05-18T04:23:34.4901681Z OK
2022-05-18T04:23:34.4901821Z 
2022-05-18T04:23:34.4902261Z Generating XML reports...
2022-05-18T04:23:34.4946176Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042328.xml
2022-05-18T04:23:35.7594602Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:23:35.7606105Z 
2022-05-18T04:23:35.7606297Z Running tests...
2022-05-18T04:23:35.7606767Z ----------------------------------------------------------------------
2022-05-18T04:23:37.4256620Z   test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:37.4705257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39666
2022-05-18T04:23:37.4825909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39667
2022-05-18T04:23:38.4493587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:38.4886502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:39.8184575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbfdyhq8y
2022-05-18T04:23:39.8185311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbfdyhq8y/_remote_module_non_scriptable.py
2022-05-18T04:23:39.8265898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rxtwreo
2022-05-18T04:23:39.8266558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rxtwreo/_remote_module_non_scriptable.py
2022-05-18T04:23:39.9361557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:39.9362987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:39.9941842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:39.9942649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:40.0510581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:40.0511118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:40.1068584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:40.1069122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:40.4916680Z ok (4.731s)
2022-05-18T04:23:40.4916922Z 
2022-05-18T04:23:40.4917370Z ----------------------------------------------------------------------
2022-05-18T04:23:40.4917743Z Ran 1 test in 4.731s
2022-05-18T04:23:40.4917923Z 
2022-05-18T04:23:40.4918026Z OK
2022-05-18T04:23:40.4918169Z 
2022-05-18T04:23:40.4918316Z Generating XML reports...
2022-05-18T04:23:40.4962169Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042335.xml
2022-05-18T04:23:41.7803223Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:23:41.7813993Z 
2022-05-18T04:23:41.7814313Z Running tests...
2022-05-18T04:23:41.7814909Z ----------------------------------------------------------------------
2022-05-18T04:23:43.4425096Z   test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:43.4871854Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39886
2022-05-18T04:23:43.4997605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39887
2022-05-18T04:23:44.4420267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:44.4420831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:44.4428920Z /opt/conda/lib/python3.9/site-packages/torch/nn/modules/lazy.py:178: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2022-05-18T04:23:44.4429641Z   warnings.warn('Lazy modules are a new feature under heavy development '
2022-05-18T04:23:44.4430425Z /opt/conda/lib/python3.9/site-packages/torch/nn/modules/lazy.py:178: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2022-05-18T04:23:44.4431085Z   warnings.warn('Lazy modules are a new feature under heavy development '
2022-05-18T04:23:44.4523259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpti3d2fue
2022-05-18T04:23:44.4523825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62aq6qjz
2022-05-18T04:23:44.4524394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpti3d2fue/_remote_module_non_scriptable.py
2022-05-18T04:23:44.4524964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62aq6qjz/_remote_module_non_scriptable.py
2022-05-18T04:23:44.6041822Z ok (2.822s)
2022-05-18T04:23:44.6042047Z 
2022-05-18T04:23:44.6042518Z ----------------------------------------------------------------------
2022-05-18T04:23:44.6042878Z Ran 1 test in 2.823s
2022-05-18T04:23:44.6043048Z 
2022-05-18T04:23:44.6043143Z OK
2022-05-18T04:23:44.6043280Z 
2022-05-18T04:23:44.6043418Z Generating XML reports...
2022-05-18T04:23:44.6087509Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042341.xml
2022-05-18T04:23:45.8606174Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:23:45.8616297Z 
2022-05-18T04:23:45.8617848Z Running tests...
2022-05-18T04:23:45.8618525Z ----------------------------------------------------------------------
2022-05-18T04:23:47.5127418Z   test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:47.5575477Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40089
2022-05-18T04:23:47.5697726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40090
2022-05-18T04:23:48.4943297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:48.5122185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:49.8004335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0j7yte8
2022-05-18T04:23:49.8004977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0j7yte8/_remote_module_non_scriptable.py
2022-05-18T04:23:49.8124767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps03_d16j
2022-05-18T04:23:49.8125363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps03_d16j/_remote_module_non_scriptable.py
2022-05-18T04:23:50.1779250Z ok (4.316s)
2022-05-18T04:23:50.1779517Z 
2022-05-18T04:23:50.1779965Z ----------------------------------------------------------------------
2022-05-18T04:23:50.1780349Z Ran 1 test in 4.316s
2022-05-18T04:23:50.1780520Z 
2022-05-18T04:23:50.1780623Z OK
2022-05-18T04:23:50.1780765Z 
2022-05-18T04:23:50.1780888Z Generating XML reports...
2022-05-18T04:23:50.1823764Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042345.xml
2022-05-18T04:23:51.4265943Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:23:51.4277161Z 
2022-05-18T04:23:51.4277416Z Running tests...
2022-05-18T04:23:51.4278207Z ----------------------------------------------------------------------
2022-05-18T04:23:53.0965072Z   test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:53.1421438Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40309
2022-05-18T04:23:53.1543899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40310
2022-05-18T04:23:54.0873613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:54.1053873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:55.4125395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnutsyp1l
2022-05-18T04:23:55.4126065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnutsyp1l/_remote_module_non_scriptable.py
2022-05-18T04:23:55.4271971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyyyqja3t
2022-05-18T04:23:55.4272881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyyyqja3t/_remote_module_non_scriptable.py
2022-05-18T04:23:55.8627160Z ok (4.435s)
2022-05-18T04:23:55.8627342Z 
2022-05-18T04:23:55.8627847Z ----------------------------------------------------------------------
2022-05-18T04:23:55.8628205Z Ran 1 test in 4.435s
2022-05-18T04:23:55.8628387Z 
2022-05-18T04:23:55.8628494Z OK
2022-05-18T04:23:55.8628635Z 
2022-05-18T04:23:55.8633543Z Generating XML reports...
2022-05-18T04:23:55.8672399Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042351.xml
2022-05-18T04:23:57.1404740Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:23:57.1414638Z 
2022-05-18T04:23:57.1414968Z Running tests...
2022-05-18T04:23:57.1415448Z ----------------------------------------------------------------------
2022-05-18T04:23:58.8105627Z   test_failure_recovery (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:23:58.8554664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40529
2022-05-18T04:23:58.8678005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40530
2022-05-18T04:23:59.8419803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:59.8876250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:01.1493506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmhu30o0
2022-05-18T04:24:01.1494441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmhu30o0/_remote_module_non_scriptable.py
2022-05-18T04:24:01.1847761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0iioz6f
2022-05-18T04:24:01.1848374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0iioz6f/_remote_module_non_scriptable.py
2022-05-18T04:24:01.4601783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:01.4603094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:01.5040643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:01.5041180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:24:01.8764723Z ok (4.735s)
2022-05-18T04:24:01.8764956Z 
2022-05-18T04:24:01.8765411Z ----------------------------------------------------------------------
2022-05-18T04:24:01.8765779Z Ran 1 test in 4.735s
2022-05-18T04:24:01.8765945Z 
2022-05-18T04:24:01.8766040Z OK
2022-05-18T04:24:01.8766182Z 
2022-05-18T04:24:01.8766322Z Generating XML reports...
2022-05-18T04:24:01.8809500Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042357.xml
2022-05-18T04:24:03.1529443Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:24:03.1540354Z 
2022-05-18T04:24:03.1540673Z Running tests...
2022-05-18T04:24:03.1541171Z ----------------------------------------------------------------------
2022-05-18T04:24:04.8151454Z   test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:04.8589950Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40760
2022-05-18T04:24:04.8713547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40761
2022-05-18T04:24:05.8418096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:05.8673827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:05.8852237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:05.8852781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:05.8853640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:05.8854354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:07.2081912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvq692e5
2022-05-18T04:24:07.2082575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvq692e5/_remote_module_non_scriptable.py
2022-05-18T04:24:07.2367927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38__mmlq
2022-05-18T04:24:07.2368533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38__mmlq/_remote_module_non_scriptable.py
2022-05-18T04:24:07.8803745Z ok (4.726s)
2022-05-18T04:24:07.8804001Z 
2022-05-18T04:24:07.8804472Z ----------------------------------------------------------------------
2022-05-18T04:24:07.8805235Z Ran 1 test in 4.726s
2022-05-18T04:24:07.8805408Z 
2022-05-18T04:24:07.8805515Z OK
2022-05-18T04:24:07.8805659Z 
2022-05-18T04:24:07.8805819Z Generating XML reports...
2022-05-18T04:24:07.8848337Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042403.xml
2022-05-18T04:24:09.1549244Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:24:09.1559681Z 
2022-05-18T04:24:09.1560252Z Running tests...
2022-05-18T04:24:09.1560800Z ----------------------------------------------------------------------
2022-05-18T04:24:10.8094950Z   test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:10.8534971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40986
2022-05-18T04:24:10.8659066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40987
2022-05-18T04:24:11.7913910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:11.7935805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:11.8016383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:11.8041255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:11.8042138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:11.8139405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:13.1428295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1wd9bbu
2022-05-18T04:24:13.1428949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1wd9bbu/_remote_module_non_scriptable.py
2022-05-18T04:24:13.1639166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4j33xx0d
2022-05-18T04:24:13.1639781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4j33xx0d/_remote_module_non_scriptable.py
2022-05-18T04:24:13.7745293Z ok (4.618s)
2022-05-18T04:24:13.7745492Z 
2022-05-18T04:24:13.7745941Z ----------------------------------------------------------------------
2022-05-18T04:24:13.7746326Z Ran 1 test in 4.618s
2022-05-18T04:24:13.7746494Z 
2022-05-18T04:24:13.7746592Z OK
2022-05-18T04:24:13.7746736Z 
2022-05-18T04:24:13.7746860Z Generating XML reports...
2022-05-18T04:24:13.7790822Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042409.xml
2022-05-18T04:24:15.0138913Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:24:15.0148768Z 
2022-05-18T04:24:15.0149214Z Running tests...
2022-05-18T04:24:15.0149690Z ----------------------------------------------------------------------
2022-05-18T04:24:16.5986411Z   test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:16.6400660Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41206
2022-05-18T04:24:16.6511927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41207
2022-05-18T04:24:17.5997810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:17.6017245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:17.6368067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:17.6391264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:17.6392136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:17.6429545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:18.9625744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzjxlpm0
2022-05-18T04:24:18.9629493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzjxlpm0/_remote_module_non_scriptable.py
2022-05-18T04:24:18.9920463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnlivdyd
2022-05-18T04:24:18.9921302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnlivdyd/_remote_module_non_scriptable.py
2022-05-18T04:24:19.5596012Z ok (4.544s)
2022-05-18T04:24:19.5596271Z 
2022-05-18T04:24:19.5596697Z ----------------------------------------------------------------------
2022-05-18T04:24:19.5597057Z Ran 1 test in 4.545s
2022-05-18T04:24:19.5597248Z 
2022-05-18T04:24:19.5597352Z OK
2022-05-18T04:24:19.5597497Z 
2022-05-18T04:24:19.5597647Z Generating XML reports...
2022-05-18T04:24:19.5641662Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042415.xml
2022-05-18T04:24:20.7909930Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:24:20.7919845Z 
2022-05-18T04:24:20.7920042Z Running tests...
2022-05-18T04:24:20.7920553Z ----------------------------------------------------------------------
2022-05-18T04:24:22.4548283Z   test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:22.4985225Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41426
2022-05-18T04:24:22.5105527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41427
2022-05-18T04:24:23.4273379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:23.4357337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:23.4590821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:23.4591365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:23.4592193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:23.4592911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:24.7848875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcyokj7f2
2022-05-18T04:24:24.7849528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcyokj7f2/_remote_module_non_scriptable.py
2022-05-18T04:24:24.7850904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekcptuce
2022-05-18T04:24:24.7851763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekcptuce/_remote_module_non_scriptable.py
2022-05-18T04:24:25.4191232Z ok (4.627s)
2022-05-18T04:24:25.4191473Z 
2022-05-18T04:24:25.4191939Z ----------------------------------------------------------------------
2022-05-18T04:24:25.4192303Z Ran 1 test in 4.627s
2022-05-18T04:24:25.4192452Z 
2022-05-18T04:24:25.4192553Z OK
2022-05-18T04:24:25.4192691Z 
2022-05-18T04:24:25.4192835Z Generating XML reports...
2022-05-18T04:24:25.4237642Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042420.xml
2022-05-18T04:24:26.6798212Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:24:26.6808637Z 
2022-05-18T04:24:26.6808967Z Running tests...
2022-05-18T04:24:26.6809442Z ----------------------------------------------------------------------
2022-05-18T04:24:28.3350936Z   test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:28.3783734Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41652
2022-05-18T04:24:28.3904771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41653
2022-05-18T04:24:29.3299811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:29.3344670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:29.3922501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:29.3947219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:29.3948066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:29.3954922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:30.7105496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwin75dde
2022-05-18T04:24:30.7106125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwin75dde/_remote_module_non_scriptable.py
2022-05-18T04:24:30.7207064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc38nsn15
2022-05-18T04:24:30.7207668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc38nsn15/_remote_module_non_scriptable.py
2022-05-18T04:24:31.2992165Z ok (4.618s)
2022-05-18T04:24:31.2992442Z 
2022-05-18T04:24:31.2992875Z ----------------------------------------------------------------------
2022-05-18T04:24:31.2993241Z Ran 1 test in 4.618s
2022-05-18T04:24:31.2993406Z 
2022-05-18T04:24:31.2993500Z OK
2022-05-18T04:24:31.2993636Z 
2022-05-18T04:24:31.2993765Z Generating XML reports...
2022-05-18T04:24:31.3038258Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042426.xml
2022-05-18T04:24:32.6012082Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:24:32.6023382Z 
2022-05-18T04:24:32.6023689Z Running tests...
2022-05-18T04:24:32.6024186Z ----------------------------------------------------------------------
2022-05-18T04:24:34.2491713Z   test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:34.2925343Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41872
2022-05-18T04:24:34.3036301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41873
2022-05-18T04:24:35.2455547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:35.2478243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:35.2619103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:35.2642634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:35.2643596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:35.2683484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:24:36.5868548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38f04n7q
2022-05-18T04:24:36.5869189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38f04n7q/_remote_module_non_scriptable.py
2022-05-18T04:24:36.5944361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbg1dhjlu
2022-05-18T04:24:36.5945306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbg1dhjlu/_remote_module_non_scriptable.py
2022-05-18T04:24:37.2123798Z ok (4.610s)
2022-05-18T04:24:37.2124082Z 
2022-05-18T04:24:37.2124515Z ----------------------------------------------------------------------
2022-05-18T04:24:37.2124861Z Ran 1 test in 4.610s
2022-05-18T04:24:37.2125032Z 
2022-05-18T04:24:37.2125127Z OK
2022-05-18T04:24:37.2125301Z 
2022-05-18T04:24:37.2125454Z Generating XML reports...
2022-05-18T04:24:37.2170842Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042432.xml
2022-05-18T04:24:38.5236234Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:24:38.5246894Z 
2022-05-18T04:24:38.5247267Z Running tests...
2022-05-18T04:24:38.5247762Z ----------------------------------------------------------------------
2022-05-18T04:24:40.1941431Z   test_fp16 (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:40.2383614Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42092
2022-05-18T04:24:40.2504407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42093
2022-05-18T04:24:41.2099518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:41.2385595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:42.5673789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqlve78p
2022-05-18T04:24:42.5674429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqlve78p/_remote_module_non_scriptable.py
2022-05-18T04:24:42.5733411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps4e_xta1
2022-05-18T04:24:42.5734060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps4e_xta1/_remote_module_non_scriptable.py
2022-05-18T04:24:43.1591133Z ok (4.634s)
2022-05-18T04:24:43.1591381Z 
2022-05-18T04:24:43.1591845Z ----------------------------------------------------------------------
2022-05-18T04:24:43.1592196Z Ran 1 test in 4.634s
2022-05-18T04:24:43.1592362Z 
2022-05-18T04:24:43.1592459Z OK
2022-05-18T04:24:43.1592575Z 
2022-05-18T04:24:43.1595954Z Generating XML reports...
2022-05-18T04:24:43.1635896Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042438.xml
2022-05-18T04:24:44.3879346Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:24:44.3887918Z 
2022-05-18T04:24:44.3888135Z Running tests...
2022-05-18T04:24:44.3888631Z ----------------------------------------------------------------------
2022-05-18T04:24:46.0177051Z   test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:46.0616800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42312
2022-05-18T04:24:46.0734555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42313
2022-05-18T04:24:47.0129871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:47.0130774Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:24:47.0377656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:47.0382003Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:24:48.3409906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstzqoedx
2022-05-18T04:24:48.3410582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstzqoedx/_remote_module_non_scriptable.py
2022-05-18T04:24:48.3512539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1408rno
2022-05-18T04:24:48.3513126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1408rno/_remote_module_non_scriptable.py
2022-05-18T04:24:48.7812428Z ok (4.392s)
2022-05-18T04:24:48.7812630Z 
2022-05-18T04:24:48.7813430Z ----------------------------------------------------------------------
2022-05-18T04:24:48.7813817Z Ran 1 test in 4.392s
2022-05-18T04:24:48.7813986Z 
2022-05-18T04:24:48.7814083Z OK
2022-05-18T04:24:48.7814220Z 
2022-05-18T04:24:48.7814340Z Generating XML reports...
2022-05-18T04:24:48.7856615Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042444.xml
2022-05-18T04:24:50.0398911Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:24:50.0408815Z 
2022-05-18T04:24:50.0409116Z Running tests...
2022-05-18T04:24:50.0409776Z ----------------------------------------------------------------------
2022-05-18T04:24:51.6579598Z   test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:51.6998347Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42532
2022-05-18T04:24:51.7113145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42533
2022-05-18T04:24:52.6841288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:52.6842228Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:24:52.6938537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:52.6939529Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:24:54.0009965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9i6wtgyb
2022-05-18T04:24:54.0010614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9i6wtgyb/_remote_module_non_scriptable.py
2022-05-18T04:24:54.0095488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vj6lu72
2022-05-18T04:24:54.0096132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vj6lu72/_remote_module_non_scriptable.py
2022-05-18T04:24:54.4190747Z ok (4.378s)
2022-05-18T04:24:54.4191073Z 
2022-05-18T04:24:54.4191517Z ----------------------------------------------------------------------
2022-05-18T04:24:54.4191883Z Ran 1 test in 4.378s
2022-05-18T04:24:54.4192065Z 
2022-05-18T04:24:54.4192167Z OK
2022-05-18T04:24:54.4192318Z 
2022-05-18T04:24:54.4192441Z Generating XML reports...
2022-05-18T04:24:54.4236355Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042450.xml
2022-05-18T04:24:55.7088753Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:24:55.7098307Z 
2022-05-18T04:24:55.7098518Z Running tests...
2022-05-18T04:24:57.3857440Z ----------------------------------------------------------------------
2022-05-18T04:24:57.3858433Z   test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:24:57.4299991Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42752
2022-05-18T04:24:57.4422562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42753
2022-05-18T04:24:58.4098052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:58.4252497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:59.7185398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswkc2e2g
2022-05-18T04:24:59.7186114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswkc2e2g/_remote_module_non_scriptable.py
2022-05-18T04:24:59.7536260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn26_vld3
2022-05-18T04:24:59.7536859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn26_vld3/_remote_module_non_scriptable.py
2022-05-18T04:25:00.3506519Z ok (4.640s)
2022-05-18T04:25:00.3506768Z 
2022-05-18T04:25:00.3507216Z ----------------------------------------------------------------------
2022-05-18T04:25:00.3507576Z Ran 1 test in 4.641s
2022-05-18T04:25:00.3507750Z 
2022-05-18T04:25:00.3507855Z OK
2022-05-18T04:25:00.3508008Z 
2022-05-18T04:25:00.3508133Z Generating XML reports...
2022-05-18T04:25:00.3551625Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042455.xml
2022-05-18T04:25:01.6532943Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:25:01.6543974Z 
2022-05-18T04:25:01.6544248Z Running tests...
2022-05-18T04:25:01.6544731Z ----------------------------------------------------------------------
2022-05-18T04:25:03.3066885Z   test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:03.3484308Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42972
2022-05-18T04:25:03.3602928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42973
2022-05-18T04:25:04.3644928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:04.3706543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:05.7065116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvl0v2ax4
2022-05-18T04:25:05.7065803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvl0v2ax4/_remote_module_non_scriptable.py
2022-05-18T04:25:05.7190386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4k00hdt6
2022-05-18T04:25:05.7191532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4k00hdt6/_remote_module_non_scriptable.py
2022-05-18T04:25:06.4345604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.4346413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.4625941Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.4626528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.4901075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.4903889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.5191140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.5191675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.5464512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.5465430Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.5744505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.5745038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.6023752Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.6024287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.6313707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.6314411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.6606512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.6607046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.6912226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.6912757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.7205020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.7205554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.7505002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.7506447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.7791245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.7791784Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.8075448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.8076006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.8357266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.8358563Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.8653980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.8654954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.8942742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.8943680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.9235175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:06.9236162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:07.2713970Z ok (5.617s)
2022-05-18T04:25:07.2714175Z 
2022-05-18T04:25:07.2714615Z ----------------------------------------------------------------------
2022-05-18T04:25:07.2714978Z Ran 1 test in 5.617s
2022-05-18T04:25:07.2715147Z 
2022-05-18T04:25:07.2715320Z OK
2022-05-18T04:25:07.2715458Z 
2022-05-18T04:25:07.2715580Z Generating XML reports...
2022-05-18T04:25:07.2760150Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042501.xml
2022-05-18T04:25:08.5547008Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:25:08.5557486Z 
2022-05-18T04:25:08.5557976Z Running tests...
2022-05-18T04:25:08.5558988Z ----------------------------------------------------------------------
2022-05-18T04:25:10.2035142Z   test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:10.2475838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43192
2022-05-18T04:25:10.2585840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43193
2022-05-18T04:25:11.2167563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:11.2770155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:13.6274421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa70j6stp
2022-05-18T04:25:13.6275316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa70j6stp/_remote_module_non_scriptable.py
2022-05-18T04:25:13.6700825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcf6819cf
2022-05-18T04:25:13.6701397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcf6819cf/_remote_module_non_scriptable.py
2022-05-18T04:25:14.8809266Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:25:14.8822972Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:25:14.8827379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:14.8827875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:14.9117154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:14.9117687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:14.9399445Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:14.9399976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:14.9689437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:14.9689974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:14.9976087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:14.9976623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.0267360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.0267888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.0560935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.0561464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.0861671Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.0862581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.1160492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.1161021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.1470552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.1471082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.1775597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.1776133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.2086663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.2087202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.2382012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.2383244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.2673188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.2673728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.2971298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.2971833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.3278278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.3278850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.3576989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.3577530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.3881359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.3881872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:15.8729243Z ok (7.317s)
2022-05-18T04:25:15.8729437Z 
2022-05-18T04:25:15.8729851Z ----------------------------------------------------------------------
2022-05-18T04:25:15.8730190Z Ran 1 test in 7.317s
2022-05-18T04:25:15.8730358Z 
2022-05-18T04:25:15.8730456Z OK
2022-05-18T04:25:15.8731288Z 
2022-05-18T04:25:15.8731446Z Generating XML reports...
2022-05-18T04:25:15.8774753Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042508.xml
2022-05-18T04:25:17.1375108Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:25:17.1384643Z 
2022-05-18T04:25:17.1384910Z Running tests...
2022-05-18T04:25:17.1385381Z ----------------------------------------------------------------------
2022-05-18T04:25:18.7804102Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:18.8244760Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43421
2022-05-18T04:25:18.8357960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43422
2022-05-18T04:25:19.7913451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:19.7917655Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:25:19.7918890Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:25:19.7920000Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:25:19.7921083Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:25:19.7922528Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:25:19.7923694Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:25:19.8277779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:19.8284265Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:25:19.8285424Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:25:19.8286518Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:25:19.8287606Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:25:19.8288693Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:25:19.8289773Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:25:20.0404390Z ok (2.902s)
2022-05-18T04:25:20.0404616Z 
2022-05-18T04:25:20.0405107Z ----------------------------------------------------------------------
2022-05-18T04:25:20.0405435Z Ran 1 test in 2.902s
2022-05-18T04:25:20.0405600Z 
2022-05-18T04:25:20.0405694Z OK
2022-05-18T04:25:20.0405838Z 
2022-05-18T04:25:20.0405972Z Generating XML reports...
2022-05-18T04:25:20.0448961Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042517.xml
2022-05-18T04:25:21.3195006Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:25:21.3205450Z 
2022-05-18T04:25:21.3205606Z Running tests...
2022-05-18T04:25:21.3206316Z ----------------------------------------------------------------------
2022-05-18T04:25:22.9812791Z   test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:23.0256486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43620
2022-05-18T04:25:23.0369385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43621
2022-05-18T04:25:24.0070599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:24.0322005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:25.3216185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_8d_zq3
2022-05-18T04:25:25.3216893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_8d_zq3/_remote_module_non_scriptable.py
2022-05-18T04:25:25.3224473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeginx5uj
2022-05-18T04:25:25.3225117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeginx5uj/_remote_module_non_scriptable.py
2022-05-18T04:25:25.9449994Z ok (4.624s)
2022-05-18T04:25:25.9450231Z 
2022-05-18T04:25:25.9450679Z ----------------------------------------------------------------------
2022-05-18T04:25:25.9451149Z Ran 1 test in 4.624s
2022-05-18T04:25:25.9451657Z 
2022-05-18T04:25:25.9451761Z OK
2022-05-18T04:25:25.9451893Z 
2022-05-18T04:25:25.9452040Z Generating XML reports...
2022-05-18T04:25:25.9495107Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042521.xml
2022-05-18T04:25:27.1922178Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:25:27.1933572Z 
2022-05-18T04:25:27.1934063Z Running tests...
2022-05-18T04:25:27.1934573Z ----------------------------------------------------------------------
2022-05-18T04:25:28.8630301Z   test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:28.9075643Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43840
2022-05-18T04:25:28.9199429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43841
2022-05-18T04:25:29.8828193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:29.9067079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:31.1868452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2y0vwo1
2022-05-18T04:25:31.1869084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2y0vwo1/_remote_module_non_scriptable.py
2022-05-18T04:25:31.1969847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzu_jnvm
2022-05-18T04:25:31.1970438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzu_jnvm/_remote_module_non_scriptable.py
2022-05-18T04:25:31.8286713Z ok (4.635s)
2022-05-18T04:25:31.8286948Z 
2022-05-18T04:25:31.8287390Z ----------------------------------------------------------------------
2022-05-18T04:25:31.8287748Z Ran 1 test in 4.635s
2022-05-18T04:25:31.8287915Z 
2022-05-18T04:25:31.8288008Z OK
2022-05-18T04:25:31.8288141Z 
2022-05-18T04:25:31.8288278Z Generating XML reports...
2022-05-18T04:25:31.8333758Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042527.xml
2022-05-18T04:25:33.1037652Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:25:33.1046973Z 
2022-05-18T04:25:33.1047251Z Running tests...
2022-05-18T04:25:33.1047730Z ----------------------------------------------------------------------
2022-05-18T04:25:34.7174912Z   test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:34.7609841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44060
2022-05-18T04:25:34.7732689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44061
2022-05-18T04:25:35.7066941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:35.7553692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:37.0008581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4by4utg6
2022-05-18T04:25:37.0009229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4by4utg6/_remote_module_non_scriptable.py
2022-05-18T04:25:37.0135586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4w3ee1zc
2022-05-18T04:25:37.0136463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4w3ee1zc/_remote_module_non_scriptable.py
2022-05-18T04:25:37.2961951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:37.2966222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:37.5813840Z ok (4.476s)
2022-05-18T04:25:37.5814072Z 
2022-05-18T04:25:37.5814516Z ----------------------------------------------------------------------
2022-05-18T04:25:37.5814880Z Ran 1 test in 4.477s
2022-05-18T04:25:37.5815052Z 
2022-05-18T04:25:37.5815148Z OK
2022-05-18T04:25:37.5815287Z 
2022-05-18T04:25:37.5815407Z Generating XML reports...
2022-05-18T04:25:37.5860936Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042533.xml
2022-05-18T04:25:38.8753141Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:25:38.8762082Z 
2022-05-18T04:25:38.8762581Z Running tests...
2022-05-18T04:25:38.8763221Z ----------------------------------------------------------------------
2022-05-18T04:25:40.5344420Z   test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:40.5790336Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44280
2022-05-18T04:25:40.5915225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44281
2022-05-18T04:25:41.5386408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:41.5808077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:42.8160704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4t14uha1
2022-05-18T04:25:42.8161363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4t14uha1/_remote_module_non_scriptable.py
2022-05-18T04:25:42.8490944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhnfply0
2022-05-18T04:25:42.8491530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhnfply0/_remote_module_non_scriptable.py
2022-05-18T04:25:43.1268656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:43.1269246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:43.4000445Z ok (4.523s)
2022-05-18T04:25:43.4000665Z 
2022-05-18T04:25:43.4001103Z ----------------------------------------------------------------------
2022-05-18T04:25:43.4001438Z Ran 1 test in 4.524s
2022-05-18T04:25:43.4003764Z 
2022-05-18T04:25:43.4004284Z OK
2022-05-18T04:25:43.4004528Z 
2022-05-18T04:25:43.4004685Z Generating XML reports...
2022-05-18T04:25:43.4046637Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042538.xml
2022-05-18T04:25:44.6838152Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:25:44.6847632Z 
2022-05-18T04:25:44.6847819Z Running tests...
2022-05-18T04:25:44.6848511Z ----------------------------------------------------------------------
2022-05-18T04:25:46.2864142Z   test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:46.3275916Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44500
2022-05-18T04:25:46.3381826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44501
2022-05-18T04:25:47.3074751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:47.3096062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:49.6940195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprw6isy1l
2022-05-18T04:25:49.6941119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprw6isy1l/_remote_module_non_scriptable.py
2022-05-18T04:25:49.7145757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1689czuy
2022-05-18T04:25:49.7146359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1689czuy/_remote_module_non_scriptable.py
2022-05-18T04:25:50.2051884Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:25:50.2052483Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:50.2096934Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:25:50.2100969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:25:50.6497197Z ok (5.965s)
2022-05-18T04:25:50.6497442Z 
2022-05-18T04:25:50.6497876Z ----------------------------------------------------------------------
2022-05-18T04:25:50.6498249Z Ran 1 test in 5.965s
2022-05-18T04:25:50.6498439Z 
2022-05-18T04:25:50.6498516Z OK
2022-05-18T04:25:50.6498654Z 
2022-05-18T04:25:50.6498798Z Generating XML reports...
2022-05-18T04:25:50.6543064Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042544.xml
2022-05-18T04:25:51.9169280Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:25:51.9180481Z 
2022-05-18T04:25:51.9180961Z Running tests...
2022-05-18T04:25:51.9181439Z ----------------------------------------------------------------------
2022-05-18T04:25:53.6022311Z   test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:53.6465613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44729
2022-05-18T04:25:53.6587878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44730
2022-05-18T04:25:54.6063918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:54.6341333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:54.8633616Z skip: Need at least 8 CUDA devices (2.945s)
2022-05-18T04:25:54.8633847Z 
2022-05-18T04:25:54.8634572Z ----------------------------------------------------------------------
2022-05-18T04:25:54.8634945Z Ran 1 test in 2.946s
2022-05-18T04:25:54.8635122Z 
2022-05-18T04:25:54.8635243Z OK (skipped=1)
2022-05-18T04:25:54.8635412Z 
2022-05-18T04:25:54.8635543Z Generating XML reports...
2022-05-18T04:25:54.8679314Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042551.xml
2022-05-18T04:25:56.1273917Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:25:56.1283410Z 
2022-05-18T04:25:56.1283724Z Running tests...
2022-05-18T04:25:56.1284306Z ----------------------------------------------------------------------
2022-05-18T04:25:57.7434746Z   test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:25:57.7865371Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44928
2022-05-18T04:25:57.7990458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44929
2022-05-18T04:25:58.7493629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:58.7587830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:00.4068627Z ok (4.278s)
2022-05-18T04:26:00.4068858Z 
2022-05-18T04:26:00.4069292Z ----------------------------------------------------------------------
2022-05-18T04:26:00.4069644Z Ran 1 test in 4.279s
2022-05-18T04:26:00.4069810Z 
2022-05-18T04:26:00.4069906Z OK
2022-05-18T04:26:00.4070040Z 
2022-05-18T04:26:00.4072653Z Generating XML reports...
2022-05-18T04:26:00.4113160Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042556.xml
2022-05-18T04:26:01.7318955Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:26:01.7330487Z 
2022-05-18T04:26:01.7330770Z Running tests...
2022-05-18T04:26:01.7331271Z ----------------------------------------------------------------------
2022-05-18T04:26:03.4067568Z   test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:03.4518446Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45133
2022-05-18T04:26:03.4640955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45134
2022-05-18T04:26:04.4300758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:04.4601506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:06.8538396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplo5wmqhc
2022-05-18T04:26:06.8539023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplo5wmqhc/_remote_module_non_scriptable.py
2022-05-18T04:26:06.8580210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4fjpxrz
2022-05-18T04:26:06.8580829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4fjpxrz/_remote_module_non_scriptable.py
2022-05-18T04:26:07.3112239Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:26:07.3112845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:26:07.3155930Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:26:07.3159418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:26:07.7762928Z ok (6.043s)
2022-05-18T04:26:07.7763204Z 
2022-05-18T04:26:07.7763631Z ----------------------------------------------------------------------
2022-05-18T04:26:07.7763958Z Ran 1 test in 6.043s
2022-05-18T04:26:07.7764134Z 
2022-05-18T04:26:07.7764234Z OK
2022-05-18T04:26:07.7764381Z 
2022-05-18T04:26:07.7764539Z Generating XML reports...
2022-05-18T04:26:07.7808334Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042601.xml
2022-05-18T04:26:09.0381654Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:26:09.0393352Z 
2022-05-18T04:26:09.0393675Z Running tests...
2022-05-18T04:26:09.0394149Z ----------------------------------------------------------------------
2022-05-18T04:26:10.7134938Z   test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:10.7590015Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45362
2022-05-18T04:26:10.7713768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45363
2022-05-18T04:26:11.7355907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:11.7356478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:13.0438312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25dm0u6x
2022-05-18T04:26:13.0439014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25dm0u6x/_remote_module_non_scriptable.py
2022-05-18T04:26:13.0556885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpdksxtr
2022-05-18T04:26:13.0557509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpdksxtr/_remote_module_non_scriptable.py
2022-05-18T04:26:13.3416866Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:26:13.3430951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:26:13.6801775Z ok (4.640s)
2022-05-18T04:26:13.6802038Z 
2022-05-18T04:26:13.6802498Z ----------------------------------------------------------------------
2022-05-18T04:26:13.6802835Z Ran 1 test in 4.641s
2022-05-18T04:26:13.6803040Z 
2022-05-18T04:26:13.6803154Z OK
2022-05-18T04:26:13.6803295Z 
2022-05-18T04:26:13.6803443Z Generating XML reports...
2022-05-18T04:26:13.6847144Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042609.xml
2022-05-18T04:26:14.9546794Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:26:14.9556543Z 
2022-05-18T04:26:14.9556735Z Running tests...
2022-05-18T04:26:14.9557207Z ----------------------------------------------------------------------
2022-05-18T04:26:16.5820178Z   test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:16.6243630Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45582
2022-05-18T04:26:16.6357049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45583
2022-05-18T04:26:17.6246898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:17.6378439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:18.9552340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_nk3sc0
2022-05-18T04:26:18.9553071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_nk3sc0/_remote_module_non_scriptable.py
2022-05-18T04:26:18.9661505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7j0rqjsu
2022-05-18T04:26:18.9663060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7j0rqjsu/_remote_module_non_scriptable.py
2022-05-18T04:26:19.2504694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:26:19.2536177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:26:19.5438806Z ok (4.588s)
2022-05-18T04:26:19.5439350Z 
2022-05-18T04:26:19.5440055Z ----------------------------------------------------------------------
2022-05-18T04:26:19.5440667Z Ran 1 test in 4.588s
2022-05-18T04:26:19.5440846Z 
2022-05-18T04:26:19.5440954Z OK
2022-05-18T04:26:19.5441073Z 
2022-05-18T04:26:19.5441220Z Generating XML reports...
2022-05-18T04:26:19.5483822Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042614.xml
2022-05-18T04:26:20.8245426Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:26:20.8256983Z 
2022-05-18T04:26:20.8257285Z Running tests...
2022-05-18T04:26:20.8257786Z ----------------------------------------------------------------------
2022-05-18T04:26:22.4848496Z   test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:22.5291555Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45802
2022-05-18T04:26:22.5405288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45803
2022-05-18T04:26:23.4816688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:23.5081384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:41.0844387Z ok (20.258s)
2022-05-18T04:26:41.0847972Z 
2022-05-18T04:26:41.0848610Z ----------------------------------------------------------------------
2022-05-18T04:26:41.0849234Z Ran 1 test in 20.259s
2022-05-18T04:26:41.0849837Z 
2022-05-18T04:26:41.0850088Z OK
2022-05-18T04:26:41.0850250Z 
2022-05-18T04:26:41.0850720Z Generating XML reports...
2022-05-18T04:26:41.0889397Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042620.xml
2022-05-18T04:26:42.3652794Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:26:42.3662890Z 
2022-05-18T04:26:42.3663194Z Running tests...
2022-05-18T04:26:42.3663684Z ----------------------------------------------------------------------
2022-05-18T04:26:42.3683275Z   test_no_grad (__main__.DistributedDataParallelTest)
2022-05-18T04:26:44.0216709Z Note: this test can be sped up by only running it on a CPU module ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:44.0645704Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46018
2022-05-18T04:26:44.0768020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46019
2022-05-18T04:26:45.0219117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:45.0481202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:46.3539619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqrreb8v
2022-05-18T04:26:46.3540521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqrreb8v/_remote_module_non_scriptable.py
2022-05-18T04:26:46.3648282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhwl5w37
2022-05-18T04:26:46.3648999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhwl5w37/_remote_module_non_scriptable.py
2022-05-18T04:26:46.9853114Z ok (4.619s)
2022-05-18T04:26:46.9853338Z 
2022-05-18T04:26:46.9854685Z ----------------------------------------------------------------------
2022-05-18T04:26:46.9855041Z Ran 1 test in 4.619s
2022-05-18T04:26:46.9855222Z 
2022-05-18T04:26:46.9855324Z OK
2022-05-18T04:26:46.9855462Z 
2022-05-18T04:26:46.9855600Z Generating XML reports...
2022-05-18T04:26:46.9899091Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042642.xml
2022-05-18T04:26:48.2595150Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:26:48.2604533Z 
2022-05-18T04:26:48.2604720Z Running tests...
2022-05-18T04:26:48.2605493Z ----------------------------------------------------------------------
2022-05-18T04:26:49.8644819Z   test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:49.9061612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46230
2022-05-18T04:26:49.9168884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46231
2022-05-18T04:26:50.8908243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:50.9249131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:52.1847593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehc2swmz
2022-05-18T04:26:52.1848205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehc2swmz/_remote_module_non_scriptable.py
2022-05-18T04:26:52.2526031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwll4l4n0
2022-05-18T04:26:52.2526934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwll4l4n0/_remote_module_non_scriptable.py
2022-05-18T04:26:52.6249029Z ok (4.364s)
2022-05-18T04:26:52.6249292Z 
2022-05-18T04:26:52.6249774Z ----------------------------------------------------------------------
2022-05-18T04:26:52.6250133Z Ran 1 test in 4.364s
2022-05-18T04:26:52.6250312Z 
2022-05-18T04:26:52.6250388Z OK
2022-05-18T04:26:52.6250528Z 
2022-05-18T04:26:52.6250680Z Generating XML reports...
2022-05-18T04:26:52.6294637Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042648.xml
2022-05-18T04:26:53.8676247Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:26:53.8685074Z 
2022-05-18T04:26:53.8685468Z Running tests...
2022-05-18T04:26:53.8685950Z ----------------------------------------------------------------------
2022-05-18T04:26:55.5262479Z   test_pass_default_pg (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:55.5703822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46442
2022-05-18T04:26:55.5817031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46443
2022-05-18T04:26:56.5503114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:56.5512068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:56.5512646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:56.5513756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:56.5514638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:56.5612358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:26:56.7863131Z ok (2.917s)
2022-05-18T04:26:56.7863368Z 
2022-05-18T04:26:56.7864632Z ----------------------------------------------------------------------
2022-05-18T04:26:56.7865010Z Ran 1 test in 2.918s
2022-05-18T04:26:56.7865182Z 
2022-05-18T04:26:56.7865281Z OK
2022-05-18T04:26:56.7865443Z 
2022-05-18T04:26:56.7865563Z Generating XML reports...
2022-05-18T04:26:56.7909474Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042653.xml
2022-05-18T04:26:58.0696030Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:26:58.0706611Z 
2022-05-18T04:26:58.0706984Z Running tests...
2022-05-18T04:26:58.0707521Z ----------------------------------------------------------------------
2022-05-18T04:26:59.7195605Z   test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:26:59.7641795Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46645
2022-05-18T04:26:59.7763428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46646
2022-05-18T04:27:00.7339981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:00.7342554Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:00.7597222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:00.7600010Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:02.0864987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3jdbx8_
2022-05-18T04:27:02.0865618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3jdbx8_/_remote_module_non_scriptable.py
2022-05-18T04:27:02.0968323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30iox1ps
2022-05-18T04:27:02.0969196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30iox1ps/_remote_module_non_scriptable.py
2022-05-18T04:27:02.1942901Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:02.1944102Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:02.1997583Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:02.1998709Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:02.2051202Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:02.2052364Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:02.2106862Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:02.2108013Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:02.2161165Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:02.2162305Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:02.2214027Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:02.2215325Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:02.2267790Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:02.2268937Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:02.5849893Z ok (4.514s)
2022-05-18T04:27:02.5850145Z 
2022-05-18T04:27:02.5850588Z ----------------------------------------------------------------------
2022-05-18T04:27:02.5850921Z Ran 1 test in 4.514s
2022-05-18T04:27:02.5851121Z 
2022-05-18T04:27:02.5851229Z OK
2022-05-18T04:27:02.5851371Z 
2022-05-18T04:27:02.5851513Z Generating XML reports...
2022-05-18T04:27:02.5894487Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042658.xml
2022-05-18T04:27:03.8773662Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:27:03.8784929Z 
2022-05-18T04:27:03.8785289Z Running tests...
2022-05-18T04:27:03.8785789Z ----------------------------------------------------------------------
2022-05-18T04:27:05.5384352Z   test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:05.5825923Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46865
2022-05-18T04:27:05.5950045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46866
2022-05-18T04:27:06.5361521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:06.5363127Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:06.5381818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:06.5384112Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:07.9013412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzlbbxsbr
2022-05-18T04:27:07.9014368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzlbbxsbr/_remote_module_non_scriptable.py
2022-05-18T04:27:07.9098087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hrg7oeq
2022-05-18T04:27:07.9098937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hrg7oeq/_remote_module_non_scriptable.py
2022-05-18T04:27:08.0077154Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:08.0080993Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:08.0133996Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:08.0135190Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:08.0190682Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:08.0191855Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:08.0248881Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:08.0250020Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:08.0302861Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:08.0304043Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:08.0356622Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:08.0357943Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-05-18T04:27:08.0411015Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:08.0412180Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:27:08.4037787Z ok (4.525s)
2022-05-18T04:27:08.4037987Z 
2022-05-18T04:27:08.4039569Z ----------------------------------------------------------------------
2022-05-18T04:27:08.4040320Z Ran 1 test in 4.525s
2022-05-18T04:27:08.4040673Z 
2022-05-18T04:27:08.4040840Z OK
2022-05-18T04:27:08.4041099Z 
2022-05-18T04:27:08.4041339Z Generating XML reports...
2022-05-18T04:27:08.4086228Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042703.xml
2022-05-18T04:27:09.7123098Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:27:09.7133163Z 
2022-05-18T04:27:09.7133820Z Running tests...
2022-05-18T04:27:09.7134297Z ----------------------------------------------------------------------
2022-05-18T04:27:11.3909712Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:11.4350833Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47085
2022-05-18T04:27:11.4475819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47086
2022-05-18T04:27:12.3899233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:12.4411694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:13.7043363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7h6if432
2022-05-18T04:27:13.7044003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7h6if432/_remote_module_non_scriptable.py
2022-05-18T04:27:13.7579403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbtfw62a6
2022-05-18T04:27:13.7580054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbtfw62a6/_remote_module_non_scriptable.py
2022-05-18T04:27:14.6036686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:27:14.6037541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:27:14.9572293Z ok (5.244s)
2022-05-18T04:27:14.9572510Z 
2022-05-18T04:27:14.9572946Z ----------------------------------------------------------------------
2022-05-18T04:27:14.9573295Z Ran 1 test in 5.244s
2022-05-18T04:27:14.9573465Z 
2022-05-18T04:27:14.9573573Z OK
2022-05-18T04:27:14.9573694Z 
2022-05-18T04:27:14.9575637Z Generating XML reports...
2022-05-18T04:27:14.9619489Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042709.xml
2022-05-18T04:27:16.2357546Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:27:16.2368860Z 
2022-05-18T04:27:16.2369326Z Running tests...
2022-05-18T04:27:16.2369949Z ----------------------------------------------------------------------
2022-05-18T04:27:17.8962053Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:17.9392235Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47305
2022-05-18T04:27:17.9514030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47306
2022-05-18T04:27:18.8973316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:18.8973870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:20.2040877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp207jkw3f
2022-05-18T04:27:20.2041536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp207jkw3f/_remote_module_non_scriptable.py
2022-05-18T04:27:20.2118789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcn5j2g_f
2022-05-18T04:27:20.2119411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcn5j2g_f/_remote_module_non_scriptable.py
2022-05-18T04:27:20.8930612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:27:20.8931178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:27:21.2607085Z ok (5.024s)
2022-05-18T04:27:21.2607321Z 
2022-05-18T04:27:21.2607794Z ----------------------------------------------------------------------
2022-05-18T04:27:21.2608159Z Ran 1 test in 5.024s
2022-05-18T04:27:21.2608308Z 
2022-05-18T04:27:21.2608411Z OK
2022-05-18T04:27:21.2608553Z 
2022-05-18T04:27:21.2608714Z Generating XML reports...
2022-05-18T04:27:21.2654001Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042716.xml
2022-05-18T04:27:22.5144690Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:27:22.5155857Z 
2022-05-18T04:27:22.5156052Z Running tests...
2022-05-18T04:27:22.5156804Z ----------------------------------------------------------------------
2022-05-18T04:27:24.1639125Z   test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:24.2091462Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47525
2022-05-18T04:27:24.2215807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47526
2022-05-18T04:27:24.2343570Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47527
2022-05-18T04:27:25.1760741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:25.2372216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:25.2387046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:25.4393699Z ok (2.923s)
2022-05-18T04:27:25.4394065Z 
2022-05-18T04:27:25.4394583Z ----------------------------------------------------------------------
2022-05-18T04:27:25.4394916Z Ran 1 test in 2.924s
2022-05-18T04:27:25.4395086Z 
2022-05-18T04:27:25.4395189Z OK
2022-05-18T04:27:25.4395339Z 
2022-05-18T04:27:25.4395479Z Generating XML reports...
2022-05-18T04:27:25.4439418Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042722.xml
2022-05-18T04:27:26.6961873Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:27:26.6972464Z 
2022-05-18T04:27:26.6972782Z Running tests...
2022-05-18T04:27:26.6973780Z ----------------------------------------------------------------------
2022-05-18T04:27:28.3312007Z   test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:28.3765892Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47790
2022-05-18T04:27:28.3888548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47791
2022-05-18T04:27:28.4018222Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47792
2022-05-18T04:27:29.3385962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:29.3881540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:29.4465584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:41.7327227Z ok (15.035s)
2022-05-18T04:27:41.7327443Z 
2022-05-18T04:27:41.7328082Z ----------------------------------------------------------------------
2022-05-18T04:27:41.7328620Z Ran 1 test in 15.035s
2022-05-18T04:27:41.7328918Z 
2022-05-18T04:27:41.7329099Z OK
2022-05-18T04:27:41.7329365Z 
2022-05-18T04:27:41.7329561Z Generating XML reports...
2022-05-18T04:27:41.7374458Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042726.xml
2022-05-18T04:27:43.0255897Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:27:43.0265499Z 
2022-05-18T04:27:43.0265819Z Running tests...
2022-05-18T04:27:43.0266331Z ----------------------------------------------------------------------
2022-05-18T04:27:43.0272139Z   test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: Frequently times out see https://github.com/pytorch/pytorch/issues/58920 (0.001s)
2022-05-18T04:27:43.0272550Z 
2022-05-18T04:27:43.0272859Z ----------------------------------------------------------------------
2022-05-18T04:27:43.0273222Z Ran 1 test in 0.001s
2022-05-18T04:27:43.0273390Z 
2022-05-18T04:27:43.0273507Z OK (skipped=1)
2022-05-18T04:27:43.0273668Z 
2022-05-18T04:27:43.0275230Z Generating XML reports...
2022-05-18T04:27:43.0309230Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042743.xml
2022-05-18T04:27:44.0966829Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:27:44.0977931Z 
2022-05-18T04:27:44.0978379Z Running tests...
2022-05-18T04:27:44.0979293Z ----------------------------------------------------------------------
2022-05-18T04:27:45.7568566Z   test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:27:45.8007955Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48136
2022-05-18T04:27:45.8130015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48137
2022-05-18T04:27:45.8259155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48138
2022-05-18T04:27:46.8113113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:46.8124961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:46.8163885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:00.7614209Z ok (16.663s)
2022-05-18T04:28:00.7614426Z 
2022-05-18T04:28:00.7615428Z ----------------------------------------------------------------------
2022-05-18T04:28:00.7615797Z Ran 1 test in 16.663s
2022-05-18T04:28:00.7615994Z 
2022-05-18T04:28:00.7616110Z OK
2022-05-18T04:28:00.7616253Z 
2022-05-18T04:28:00.7616410Z Generating XML reports...
2022-05-18T04:28:00.7659511Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042744.xml
2022-05-18T04:28:02.0704996Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:28:02.0715120Z 
2022-05-18T04:28:02.0715431Z Running tests...
2022-05-18T04:28:02.0715911Z ----------------------------------------------------------------------
2022-05-18T04:28:03.6866216Z   test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:03.7288362Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48417
2022-05-18T04:28:03.7414674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48418
2022-05-18T04:28:03.7535156Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48419
2022-05-18T04:28:04.7437082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:04.7542879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:04.7548866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:16.6993714Z [W ProcessGroupNCCL.cpp:865] [Rank 2] Found key in store: NCCLABORTEDCOMM:20d68fac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-05-18T04:28:16.6997286Z [W ProcessGroupNCCL.cpp:865] [Rank 0] Found key in store: NCCLABORTEDCOMM:20d68fac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-05-18T04:28:17.3867851Z ok (15.315s)
2022-05-18T04:28:17.3868091Z 
2022-05-18T04:28:17.3868540Z ----------------------------------------------------------------------
2022-05-18T04:28:17.3868891Z Ran 1 test in 15.315s
2022-05-18T04:28:17.3869082Z 
2022-05-18T04:28:17.3869182Z OK
2022-05-18T04:28:17.3869327Z 
2022-05-18T04:28:17.3869473Z Generating XML reports...
2022-05-18T04:28:17.3913806Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042802.xml
2022-05-18T04:28:18.6674706Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:28:18.6685057Z 
2022-05-18T04:28:18.6685264Z Running tests...
2022-05-18T04:28:18.6686270Z ----------------------------------------------------------------------
2022-05-18T04:28:20.3255769Z   test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:20.3699749Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48698
2022-05-18T04:28:20.3820854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48699
2022-05-18T04:28:20.3952662Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48700
2022-05-18T04:28:21.3169439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:21.3505880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:21.3796001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:35.2313362Z ok (16.562s)
2022-05-18T04:28:35.2313601Z 
2022-05-18T04:28:35.2314049Z ----------------------------------------------------------------------
2022-05-18T04:28:35.2314404Z Ran 1 test in 16.563s
2022-05-18T04:28:35.2314576Z 
2022-05-18T04:28:35.2314704Z OK
2022-05-18T04:28:35.2314850Z 
2022-05-18T04:28:35.2314993Z Generating XML reports...
2022-05-18T04:28:35.2358423Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042818.xml
2022-05-18T04:28:36.5179305Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:28:36.5190514Z 
2022-05-18T04:28:36.5190786Z Running tests...
2022-05-18T04:28:36.5191256Z ----------------------------------------------------------------------
2022-05-18T04:28:38.1820024Z   test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:38.2271512Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48979
2022-05-18T04:28:38.2392338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48980
2022-05-18T04:28:38.2517554Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48981
2022-05-18T04:28:39.2083090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:39.2411911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:39.2609298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:51.1636790Z [W ProcessGroupNCCL.cpp:865] [Rank 0] Found key in store: NCCLABORTEDCOMM:20cee5ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-05-18T04:28:51.1656618Z [W ProcessGroupNCCL.cpp:865] [Rank 2] Found key in store: NCCLABORTEDCOMM:20cee5ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-05-18T04:28:51.8854567Z ok (15.366s)
2022-05-18T04:28:51.8854901Z 
2022-05-18T04:28:51.8855925Z ----------------------------------------------------------------------
2022-05-18T04:28:51.8856295Z Ran 1 test in 15.366s
2022-05-18T04:28:51.8856489Z 
2022-05-18T04:28:51.8856599Z OK
2022-05-18T04:28:51.8856739Z 
2022-05-18T04:28:51.8856878Z Generating XML reports...
2022-05-18T04:28:51.8900370Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042836.xml
2022-05-18T04:28:53.1495630Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:28:53.1503703Z 
2022-05-18T04:28:53.1504026Z Running tests...
2022-05-18T04:28:53.1504510Z ----------------------------------------------------------------------
2022-05-18T04:28:53.1517482Z   test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: Test does not pass when run locally (0.001s)
2022-05-18T04:28:53.1517826Z 
2022-05-18T04:28:53.1518140Z ----------------------------------------------------------------------
2022-05-18T04:28:53.1518480Z Ran 1 test in 0.002s
2022-05-18T04:28:53.1518645Z 
2022-05-18T04:28:53.1518761Z OK (skipped=1)
2022-05-18T04:28:53.1518922Z 
2022-05-18T04:28:53.1519045Z Generating XML reports...
2022-05-18T04:28:53.1553759Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042853.xml
2022-05-18T04:28:54.2336371Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:28:54.2346269Z 
2022-05-18T04:28:54.2346747Z Running tests...
2022-05-18T04:28:54.2347227Z ----------------------------------------------------------------------
2022-05-18T04:28:55.8835767Z   test_nccl_timeout (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:28:55.9274596Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49325
2022-05-18T04:28:55.9399746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49326
2022-05-18T04:28:55.9522431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49327
2022-05-18T04:28:56.8844710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:56.8874956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:56.9102853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:08.4687440Z [W ProcessGroupNCCL.cpp:865] [Rank 0] Found key in store: NCCLABORTEDCOMM:20861dac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-05-18T04:29:08.4703368Z [W ProcessGroupNCCL.cpp:865] [Rank 1] Found key in store: NCCLABORTEDCOMM:20861dac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-05-18T04:29:08.4749091Z [W ProcessGroupNCCL.cpp:865] [Rank 2] Found key in store: NCCLABORTEDCOMM:20861dac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-05-18T04:29:09.7851389Z ok (15.550s)
2022-05-18T04:29:09.7851589Z 
2022-05-18T04:29:09.7852137Z ----------------------------------------------------------------------
2022-05-18T04:29:09.7852505Z Ran 1 test in 15.550s
2022-05-18T04:29:09.7852656Z 
2022-05-18T04:29:09.7856104Z OK
2022-05-18T04:29:09.7856423Z 
2022-05-18T04:29:09.7856574Z Generating XML reports...
2022-05-18T04:29:09.7896021Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042854.xml
2022-05-18T04:29:11.0607888Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:29:11.0618396Z 
2022-05-18T04:29:11.0618773Z Running tests...
2022-05-18T04:29:11.0619288Z ----------------------------------------------------------------------
2022-05-18T04:29:11.0624313Z   test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: GPUs are available, skipping test (0.001s)
2022-05-18T04:29:11.0624642Z 
2022-05-18T04:29:11.0624940Z ----------------------------------------------------------------------
2022-05-18T04:29:11.0625262Z Ran 1 test in 0.001s
2022-05-18T04:29:11.0625433Z 
2022-05-18T04:29:11.0625554Z OK (skipped=1)
2022-05-18T04:29:11.0626223Z 
2022-05-18T04:29:11.0626375Z Generating XML reports...
2022-05-18T04:29:11.0663261Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220518042911.xml
2022-05-18T04:29:12.1591486Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:29:12.1601638Z 
2022-05-18T04:29:12.1601912Z Running tests...
2022-05-18T04:29:12.1602447Z ----------------------------------------------------------------------
2022-05-18T04:29:13.8285050Z   test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:13.8736070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49680
2022-05-18T04:29:13.8860725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49681
2022-05-18T04:29:14.8354867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:14.8355464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:14.8449189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:14.8450856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:14.8451733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:14.8461266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:16.4938177Z ok (4.333s)
2022-05-18T04:29:16.4938391Z 
2022-05-18T04:29:16.4938828Z ----------------------------------------------------------------------
2022-05-18T04:29:16.4939209Z Ran 1 test in 4.334s
2022-05-18T04:29:16.4939383Z 
2022-05-18T04:29:16.4939460Z OK
2022-05-18T04:29:16.4939602Z 
2022-05-18T04:29:16.4939741Z Generating XML reports...
2022-05-18T04:29:16.4983395Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042912.xml
2022-05-18T04:29:17.7693968Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:29:17.7703838Z 
2022-05-18T04:29:17.7704269Z Running tests...
2022-05-18T04:29:17.7704785Z ----------------------------------------------------------------------
2022-05-18T04:29:19.3755020Z   test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:19.4174534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49885
2022-05-18T04:29:19.4290498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49886
2022-05-18T04:29:20.3680762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:20.3685342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:20.3803585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:20.3804697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:20.3805534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:20.3889294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:22.1367644Z ok (4.366s)
2022-05-18T04:29:22.1367905Z 
2022-05-18T04:29:22.1368360Z ----------------------------------------------------------------------
2022-05-18T04:29:22.1368725Z Ran 1 test in 4.366s
2022-05-18T04:29:22.1368883Z 
2022-05-18T04:29:22.1368981Z OK
2022-05-18T04:29:22.1369117Z 
2022-05-18T04:29:22.1369258Z Generating XML reports...
2022-05-18T04:29:22.1412401Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042917.xml
2022-05-18T04:29:23.4043066Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:29:23.4053235Z 
2022-05-18T04:29:23.4053605Z Running tests...
2022-05-18T04:29:23.4054109Z ----------------------------------------------------------------------
2022-05-18T04:29:25.0114622Z   test_allgather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:25.0541822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50097
2022-05-18T04:29:25.0664324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50098
2022-05-18T04:29:25.9951648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:25.9955852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:26.0054461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:26.0055017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:26.0055876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:26.0056573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:27.6744108Z ok (4.269s)
2022-05-18T04:29:27.6744323Z 
2022-05-18T04:29:27.6745031Z ----------------------------------------------------------------------
2022-05-18T04:29:27.6745395Z Ran 1 test in 4.269s
2022-05-18T04:29:27.6745572Z 
2022-05-18T04:29:27.6745693Z OK
2022-05-18T04:29:27.6745835Z 
2022-05-18T04:29:27.6745980Z Generating XML reports...
2022-05-18T04:29:27.6788813Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042923.xml
2022-05-18T04:29:28.9363981Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:29:28.9375207Z 
2022-05-18T04:29:28.9375859Z Running tests...
2022-05-18T04:29:28.9376379Z ----------------------------------------------------------------------
2022-05-18T04:29:30.5652368Z   test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:30.6098561Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50309
2022-05-18T04:29:30.6219575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50310
2022-05-18T04:29:31.5647845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:31.5648653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:31.6004456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:31.6009599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:31.6010519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:31.6059881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:33.3300210Z ok (4.392s)
2022-05-18T04:29:33.3300509Z 
2022-05-18T04:29:33.3300977Z ----------------------------------------------------------------------
2022-05-18T04:29:33.3301364Z Ran 1 test in 4.393s
2022-05-18T04:29:33.3301539Z 
2022-05-18T04:29:33.3301643Z OK
2022-05-18T04:29:33.3301760Z 
2022-05-18T04:29:33.3302156Z Generating XML reports...
2022-05-18T04:29:33.3346812Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042928.xml
2022-05-18T04:29:34.5909464Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:29:34.5918527Z 
2022-05-18T04:29:34.5918988Z Running tests...
2022-05-18T04:29:34.5919492Z ----------------------------------------------------------------------
2022-05-18T04:29:36.2632942Z   test_barrier (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:36.3084897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50521
2022-05-18T04:29:36.3207218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50522
2022-05-18T04:29:37.3047278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:37.3051317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:37.3347133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:37.3350024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:37.3350910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:37.3355051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:39.0291147Z ok (4.437s)
2022-05-18T04:29:39.0291501Z 
2022-05-18T04:29:39.0291957Z ----------------------------------------------------------------------
2022-05-18T04:29:39.0292681Z Ran 1 test in 4.437s
2022-05-18T04:29:39.0292855Z 
2022-05-18T04:29:39.0292961Z OK
2022-05-18T04:29:39.0295102Z 
2022-05-18T04:29:39.0295460Z Generating XML reports...
2022-05-18T04:29:39.0336062Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042934.xml
2022-05-18T04:29:40.3107580Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:29:40.3118543Z 
2022-05-18T04:29:40.3118907Z Running tests...
2022-05-18T04:29:40.3119381Z ----------------------------------------------------------------------
2022-05-18T04:29:41.9758043Z   test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:42.0200929Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50733
2022-05-18T04:29:42.0314527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50734
2022-05-18T04:29:43.0079969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:43.0081873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:43.0496030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:43.0498299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:43.0499162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:43.0593107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:44.7396840Z ok (4.427s)
2022-05-18T04:29:44.7397069Z 
2022-05-18T04:29:44.7397503Z ----------------------------------------------------------------------
2022-05-18T04:29:44.7397865Z Ran 1 test in 4.428s
2022-05-18T04:29:44.7398035Z 
2022-05-18T04:29:44.7398159Z OK
2022-05-18T04:29:44.7398297Z 
2022-05-18T04:29:44.7398420Z Generating XML reports...
2022-05-18T04:29:44.7441780Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042940.xml
2022-05-18T04:29:46.0236093Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:29:46.0246179Z 
2022-05-18T04:29:46.0246435Z Running tests...
2022-05-18T04:29:46.0246929Z ----------------------------------------------------------------------
2022-05-18T04:29:47.6888646Z   test_empty_tensors (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:47.7343352Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50945
2022-05-18T04:29:47.7465545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50946
2022-05-18T04:29:48.6654295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:48.6655076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:48.6789993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:48.6794069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:48.6794952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:48.6860890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:51.5574543Z ok (5.532s)
2022-05-18T04:29:51.5574769Z 
2022-05-18T04:29:51.5575229Z ----------------------------------------------------------------------
2022-05-18T04:29:51.5575583Z Ran 1 test in 5.533s
2022-05-18T04:29:51.5575836Z 
2022-05-18T04:29:51.5575937Z OK
2022-05-18T04:29:51.5576085Z 
2022-05-18T04:29:51.5576202Z Generating XML reports...
2022-05-18T04:29:51.5620155Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042946.xml
2022-05-18T04:29:52.8370852Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:29:52.8380176Z 
2022-05-18T04:29:52.8380622Z Running tests...
2022-05-18T04:29:52.8381127Z ----------------------------------------------------------------------
2022-05-18T04:29:54.4853336Z   test_gather_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:29:54.5305578Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51157
2022-05-18T04:29:54.5430716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51158
2022-05-18T04:29:55.4859203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:55.4860552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:55.5122758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:55.5123604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:55.5124630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:55.5168687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:29:57.1513369Z ok (4.313s)
2022-05-18T04:29:57.1513616Z 
2022-05-18T04:29:57.1514067Z ----------------------------------------------------------------------
2022-05-18T04:29:57.1514429Z Ran 1 test in 4.313s
2022-05-18T04:29:57.1514605Z 
2022-05-18T04:29:57.1514680Z OK
2022-05-18T04:29:57.1514827Z 
2022-05-18T04:29:57.1514972Z Generating XML reports...
2022-05-18T04:29:57.1559274Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042952.xml
2022-05-18T04:29:58.4411097Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:29:58.4421046Z 
2022-05-18T04:29:58.4421502Z Running tests...
2022-05-18T04:29:58.4422433Z ----------------------------------------------------------------------
2022-05-18T04:30:00.1135267Z   test_gather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:00.1580254Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51362
2022-05-18T04:30:00.1702678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51363
2022-05-18T04:30:01.1100562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:01.1104058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:01.1390374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:01.1393581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:01.1394463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:01.1409819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:02.8783422Z ok (4.436s)
2022-05-18T04:30:02.8783679Z 
2022-05-18T04:30:02.8784143Z ----------------------------------------------------------------------
2022-05-18T04:30:02.8784490Z Ran 1 test in 4.436s
2022-05-18T04:30:02.8784665Z 
2022-05-18T04:30:02.8784766Z OK
2022-05-18T04:30:02.8784908Z 
2022-05-18T04:30:02.8785587Z Generating XML reports...
2022-05-18T04:30:02.8832144Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042958.xml
2022-05-18T04:30:04.1589769Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:30:04.1599399Z 
2022-05-18T04:30:04.1599594Z Running tests...
2022-05-18T04:30:04.1600600Z ----------------------------------------------------------------------
2022-05-18T04:30:05.7764432Z   test_gather_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:05.8222052Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51574
2022-05-18T04:30:05.8339070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51575
2022-05-18T04:30:06.8012564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:06.8015539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:06.8241798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:06.8242460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:06.8243352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:06.8322433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:11.1471028Z ok (6.987s)
2022-05-18T04:30:11.1471261Z 
2022-05-18T04:30:11.1471733Z ----------------------------------------------------------------------
2022-05-18T04:30:11.1472079Z Ran 1 test in 6.987s
2022-05-18T04:30:11.1472251Z 
2022-05-18T04:30:11.1472353Z OK
2022-05-18T04:30:11.1472513Z 
2022-05-18T04:30:11.1472634Z Generating XML reports...
2022-05-18T04:30:11.1517997Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043004.xml
2022-05-18T04:30:12.4655511Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:30:12.4666072Z 
2022-05-18T04:30:12.4666437Z Running tests...
2022-05-18T04:30:12.4667160Z ----------------------------------------------------------------------
2022-05-18T04:30:14.1357232Z   test_reduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:14.1806252Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51786
2022-05-18T04:30:14.1929112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51787
2022-05-18T04:30:15.1348003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:15.1353444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:15.1697242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:15.1699017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:15.1759154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:15.1759888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:16.9012138Z ok (4.434s)
2022-05-18T04:30:16.9012556Z 
2022-05-18T04:30:16.9013311Z ----------------------------------------------------------------------
2022-05-18T04:30:16.9013689Z Ran 1 test in 4.435s
2022-05-18T04:30:16.9013865Z 
2022-05-18T04:30:16.9013964Z OK
2022-05-18T04:30:16.9014104Z 
2022-05-18T04:30:16.9014230Z Generating XML reports...
2022-05-18T04:30:16.9058229Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043012.xml
2022-05-18T04:30:18.1885419Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:30:18.1895941Z 
2022-05-18T04:30:18.1896288Z Running tests...
2022-05-18T04:30:18.1897221Z ----------------------------------------------------------------------
2022-05-18T04:30:19.8281263Z   test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:19.8711584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51998
2022-05-18T04:30:19.8834160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51999
2022-05-18T04:30:20.8446181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:20.8447336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:20.8474415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:20.8475833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:20.8476924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:20.8551606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:22.4912160Z ok (4.301s)
2022-05-18T04:30:22.4912640Z 
2022-05-18T04:30:22.4913470Z ----------------------------------------------------------------------
2022-05-18T04:30:22.4913824Z Ran 1 test in 4.302s
2022-05-18T04:30:22.4914002Z 
2022-05-18T04:30:22.4914145Z OK
2022-05-18T04:30:22.4914286Z 
2022-05-18T04:30:22.4914437Z Generating XML reports...
2022-05-18T04:30:22.4958418Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043018.xml
2022-05-18T04:30:23.7442449Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:30:23.7453273Z 
2022-05-18T04:30:23.7453718Z Running tests...
2022-05-18T04:30:23.7454324Z ----------------------------------------------------------------------
2022-05-18T04:30:25.3963150Z   test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:25.4404462Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52203
2022-05-18T04:30:25.4527740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52204
2022-05-18T04:30:26.4085581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:26.4089724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:26.4397133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:26.4397838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:26.4398696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:26.4497824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:28.1611052Z ok (4.415s)
2022-05-18T04:30:28.1611295Z 
2022-05-18T04:30:28.1611724Z ----------------------------------------------------------------------
2022-05-18T04:30:28.1612048Z Ran 1 test in 4.416s
2022-05-18T04:30:28.1612220Z 
2022-05-18T04:30:28.1612318Z OK
2022-05-18T04:30:28.1612452Z 
2022-05-18T04:30:28.1612591Z Generating XML reports...
2022-05-18T04:30:28.1656096Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043023.xml
2022-05-18T04:30:29.4476353Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:30:29.4487712Z 
2022-05-18T04:30:29.4488029Z Running tests...
2022-05-18T04:30:29.4488520Z ----------------------------------------------------------------------
2022-05-18T04:30:31.1160829Z   test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:31.1605539Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52415
2022-05-18T04:30:31.1728381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52416
2022-05-18T04:30:32.1443508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:32.1445832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:32.1990286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:32.1992162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:32.1993058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:32.2054771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:33.8807462Z ok (4.432s)
2022-05-18T04:30:33.8807727Z 
2022-05-18T04:30:33.8808156Z ----------------------------------------------------------------------
2022-05-18T04:30:33.8808504Z Ran 1 test in 4.432s
2022-05-18T04:30:33.8808672Z 
2022-05-18T04:30:33.8808769Z OK
2022-05-18T04:30:33.8808886Z 
2022-05-18T04:30:33.8809033Z Generating XML reports...
2022-05-18T04:30:33.8851421Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043029.xml
2022-05-18T04:30:35.1248268Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:30:35.1259086Z 
2022-05-18T04:30:35.1260049Z Running tests...
2022-05-18T04:30:35.1260570Z ----------------------------------------------------------------------
2022-05-18T04:30:36.7815159Z   test_scatter_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:36.8250816Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52627
2022-05-18T04:30:36.8371706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52628
2022-05-18T04:30:37.7601203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:37.7602126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:37.7805818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:37.7806851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:37.7807978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:37.7909845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:39.3448186Z ok (4.219s)
2022-05-18T04:30:39.3448418Z 
2022-05-18T04:30:39.3448847Z ----------------------------------------------------------------------
2022-05-18T04:30:39.3449208Z Ran 1 test in 4.219s
2022-05-18T04:30:39.3449379Z 
2022-05-18T04:30:39.3449480Z OK
2022-05-18T04:30:39.3449618Z 
2022-05-18T04:30:39.3449737Z Generating XML reports...
2022-05-18T04:30:39.3491948Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043035.xml
2022-05-18T04:30:40.6163076Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:30:40.6174092Z 
2022-05-18T04:30:40.6174541Z Running tests...
2022-05-18T04:30:40.6175041Z ----------------------------------------------------------------------
2022-05-18T04:30:42.2717629Z   test_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:42.3170387Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52832
2022-05-18T04:30:42.3295443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52833
2022-05-18T04:30:43.2987882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:43.2988417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:43.2991736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:43.2992256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:43.2993436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:43.2994158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:44.9376820Z ok (4.320s)
2022-05-18T04:30:44.9377071Z 
2022-05-18T04:30:44.9377520Z ----------------------------------------------------------------------
2022-05-18T04:30:44.9377893Z Ran 1 test in 4.320s
2022-05-18T04:30:44.9378039Z 
2022-05-18T04:30:44.9378137Z OK
2022-05-18T04:30:44.9378275Z 
2022-05-18T04:30:44.9378415Z Generating XML reports...
2022-05-18T04:30:44.9424482Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043040.xml
2022-05-18T04:30:46.2067677Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:30:46.2078418Z 
2022-05-18T04:30:46.2079161Z Running tests...
2022-05-18T04:30:46.2079645Z ----------------------------------------------------------------------
2022-05-18T04:30:47.8577209Z   test_scatter_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:47.9024647Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53044
2022-05-18T04:30:47.9145317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53045
2022-05-18T04:30:48.8553151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:48.8555684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:48.8889860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:48.8891102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:48.8891984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:48.8962380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:30:53.2273722Z ok (7.019s)
2022-05-18T04:30:53.2273967Z 
2022-05-18T04:30:53.2274395Z ----------------------------------------------------------------------
2022-05-18T04:30:53.2274730Z Ran 1 test in 7.020s
2022-05-18T04:30:53.2274918Z 
2022-05-18T04:30:53.2275018Z OK
2022-05-18T04:30:53.2275159Z 
2022-05-18T04:30:53.2275305Z Generating XML reports...
2022-05-18T04:30:53.2318569Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043046.xml
2022-05-18T04:30:54.5097859Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:30:54.5108410Z 
2022-05-18T04:30:54.5108596Z Running tests...
2022-05-18T04:30:54.5109476Z ----------------------------------------------------------------------
2022-05-18T04:30:56.1528953Z   test_common_errors (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:56.1720285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:56.1721222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T04:30:56.1742265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:56.1743357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T04:30:56.1762902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:56.1763655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T04:30:56.1781869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:56.1783156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T04:30:56.1846366Z ok (1.674s)
2022-05-18T04:30:56.1846709Z 
2022-05-18T04:30:56.1847399Z ----------------------------------------------------------------------
2022-05-18T04:30:56.1847800Z Ran 1 test in 1.674s
2022-05-18T04:30:56.1847978Z 
2022-05-18T04:30:56.1848081Z OK
2022-05-18T04:30:56.1848232Z 
2022-05-18T04:30:56.1848376Z Generating XML reports...
2022-05-18T04:30:56.1882951Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220518043054.xml
2022-05-18T04:30:57.3756533Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:30:57.3766859Z 
2022-05-18T04:30:57.3767136Z Running tests...
2022-05-18T04:30:57.3767884Z ----------------------------------------------------------------------
2022-05-18T04:30:59.0305350Z   test_default_store_timeout_nccl (__main__.TimeoutTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:30:59.0484279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:59.0485180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T04:31:01.0580090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:01.0581032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T04:31:02.0633684Z ok (4.686s)
2022-05-18T04:31:02.0633914Z 
2022-05-18T04:31:02.0634348Z ----------------------------------------------------------------------
2022-05-18T04:31:02.0634680Z Ran 1 test in 4.687s
2022-05-18T04:31:02.0634863Z 
2022-05-18T04:31:02.0634958Z OK
2022-05-18T04:31:02.0635157Z 
2022-05-18T04:31:02.0635294Z Generating XML reports...
2022-05-18T04:31:02.0675791Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220518043057.xml
2022-05-18T04:31:03.9106798Z 
2022-05-18T04:31:03.9107384Z real	11m52.724s
2022-05-18T04:31:03.9107668Z user	16m57.948s
2022-05-18T04:31:03.9108025Z sys	23m40.588s
2022-05-18T04:31:03.9108623Z + python test/run_test.py --verbose -i distributed/test_c10d_spawn_gloo
2022-05-18T04:31:13.2824688Z Ignoring disabled issues:  []
2022-05-18T04:31:13.2957958Z /var/lib/jenkins/workspace/test/run_test.py:894: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-05-18T04:31:13.2958552Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) == "11.6":
2022-05-18T04:31:13.2958922Z Selected tests:
2022-05-18T04:31:13.2959204Z  distributed/test_c10d_spawn_gloo
2022-05-18T04:31:13.3065023Z Prioritized test from test file changes.
2022-05-18T04:31:13.3065382Z reordering tests for PR:
2022-05-18T04:31:13.3065666Z prioritized: []
2022-05-18T04:31:13.3066237Z the rest: ['distributed/test_c10d_spawn_gloo']
2022-05-18T04:31:13.3066452Z 
2022-05-18T04:31:13.3075371Z Running distributed/test_c10d_spawn_gloo ... [2022-05-18 04:31:13.307083]
2022-05-18T04:31:13.3076124Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:31:13.307177]
2022-05-18T04:31:14.2794667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpghg3u799
2022-05-18T04:31:14.2795289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpghg3u799/_remote_module_non_scriptable.py
2022-05-18T04:31:15.9405215Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:15.9448809Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelSingleProcessTest testMethod=test_cpu>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]>
2022-05-18T04:31:15.9450859Z test_cpu (__main__.DistributedDataParallelSingleProcessTest)
2022-05-18T04:31:15.9451769Z test_cuda (__main__.DistributedDataParallelSingleProcessTest)
2022-05-18T04:31:15.9452611Z test_rnn (__main__.DistributedDataParallelSingleProcessTest)
2022-05-18T04:31:15.9453287Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:31:15.9453846Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:31:15.9456059Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsGloo testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]>
2022-05-18T04:31:15.9457505Z test_all_gather (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:31:15.9457975Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:31:15.9458634Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:31:15.9459068Z test_allreduce (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:31:15.9459480Z test_broadcast (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:31:15.9459880Z test_gather (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:31:15.9460253Z test_reduce (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:31:15.9460641Z test_scatter (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:31:16.8839433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk91l6k1y
2022-05-18T04:31:16.8840072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk91l6k1y/_remote_module_non_scriptable.py
2022-05-18T04:31:18.5318692Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:18.5383113Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:31:18.5395549Z 
2022-05-18T04:31:18.5395971Z Running tests...
2022-05-18T04:31:18.5396470Z ----------------------------------------------------------------------
2022-05-18T04:31:18.5503732Z   test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:18.5653802Z ok (0.026s)
2022-05-18T04:31:18.5653999Z 
2022-05-18T04:31:18.5654323Z ----------------------------------------------------------------------
2022-05-18T04:31:18.5654687Z Ran 1 test in 0.026s
2022-05-18T04:31:18.5654857Z 
2022-05-18T04:31:18.5654955Z OK
2022-05-18T04:31:18.5655103Z 
2022-05-18T04:31:18.5655244Z Generating XML reports...
2022-05-18T04:31:18.5688103Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220518043118.xml
2022-05-18T04:31:19.7650561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyfe3u0c1
2022-05-18T04:31:19.7651158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyfe3u0c1/_remote_module_non_scriptable.py
2022-05-18T04:31:21.4269120Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:21.4335910Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:31:21.4350634Z 
2022-05-18T04:31:21.4351017Z Running tests...
2022-05-18T04:31:21.4351526Z ----------------------------------------------------------------------
2022-05-18T04:31:21.6312773Z   test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:21.6538515Z ok (0.219s)
2022-05-18T04:31:21.6538713Z 
2022-05-18T04:31:21.6539685Z ----------------------------------------------------------------------
2022-05-18T04:31:21.6540053Z Ran 1 test in 0.219s
2022-05-18T04:31:21.6540227Z 
2022-05-18T04:31:21.6540331Z OK
2022-05-18T04:31:21.6540736Z 
2022-05-18T04:31:21.6540909Z Generating XML reports...
2022-05-18T04:31:21.6579357Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220518043121.xml
2022-05-18T04:31:22.9030001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp623bv5sg
2022-05-18T04:31:22.9031169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp623bv5sg/_remote_module_non_scriptable.py
2022-05-18T04:31:24.5517807Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:24.5583622Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:31:24.5597017Z 
2022-05-18T04:31:24.5597399Z Running tests...
2022-05-18T04:31:24.5597906Z ----------------------------------------------------------------------
2022-05-18T04:31:25.4236313Z   test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:25.9103263Z ok (1.350s)
2022-05-18T04:31:25.9103609Z 
2022-05-18T04:31:25.9104315Z ----------------------------------------------------------------------
2022-05-18T04:31:25.9104931Z Ran 1 test in 1.351s
2022-05-18T04:31:25.9105236Z 
2022-05-18T04:31:25.9105380Z OK
2022-05-18T04:31:25.9105668Z 
2022-05-18T04:31:25.9105917Z Generating XML reports...
2022-05-18T04:31:25.9145425Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220518043124.xml
2022-05-18T04:31:27.2044592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_kwb7j7
2022-05-18T04:31:28.8647774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_kwb7j7/_remote_module_non_scriptable.py
2022-05-18T04:31:28.8648265Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:28.8711723Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:31:28.8724137Z 
2022-05-18T04:31:28.8724584Z Running tests...
2022-05-18T04:31:28.8725081Z ----------------------------------------------------------------------
2022-05-18T04:31:28.9130234Z   test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53767
2022-05-18T04:31:28.9244955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53768
2022-05-18T04:31:29.8614998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgduwasfc
2022-05-18T04:31:29.8615625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1ut1uhi
2022-05-18T04:31:29.8616192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgduwasfc/_remote_module_non_scriptable.py
2022-05-18T04:31:29.8616738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1ut1uhi/_remote_module_non_scriptable.py
2022-05-18T04:31:31.5571972Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:31.5613930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:31.5659302Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:31.5704023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:31.5826444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:31.5827318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:31.5828162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:31.5828882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:33.1358857Z ok (4.263s)
2022-05-18T04:31:33.1359108Z 
2022-05-18T04:31:33.1359553Z ----------------------------------------------------------------------
2022-05-18T04:31:33.1360255Z Ran 1 test in 4.263s
2022-05-18T04:31:33.1360451Z 
2022-05-18T04:31:33.1360559Z OK
2022-05-18T04:31:33.1360710Z 
2022-05-18T04:31:33.1360856Z Generating XML reports...
2022-05-18T04:31:33.1401412Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043128.xml
2022-05-18T04:31:34.3526580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzwrnecza
2022-05-18T04:31:34.3527771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzwrnecza/_remote_module_non_scriptable.py
2022-05-18T04:31:35.9718252Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:35.9779759Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:31:35.9791966Z 
2022-05-18T04:31:35.9792408Z Running tests...
2022-05-18T04:31:35.9793183Z ----------------------------------------------------------------------
2022-05-18T04:31:36.0185838Z   test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53983
2022-05-18T04:31:36.0304147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53984
2022-05-18T04:31:36.9572562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptc10khls
2022-05-18T04:31:36.9573192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptc10khls/_remote_module_non_scriptable.py
2022-05-18T04:31:36.9631878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28cg0dlg
2022-05-18T04:31:36.9632467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28cg0dlg/_remote_module_non_scriptable.py
2022-05-18T04:31:38.6352256Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:38.6393062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:38.6494555Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:38.6539569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:38.6708247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:38.6708825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:38.6709728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:38.6710454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:40.1416793Z ok (4.162s)
2022-05-18T04:31:40.1417532Z 
2022-05-18T04:31:40.1418128Z ----------------------------------------------------------------------
2022-05-18T04:31:40.1418480Z Ran 1 test in 4.162s
2022-05-18T04:31:40.1418651Z 
2022-05-18T04:31:40.1418761Z OK
2022-05-18T04:31:40.1418879Z 
2022-05-18T04:31:40.1419025Z Generating XML reports...
2022-05-18T04:31:40.1460715Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043135.xml
2022-05-18T04:31:41.3809823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1k5ssr0
2022-05-18T04:31:41.3810434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1k5ssr0/_remote_module_non_scriptable.py
2022-05-18T04:31:43.0561946Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:43.0625234Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:31:43.0637160Z 
2022-05-18T04:31:43.0637465Z Running tests...
2022-05-18T04:31:43.0637941Z ----------------------------------------------------------------------
2022-05-18T04:31:43.1054957Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54199
2022-05-18T04:31:43.1160168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54200
2022-05-18T04:31:44.0599969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpls7eqttt
2022-05-18T04:31:44.0600642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpls7eqttt/_remote_module_non_scriptable.py
2022-05-18T04:31:44.0645447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbbwms97
2022-05-18T04:31:44.0646061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbbwms97/_remote_module_non_scriptable.py
2022-05-18T04:31:45.7311553Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:45.7338926Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:45.7353391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:45.7379512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:45.7592014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:45.7592560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:45.7593427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:45.7594133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:47.2271007Z ok (4.163s)
2022-05-18T04:31:47.2271234Z 
2022-05-18T04:31:47.2271727Z ----------------------------------------------------------------------
2022-05-18T04:31:47.2272056Z Ran 1 test in 4.163s
2022-05-18T04:31:47.2272301Z 
2022-05-18T04:31:47.2272400Z OK
2022-05-18T04:31:47.2272540Z 
2022-05-18T04:31:47.2273198Z Generating XML reports...
2022-05-18T04:31:47.2316540Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043143.xml
2022-05-18T04:31:48.4929983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2pxeonot
2022-05-18T04:31:48.4930614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2pxeonot/_remote_module_non_scriptable.py
2022-05-18T04:31:50.1748618Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:50.1813058Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:31:50.1825690Z 
2022-05-18T04:31:50.1825967Z Running tests...
2022-05-18T04:31:50.1826432Z ----------------------------------------------------------------------
2022-05-18T04:31:50.2253408Z   test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54415
2022-05-18T04:31:50.2358754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54416
2022-05-18T04:31:51.2277791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2doxqj60
2022-05-18T04:31:51.2278446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2doxqj60/_remote_module_non_scriptable.py
2022-05-18T04:31:51.2606243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdx3xipb
2022-05-18T04:31:51.2606871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdx3xipb/_remote_module_non_scriptable.py
2022-05-18T04:31:52.8838839Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:52.8880490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:52.9328104Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:52.9373287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:52.9486937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:52.9488112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:52.9490375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:52.9491121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:31:54.4475858Z ok (4.265s)
2022-05-18T04:31:54.4476099Z 
2022-05-18T04:31:54.4476558Z ----------------------------------------------------------------------
2022-05-18T04:31:54.4476883Z Ran 1 test in 4.265s
2022-05-18T04:31:54.4477052Z 
2022-05-18T04:31:54.4477146Z OK
2022-05-18T04:31:54.4477283Z 
2022-05-18T04:31:54.4477422Z Generating XML reports...
2022-05-18T04:31:54.4521963Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043150.xml
2022-05-18T04:31:55.6975579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xn_hj5c
2022-05-18T04:31:55.6976278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xn_hj5c/_remote_module_non_scriptable.py
2022-05-18T04:31:57.3325908Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:31:57.3388488Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:31:57.3399870Z 
2022-05-18T04:31:57.3400321Z Running tests...
2022-05-18T04:31:57.3400811Z ----------------------------------------------------------------------
2022-05-18T04:31:57.3819866Z   test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54631
2022-05-18T04:31:57.3923994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54632
2022-05-18T04:31:58.3162301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeba6clz6
2022-05-18T04:31:58.3163737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeba6clz6/_remote_module_non_scriptable.py
2022-05-18T04:31:58.3164606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsf4y_a7
2022-05-18T04:31:58.3165152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsf4y_a7/_remote_module_non_scriptable.py
2022-05-18T04:32:00.0029203Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:00.0071445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:00.0234397Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:00.0277782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:00.0388021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:00.0388565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:00.0389406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:00.0390106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:01.6040028Z ok (4.264s)
2022-05-18T04:32:01.6040510Z 
2022-05-18T04:32:01.6041368Z ----------------------------------------------------------------------
2022-05-18T04:32:01.6042107Z Ran 1 test in 4.264s
2022-05-18T04:32:01.6042280Z 
2022-05-18T04:32:01.6042799Z OK
2022-05-18T04:32:01.6042966Z 
2022-05-18T04:32:01.6043102Z Generating XML reports...
2022-05-18T04:32:01.6088565Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043157.xml
2022-05-18T04:32:02.9091875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0yi3s6r
2022-05-18T04:32:02.9092529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0yi3s6r/_remote_module_non_scriptable.py
2022-05-18T04:32:04.5640736Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:04.5703469Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:32:04.5714208Z 
2022-05-18T04:32:04.5714730Z Running tests...
2022-05-18T04:32:04.5715606Z ----------------------------------------------------------------------
2022-05-18T04:32:04.6135841Z   test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54847
2022-05-18T04:32:04.6256698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54848
2022-05-18T04:32:05.5629730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplwitehx6
2022-05-18T04:32:05.5630359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplwitehx6/_remote_module_non_scriptable.py
2022-05-18T04:32:05.5660392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppc5relcr
2022-05-18T04:32:05.5660991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppc5relcr/_remote_module_non_scriptable.py
2022-05-18T04:32:07.2560654Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:07.2601068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:07.2760229Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:07.2804817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:07.3016557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:07.3017187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:07.3018031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:07.3018736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:08.8369275Z ok (4.265s)
2022-05-18T04:32:08.8369567Z 
2022-05-18T04:32:08.8369981Z ----------------------------------------------------------------------
2022-05-18T04:32:08.8370329Z Ran 1 test in 4.266s
2022-05-18T04:32:08.8370495Z 
2022-05-18T04:32:08.8370593Z OK
2022-05-18T04:32:08.8370712Z 
2022-05-18T04:32:08.8370847Z Generating XML reports...
2022-05-18T04:32:08.8414752Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043204.xml
2022-05-18T04:32:10.1222895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5sx4zme
2022-05-18T04:32:10.1223576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5sx4zme/_remote_module_non_scriptable.py
2022-05-18T04:32:11.8006182Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:11.8071502Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:32:11.8082923Z 
2022-05-18T04:32:11.8083247Z Running tests...
2022-05-18T04:32:11.8083729Z ----------------------------------------------------------------------
2022-05-18T04:32:11.8496388Z   test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55063
2022-05-18T04:32:11.8614265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55064
2022-05-18T04:32:12.8234038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp322few9e
2022-05-18T04:32:12.8235185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp322few9e/_remote_module_non_scriptable.py
2022-05-18T04:32:12.8554812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsr0k97fc
2022-05-18T04:32:12.8555416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsr0k97fc/_remote_module_non_scriptable.py
2022-05-18T04:32:14.5059782Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:14.5099166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:14.5240980Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:14.5287190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:14.5500731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:14.5501293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:14.5502579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:14.5503321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:15.9721043Z ok (4.164s)
2022-05-18T04:32:15.9721232Z 
2022-05-18T04:32:15.9721662Z ----------------------------------------------------------------------
2022-05-18T04:32:15.9722021Z Ran 1 test in 4.164s
2022-05-18T04:32:15.9722169Z 
2022-05-18T04:32:15.9722276Z OK
2022-05-18T04:32:15.9722415Z 
2022-05-18T04:32:15.9722560Z Generating XML reports...
2022-05-18T04:32:15.9764601Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043211.xml
2022-05-18T04:32:17.2371926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_i4j00j
2022-05-18T04:32:17.2372637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_i4j00j/_remote_module_non_scriptable.py
2022-05-18T04:32:18.9048458Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:18.9110171Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:32:18.9122702Z 
2022-05-18T04:32:18.9123014Z Running tests...
2022-05-18T04:32:18.9123481Z ----------------------------------------------------------------------
2022-05-18T04:32:18.9560867Z   test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55279
2022-05-18T04:32:18.9680392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55280
2022-05-18T04:32:19.8912021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbt7cq4ok
2022-05-18T04:32:19.8912690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbt7cq4ok/_remote_module_non_scriptable.py
2022-05-18T04:32:19.9245223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1iqgqypo
2022-05-18T04:32:19.9245821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1iqgqypo/_remote_module_non_scriptable.py
2022-05-18T04:32:21.5662296Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:21.5702862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:21.5923214Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:21.5967865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:21.6117028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:21.6117633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:21.6118479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:21.6119463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:23.0794481Z ok (4.167s)
2022-05-18T04:32:23.0794866Z 
2022-05-18T04:32:23.0795572Z ----------------------------------------------------------------------
2022-05-18T04:32:23.0796153Z Ran 1 test in 4.167s
2022-05-18T04:32:23.0796479Z 
2022-05-18T04:32:23.0796645Z OK
2022-05-18T04:32:23.0796898Z 
2022-05-18T04:32:23.0798208Z Generating XML reports...
2022-05-18T04:32:23.0841893Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043218.xml
2022-05-18T04:32:25.0384756Z 
2022-05-18T04:32:25.0385165Z real	1m21.128s
2022-05-18T04:32:25.0385473Z user	1m44.947s
2022-05-18T04:32:25.0385723Z sys	2m24.590s
2022-05-18T04:32:25.0386295Z + python test/run_test.py --verbose -i distributed/test_c10d_spawn_nccl
2022-05-18T04:32:34.7564322Z Ignoring disabled issues:  []
2022-05-18T04:32:34.7695336Z /var/lib/jenkins/workspace/test/run_test.py:894: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-05-18T04:32:34.7695945Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) == "11.6":
2022-05-18T04:32:34.7696279Z Selected tests:
2022-05-18T04:32:34.7696552Z  distributed/test_c10d_spawn_nccl
2022-05-18T04:32:34.7805971Z Prioritized test from test file changes.
2022-05-18T04:32:34.7806382Z reordering tests for PR:
2022-05-18T04:32:34.7806643Z prioritized: []
2022-05-18T04:32:34.7807164Z the rest: ['distributed/test_c10d_spawn_nccl']
2022-05-18T04:32:34.7807372Z 
2022-05-18T04:32:34.7815348Z Running distributed/test_c10d_spawn_nccl ... [2022-05-18 04:32:34.781076]
2022-05-18T04:32:34.7816079Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:32:34.781146]
2022-05-18T04:32:35.7385544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpttnnrkgf
2022-05-18T04:32:35.7386275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpttnnrkgf/_remote_module_non_scriptable.py
2022-05-18T04:32:37.3335134Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:37.3373817Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:32:37.3374886Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:32:37.3376803Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>]>
2022-05-18T04:32:37.3377985Z test_all_gather (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:32:37.3378427Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:32:37.3378826Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:32:37.3379256Z test_allreduce (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:32:37.3379672Z test_broadcast (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:32:37.3380086Z test_reduce (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:32:37.3380489Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:32:38.2994324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_i_cyfr
2022-05-18T04:32:38.2994979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_i_cyfr/_remote_module_non_scriptable.py
2022-05-18T04:32:39.9699184Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:39.9764262Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:32:39.9775652Z 
2022-05-18T04:32:39.9775956Z Running tests...
2022-05-18T04:32:39.9776671Z ----------------------------------------------------------------------
2022-05-18T04:32:40.0192432Z   test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55631
2022-05-18T04:32:40.0311175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55632
2022-05-18T04:32:40.9621492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqotpyp2
2022-05-18T04:32:40.9622495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp18bym62x
2022-05-18T04:32:40.9623328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqotpyp2/_remote_module_non_scriptable.py
2022-05-18T04:32:40.9623931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp18bym62x/_remote_module_non_scriptable.py
2022-05-18T04:32:42.6438872Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:42.6480567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:42.6484390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:42.6542902Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:42.6584622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:42.6586412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:42.6587255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:42.6587981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:44.1423597Z ok (4.164s)
2022-05-18T04:32:44.1423819Z 
2022-05-18T04:32:44.1424280Z ----------------------------------------------------------------------
2022-05-18T04:32:44.1424615Z Ran 1 test in 4.165s
2022-05-18T04:32:44.1424799Z 
2022-05-18T04:32:44.1424893Z OK
2022-05-18T04:32:44.1425029Z 
2022-05-18T04:32:44.1425165Z Generating XML reports...
2022-05-18T04:32:44.1467350Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043239.xml
2022-05-18T04:32:45.4324266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjo4gi6m_
2022-05-18T04:32:45.4324870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjo4gi6m_/_remote_module_non_scriptable.py
2022-05-18T04:32:47.1046136Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:47.1107782Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:32:47.1119273Z 
2022-05-18T04:32:47.1119584Z Running tests...
2022-05-18T04:32:47.1120048Z ----------------------------------------------------------------------
2022-05-18T04:32:47.1545655Z   test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55850
2022-05-18T04:32:47.1656841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55851
2022-05-18T04:32:48.1426282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ogce9u8
2022-05-18T04:32:48.1426869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz6n4yx2z
2022-05-18T04:32:48.1427430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ogce9u8/_remote_module_non_scriptable.py
2022-05-18T04:32:48.1427988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz6n4yx2z/_remote_module_non_scriptable.py
2022-05-18T04:32:49.8435819Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:49.8476846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:49.8477407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:49.8562168Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:49.8605455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:49.8610612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:49.8611520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:49.8686184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:51.3769876Z ok (4.265s)
2022-05-18T04:32:51.3770143Z 
2022-05-18T04:32:51.3771023Z ----------------------------------------------------------------------
2022-05-18T04:32:51.3771394Z Ran 1 test in 4.265s
2022-05-18T04:32:51.3771558Z 
2022-05-18T04:32:51.3771654Z OK
2022-05-18T04:32:51.3771791Z 
2022-05-18T04:32:51.3771918Z Generating XML reports...
2022-05-18T04:32:51.3813713Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043247.xml
2022-05-18T04:32:52.6318929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb42nzgn8
2022-05-18T04:32:52.6319533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb42nzgn8/_remote_module_non_scriptable.py
2022-05-18T04:32:54.2565441Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:54.2624327Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:32:54.2637087Z 
2022-05-18T04:32:54.2637381Z Running tests...
2022-05-18T04:32:54.2637875Z ----------------------------------------------------------------------
2022-05-18T04:32:54.3057393Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56069
2022-05-18T04:32:54.3173873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56070
2022-05-18T04:32:55.2294874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2yj1pqlk
2022-05-18T04:32:55.2296032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2yj1pqlk/_remote_module_non_scriptable.py
2022-05-18T04:32:55.2359510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm84darbt
2022-05-18T04:32:55.2360214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm84darbt/_remote_module_non_scriptable.py
2022-05-18T04:32:56.8979987Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:56.9021015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:56.9022359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:56.9054705Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:32:56.9096690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:56.9099036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:56.9099890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:56.9126503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:32:58.4282227Z ok (4.164s)
2022-05-18T04:32:58.4282532Z 
2022-05-18T04:32:58.4282969Z ----------------------------------------------------------------------
2022-05-18T04:32:58.4283338Z Ran 1 test in 4.164s
2022-05-18T04:32:58.4283519Z 
2022-05-18T04:32:58.4283621Z OK
2022-05-18T04:32:58.4283791Z 
2022-05-18T04:32:58.4287969Z Generating XML reports...
2022-05-18T04:32:58.4333819Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043254.xml
2022-05-18T04:32:59.6914005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyiv0hvm7
2022-05-18T04:32:59.6914926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyiv0hvm7/_remote_module_non_scriptable.py
2022-05-18T04:33:01.3658982Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:01.3721991Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:33:01.3734013Z 
2022-05-18T04:33:01.3734274Z Running tests...
2022-05-18T04:33:01.3734753Z ----------------------------------------------------------------------
2022-05-18T04:33:01.4162203Z   test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56288
2022-05-18T04:33:01.4281947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56289
2022-05-18T04:33:02.3695714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsp9y1weh
2022-05-18T04:33:02.3696362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsp9y1weh/_remote_module_non_scriptable.py
2022-05-18T04:33:02.3966535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42axwxmq
2022-05-18T04:33:02.3967238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42axwxmq/_remote_module_non_scriptable.py
2022-05-18T04:33:04.0737635Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:04.0754853Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:04.0780100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:04.0783396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:04.0799513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:04.0803330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:04.0804216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:04.0887083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:05.6401358Z ok (4.266s)
2022-05-18T04:33:05.6401608Z 
2022-05-18T04:33:05.6402144Z ----------------------------------------------------------------------
2022-05-18T04:33:05.6402491Z Ran 1 test in 4.267s
2022-05-18T04:33:05.6402663Z 
2022-05-18T04:33:05.6402760Z OK
2022-05-18T04:33:05.6402897Z 
2022-05-18T04:33:05.6403045Z Generating XML reports...
2022-05-18T04:33:05.6447980Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043301.xml
2022-05-18T04:33:06.9390906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8nwvgq8
2022-05-18T04:33:06.9391684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8nwvgq8/_remote_module_non_scriptable.py
2022-05-18T04:33:08.5942934Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:08.6002960Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:33:08.6014838Z 
2022-05-18T04:33:08.6015182Z Running tests...
2022-05-18T04:33:08.6015986Z ----------------------------------------------------------------------
2022-05-18T04:33:08.6424496Z   test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56507
2022-05-18T04:33:08.6534929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56508
2022-05-18T04:33:09.5893536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl82648ap
2022-05-18T04:33:09.5894170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl82648ap/_remote_module_non_scriptable.py
2022-05-18T04:33:09.6190912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13k4t6e0
2022-05-18T04:33:09.6191478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13k4t6e0/_remote_module_non_scriptable.py
2022-05-18T04:33:11.2898644Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:11.2941262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:11.2944087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:11.2964905Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:11.3007263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:11.3008484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:11.3009375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:11.3049364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:12.7643133Z ok (4.163s)
2022-05-18T04:33:12.7643545Z 
2022-05-18T04:33:12.7644371Z ----------------------------------------------------------------------
2022-05-18T04:33:12.7644847Z Ran 1 test in 4.163s
2022-05-18T04:33:12.7644998Z 
2022-05-18T04:33:12.7645091Z OK
2022-05-18T04:33:12.7645223Z 
2022-05-18T04:33:12.7645359Z Generating XML reports...
2022-05-18T04:33:12.7689370Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043308.xml
2022-05-18T04:33:14.0225848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwy6pg7fe
2022-05-18T04:33:14.0226502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwy6pg7fe/_remote_module_non_scriptable.py
2022-05-18T04:33:15.6729142Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:15.6791977Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:33:15.6803078Z 
2022-05-18T04:33:15.6803546Z Running tests...
2022-05-18T04:33:15.6804043Z ----------------------------------------------------------------------
2022-05-18T04:33:15.7229177Z   test_reduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56726
2022-05-18T04:33:15.7344888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56727
2022-05-18T04:33:16.6893205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7arbktb
2022-05-18T04:33:16.6893807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7arbktb/_remote_module_non_scriptable.py
2022-05-18T04:33:16.6925554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xto7a55
2022-05-18T04:33:16.6926217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xto7a55/_remote_module_non_scriptable.py
2022-05-18T04:33:18.3590406Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:18.3630381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:18.3633356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:18.3673783Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:18.3715729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:18.3718277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:18.3719160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:18.3737350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:19.8463842Z ok (4.166s)
2022-05-18T04:33:19.8464272Z 
2022-05-18T04:33:19.8464696Z ----------------------------------------------------------------------
2022-05-18T04:33:19.8465062Z Ran 1 test in 4.166s
2022-05-18T04:33:19.8465662Z 
2022-05-18T04:33:19.8465757Z OK
2022-05-18T04:33:19.8465897Z 
2022-05-18T04:33:19.8466035Z Generating XML reports...
2022-05-18T04:33:19.8508567Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043315.xml
2022-05-18T04:33:21.1146236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiyr2vt4s
2022-05-18T04:33:21.1146861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiyr2vt4s/_remote_module_non_scriptable.py
2022-05-18T04:33:22.7711061Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:22.7772902Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:33:22.7783334Z 
2022-05-18T04:33:22.7783802Z Running tests...
2022-05-18T04:33:22.7784734Z ----------------------------------------------------------------------
2022-05-18T04:33:22.8210182Z   test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56945
2022-05-18T04:33:22.8328841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56946
2022-05-18T04:33:23.7682327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9g7r5i2f
2022-05-18T04:33:23.7682943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9g7r5i2f/_remote_module_non_scriptable.py
2022-05-18T04:33:23.7701128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgk4ojqcx
2022-05-18T04:33:23.7701735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgk4ojqcx/_remote_module_non_scriptable.py
2022-05-18T04:33:25.4644466Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:25.4686564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:25.4688825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:25.4793854Z INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:25.4836734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:25.4837276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:25.4838133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:25.4896661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:33:27.0442920Z ok (4.266s)
2022-05-18T04:33:27.0443173Z 
2022-05-18T04:33:27.0443611Z ----------------------------------------------------------------------
2022-05-18T04:33:27.0443938Z Ran 1 test in 4.266s
2022-05-18T04:33:27.0444105Z 
2022-05-18T04:33:27.0444200Z OK
2022-05-18T04:33:27.0445297Z 
2022-05-18T04:33:27.0445658Z Generating XML reports...
2022-05-18T04:33:27.0487670Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043322.xml
2022-05-18T04:33:29.0744394Z 
2022-05-18T04:33:29.0744926Z real	1m4.036s
2022-05-18T04:33:29.0745247Z user	1m28.512s
2022-05-18T04:33:29.0745515Z sys	1m51.983s
2022-05-18T04:33:29.0746091Z + python test/run_test.py --verbose -i distributed/test_store
2022-05-18T04:33:38.5617240Z Ignoring disabled issues:  []
2022-05-18T04:33:38.5749715Z /var/lib/jenkins/workspace/test/run_test.py:894: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-05-18T04:33:38.5750416Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) == "11.6":
2022-05-18T04:33:38.5750781Z Selected tests:
2022-05-18T04:33:38.5751050Z  distributed/test_store
2022-05-18T04:33:38.5855193Z Prioritized test from test file changes.
2022-05-18T04:33:38.5856072Z reordering tests for PR:
2022-05-18T04:33:38.5856405Z prioritized: []
2022-05-18T04:33:38.5857212Z the rest: ['distributed/test_store']
2022-05-18T04:33:38.5857413Z 
2022-05-18T04:33:38.5865892Z Running distributed/test_store ... [2022-05-18 04:33:38.586046]
2022-05-18T04:33:38.5866640Z Executing ['/opt/conda/bin/python', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:33:38.586119]
2022-05-18T04:33:39.5488164Z test_compare_set (__main__.FileStoreTest)
2022-05-18T04:33:39.5488785Z test_set_get (__main__.FileStoreTest)
2022-05-18T04:33:39.5489344Z test_compare_set (__main__.HashStoreTest)
2022-05-18T04:33:39.5492402Z test_set_get (__main__.HashStoreTest)
2022-05-18T04:33:39.5493607Z test_compare_set (__main__.PrefixFileStoreTest)
2022-05-18T04:33:39.5494338Z test_set_get (__main__.PrefixFileStoreTest)
2022-05-18T04:33:39.5494829Z test_compare_set (__main__.PrefixTCPStoreTest)
2022-05-18T04:33:39.5495392Z test_set_get (__main__.PrefixTCPStoreTest)
2022-05-18T04:33:39.5495723Z test_set_get (__main__.PythonStoreTest)
2022-05-18T04:33:39.5496041Z test_nominal (__main__.RendezvousEnvTest)
2022-05-18T04:33:39.5496380Z test_common_errors (__main__.RendezvousFileTest)
2022-05-18T04:33:39.5496722Z test_nominal (__main__.RendezvousFileTest)
2022-05-18T04:33:39.5497045Z test_common_errors (__main__.RendezvousTCPTest)
2022-05-18T04:33:39.5497398Z test_dns_timeout (__main__.RendezvousTCPTest)
2022-05-18T04:33:39.5497728Z test_nominal (__main__.RendezvousTCPTest)
2022-05-18T04:33:39.5498070Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest)
2022-05-18T04:33:39.5498401Z test_unknown_handler (__main__.RendezvousTest)
2022-05-18T04:33:39.5498740Z test_address_already_in_use (__main__.TCPStoreTest)
2022-05-18T04:33:39.5499074Z test_compare_set (__main__.TCPStoreTest)
2022-05-18T04:33:39.5499404Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest)
2022-05-18T04:33:39.5499861Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest)
2022-05-18T04:33:39.5500253Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest)
2022-05-18T04:33:39.5500597Z test_multitenancy (__main__.TCPStoreTest)
2022-05-18T04:33:39.5500927Z test_numkeys_delkeys (__main__.TCPStoreTest)
2022-05-18T04:33:39.5501249Z test_set_get (__main__.TCPStoreTest)
2022-05-18T04:33:40.4533601Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:33:40.4544324Z 
2022-05-18T04:33:40.4544699Z Running tests...
2022-05-18T04:33:40.4545327Z ----------------------------------------------------------------------
2022-05-18T04:33:42.1176746Z   test_compare_set (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:42.1381038Z ok (1.684s)
2022-05-18T04:33:42.1381398Z 
2022-05-18T04:33:42.1382755Z ----------------------------------------------------------------------
2022-05-18T04:33:42.1383153Z Ran 1 test in 1.684s
2022-05-18T04:33:42.1383328Z 
2022-05-18T04:33:42.1383406Z OK
2022-05-18T04:33:42.1383554Z 
2022-05-18T04:33:42.1383688Z Generating XML reports...
2022-05-18T04:33:42.1418157Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518043340.xml
2022-05-18T04:33:43.3915501Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:33:43.3925066Z 
2022-05-18T04:33:43.3925317Z Running tests...
2022-05-18T04:33:43.3926913Z ----------------------------------------------------------------------
2022-05-18T04:33:45.0628694Z   test_set_get (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:45.0822075Z ok (1.689s)
2022-05-18T04:33:45.0822290Z 
2022-05-18T04:33:45.0823245Z ----------------------------------------------------------------------
2022-05-18T04:33:45.0823616Z Ran 1 test in 1.690s
2022-05-18T04:33:45.0823787Z 
2022-05-18T04:33:45.0823892Z OK
2022-05-18T04:33:45.0824033Z 
2022-05-18T04:33:45.0824151Z Generating XML reports...
2022-05-18T04:33:45.0857820Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518043343.xml
2022-05-18T04:33:46.2769169Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:33:46.2780852Z 
2022-05-18T04:33:46.2781302Z Running tests...
2022-05-18T04:33:46.2781783Z ----------------------------------------------------------------------
2022-05-18T04:33:47.9340322Z   test_compare_set (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:47.9531564Z ok (1.675s)
2022-05-18T04:33:47.9531869Z 
2022-05-18T04:33:47.9532760Z ----------------------------------------------------------------------
2022-05-18T04:33:47.9533387Z Ran 1 test in 1.675s
2022-05-18T04:33:47.9533562Z 
2022-05-18T04:33:47.9533673Z OK
2022-05-18T04:33:47.9533816Z 
2022-05-18T04:33:47.9534319Z Generating XML reports...
2022-05-18T04:33:47.9567657Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518043346.xml
2022-05-18T04:33:49.1460159Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:33:49.1469901Z 
2022-05-18T04:33:49.1470195Z Running tests...
2022-05-18T04:33:49.1470696Z ----------------------------------------------------------------------
2022-05-18T04:33:50.8056022Z   test_set_get (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:50.8239820Z ok (1.677s)
2022-05-18T04:33:50.8240260Z 
2022-05-18T04:33:50.8240749Z ----------------------------------------------------------------------
2022-05-18T04:33:50.8241088Z Ran 1 test in 1.677s
2022-05-18T04:33:50.8241264Z 
2022-05-18T04:33:50.8241372Z OK
2022-05-18T04:33:50.8241532Z 
2022-05-18T04:33:50.8241678Z Generating XML reports...
2022-05-18T04:33:50.8276286Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518043349.xml
2022-05-18T04:33:52.0070347Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:33:52.0080306Z 
2022-05-18T04:33:52.0080748Z Running tests...
2022-05-18T04:33:52.0081232Z ----------------------------------------------------------------------
2022-05-18T04:33:53.6445107Z   test_compare_set (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:53.6649327Z ok (1.657s)
2022-05-18T04:33:53.6649534Z 
2022-05-18T04:33:53.6649979Z ----------------------------------------------------------------------
2022-05-18T04:33:53.6650310Z Ran 1 test in 1.657s
2022-05-18T04:33:53.6650543Z 
2022-05-18T04:33:53.6650654Z OK
2022-05-18T04:33:53.6650798Z 
2022-05-18T04:33:53.6650937Z Generating XML reports...
2022-05-18T04:33:53.6686970Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518043352.xml
2022-05-18T04:33:54.8441205Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:33:54.8449978Z 
2022-05-18T04:33:54.8450318Z Running tests...
2022-05-18T04:33:54.8450793Z ----------------------------------------------------------------------
2022-05-18T04:33:56.4927339Z   test_set_get (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:56.5119811Z ok (1.667s)
2022-05-18T04:33:56.5120005Z 
2022-05-18T04:33:56.5120437Z ----------------------------------------------------------------------
2022-05-18T04:33:56.5120790Z Ran 1 test in 1.667s
2022-05-18T04:33:56.5120938Z 
2022-05-18T04:33:56.5121046Z OK
2022-05-18T04:33:56.5121196Z 
2022-05-18T04:33:56.5121330Z Generating XML reports...
2022-05-18T04:33:56.5156078Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518043354.xml
2022-05-18T04:33:57.6960045Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:33:57.6970143Z 
2022-05-18T04:33:57.6970384Z Running tests...
2022-05-18T04:33:57.6970885Z ----------------------------------------------------------------------
2022-05-18T04:33:59.3541786Z   test_compare_set (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:33:59.3761431Z ok (1.679s)
2022-05-18T04:33:59.3763252Z 
2022-05-18T04:33:59.3763906Z ----------------------------------------------------------------------
2022-05-18T04:33:59.3764283Z Ran 1 test in 1.679s
2022-05-18T04:33:59.3764454Z 
2022-05-18T04:33:59.3764530Z OK
2022-05-18T04:33:59.3764700Z 
2022-05-18T04:33:59.3764940Z Generating XML reports...
2022-05-18T04:33:59.3799533Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518043357.xml
2022-05-18T04:34:00.5501559Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:00.5512373Z 
2022-05-18T04:34:00.5512565Z Running tests...
2022-05-18T04:34:00.5513615Z ----------------------------------------------------------------------
2022-05-18T04:34:02.2057091Z   test_set_get (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:02.2257139Z ok (1.674s)
2022-05-18T04:34:02.2259044Z 
2022-05-18T04:34:02.2259525Z ----------------------------------------------------------------------
2022-05-18T04:34:02.2259918Z Ran 1 test in 1.675s
2022-05-18T04:34:02.2260077Z 
2022-05-18T04:34:02.2260249Z OK
2022-05-18T04:34:02.2260506Z 
2022-05-18T04:34:02.2260648Z Generating XML reports...
2022-05-18T04:34:02.2293926Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518043400.xml
2022-05-18T04:34:03.4524349Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:03.4534246Z 
2022-05-18T04:34:03.4534572Z Running tests...
2022-05-18T04:34:03.4535063Z ----------------------------------------------------------------------
2022-05-18T04:34:05.1219438Z   test_set_get (__main__.PythonStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:05.1393613Z ok (1.686s)
2022-05-18T04:34:05.1393810Z 
2022-05-18T04:34:05.1394321Z ----------------------------------------------------------------------
2022-05-18T04:34:05.1394702Z Ran 1 test in 1.686s
2022-05-18T04:34:05.1394881Z 
2022-05-18T04:34:05.1394996Z OK
2022-05-18T04:34:05.1395134Z 
2022-05-18T04:34:05.1395268Z Generating XML reports...
2022-05-18T04:34:05.1429797Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220518043403.xml
2022-05-18T04:34:06.3318866Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:06.3330200Z 
2022-05-18T04:34:06.3330395Z Running tests...
2022-05-18T04:34:06.3331181Z ----------------------------------------------------------------------
2022-05-18T04:34:07.9883773Z   test_nominal (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:08.0075743Z ok (1.674s)
2022-05-18T04:34:08.0076177Z 
2022-05-18T04:34:08.0076777Z ----------------------------------------------------------------------
2022-05-18T04:34:08.0077157Z Ran 1 test in 1.675s
2022-05-18T04:34:08.0077328Z 
2022-05-18T04:34:08.0077440Z OK
2022-05-18T04:34:08.0077557Z 
2022-05-18T04:34:08.0077699Z Generating XML reports...
2022-05-18T04:34:08.0115319Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220518043406.xml
2022-05-18T04:34:09.2279144Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:09.2288750Z 
2022-05-18T04:34:09.2289068Z Running tests...
2022-05-18T04:34:09.2289566Z ----------------------------------------------------------------------
2022-05-18T04:34:10.8784322Z   test_common_errors (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:10.8953627Z ok (1.666s)
2022-05-18T04:34:10.8953818Z 
2022-05-18T04:34:10.8954489Z ----------------------------------------------------------------------
2022-05-18T04:34:10.8954841Z Ran 1 test in 1.667s
2022-05-18T04:34:10.8955027Z 
2022-05-18T04:34:10.8955127Z OK
2022-05-18T04:34:10.8955247Z 
2022-05-18T04:34:10.8955394Z Generating XML reports...
2022-05-18T04:34:10.8989157Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518043409.xml
2022-05-18T04:34:12.0738738Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:12.0749840Z 
2022-05-18T04:34:12.0750209Z Running tests...
2022-05-18T04:34:12.0750695Z ----------------------------------------------------------------------
2022-05-18T04:34:13.7524762Z   test_nominal (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:13.7716038Z ok (1.697s)
2022-05-18T04:34:13.7716229Z 
2022-05-18T04:34:13.7717099Z ----------------------------------------------------------------------
2022-05-18T04:34:13.7717518Z Ran 1 test in 1.697s
2022-05-18T04:34:13.7717689Z 
2022-05-18T04:34:13.7718022Z OK
2022-05-18T04:34:13.7718185Z 
2022-05-18T04:34:13.7718328Z Generating XML reports...
2022-05-18T04:34:13.7751859Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518043412.xml
2022-05-18T04:34:14.9499633Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:14.9509618Z 
2022-05-18T04:34:14.9509806Z Running tests...
2022-05-18T04:34:14.9510301Z ----------------------------------------------------------------------
2022-05-18T04:34:16.5633824Z   test_common_errors (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:16.5803286Z ok (1.629s)
2022-05-18T04:34:16.5803489Z 
2022-05-18T04:34:16.5804455Z ----------------------------------------------------------------------
2022-05-18T04:34:16.5804841Z Ran 1 test in 1.629s
2022-05-18T04:34:16.5805014Z 
2022-05-18T04:34:16.5805120Z OK
2022-05-18T04:34:16.5805267Z 
2022-05-18T04:34:16.5805402Z Generating XML reports...
2022-05-18T04:34:16.5838790Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043414.xml
2022-05-18T04:34:17.7501698Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:17.7512519Z 
2022-05-18T04:34:17.7512880Z Running tests...
2022-05-18T04:34:17.7513496Z ----------------------------------------------------------------------
2022-05-18T04:34:19.4444656Z   test_dns_timeout (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:19.4700646Z [W socket.cpp:558] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known).
2022-05-18T04:34:19.4701237Z [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456).
2022-05-18T04:34:19.4703980Z ok (1.719s)
2022-05-18T04:34:19.4704873Z 
2022-05-18T04:34:19.4705283Z ----------------------------------------------------------------------
2022-05-18T04:34:19.4705629Z Ran 1 test in 1.719s
2022-05-18T04:34:19.4705798Z 
2022-05-18T04:34:19.4705897Z OK
2022-05-18T04:34:19.4706039Z 
2022-05-18T04:34:19.4706171Z Generating XML reports...
2022-05-18T04:34:19.4740928Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043417.xml
2022-05-18T04:34:20.6906119Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:20.6915963Z 
2022-05-18T04:34:20.6916561Z Running tests...
2022-05-18T04:34:20.6917051Z ----------------------------------------------------------------------
2022-05-18T04:34:22.3693410Z   test_nominal (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:22.3889379Z ok (1.697s)
2022-05-18T04:34:22.3889577Z 
2022-05-18T04:34:22.3890149Z ----------------------------------------------------------------------
2022-05-18T04:34:22.3890505Z Ran 1 test in 1.697s
2022-05-18T04:34:22.3890655Z 
2022-05-18T04:34:22.3890776Z OK
2022-05-18T04:34:22.3890917Z 
2022-05-18T04:34:22.3891141Z Generating XML reports...
2022-05-18T04:34:22.3924230Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043420.xml
2022-05-18T04:34:23.5609373Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:23.5621857Z 
2022-05-18T04:34:23.5622252Z Running tests...
2022-05-18T04:34:23.5623017Z ----------------------------------------------------------------------
2022-05-18T04:34:25.2491228Z   test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:35.3753176Z ok (11.813s)
2022-05-18T04:34:35.3753443Z 
2022-05-18T04:34:35.3753887Z ----------------------------------------------------------------------
2022-05-18T04:34:35.3754242Z Ran 1 test in 11.813s
2022-05-18T04:34:35.3754414Z 
2022-05-18T04:34:35.3754514Z OK
2022-05-18T04:34:35.3754632Z 
2022-05-18T04:34:35.3754776Z Generating XML reports...
2022-05-18T04:34:35.3796825Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043423.xml
2022-05-18T04:34:36.6155419Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:36.6165529Z 
2022-05-18T04:34:36.6165827Z Running tests...
2022-05-18T04:34:36.6166320Z ----------------------------------------------------------------------
2022-05-18T04:34:38.2707039Z   test_unknown_handler (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:38.2881503Z ok (1.671s)
2022-05-18T04:34:38.2881705Z 
2022-05-18T04:34:38.2882416Z ----------------------------------------------------------------------
2022-05-18T04:34:38.2882823Z Ran 1 test in 1.672s
2022-05-18T04:34:38.2882998Z 
2022-05-18T04:34:38.2883097Z OK
2022-05-18T04:34:38.2883241Z 
2022-05-18T04:34:38.2883387Z Generating XML reports...
2022-05-18T04:34:38.2916742Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220518043436.xml
2022-05-18T04:34:39.4697425Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:39.4708208Z 
2022-05-18T04:34:39.4708399Z Running tests...
2022-05-18T04:34:39.4709106Z ----------------------------------------------------------------------
2022-05-18T04:34:41.1167297Z   test_address_already_in_use (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:41.1352352Z [W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:37531 (errno: 98 - Address already in use).
2022-05-18T04:34:41.1373545Z [W socket.cpp:401] [c10d] The server socket has failed to bind to 0.0.0.0:37531 (errno: 98 - Address already in use).
2022-05-18T04:34:41.1374069Z [E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address.
2022-05-18T04:34:41.1377676Z ok (1.667s)
2022-05-18T04:34:41.1377962Z 
2022-05-18T04:34:41.1378468Z ----------------------------------------------------------------------
2022-05-18T04:34:41.1378933Z Ran 1 test in 1.667s
2022-05-18T04:34:41.1379104Z 
2022-05-18T04:34:41.1379180Z OK
2022-05-18T04:34:41.1379320Z 
2022-05-18T04:34:41.1379456Z Generating XML reports...
2022-05-18T04:34:41.1414478Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043439.xml
2022-05-18T04:34:42.2989279Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:42.2999465Z 
2022-05-18T04:34:42.2999656Z Running tests...
2022-05-18T04:34:42.3000150Z ----------------------------------------------------------------------
2022-05-18T04:34:43.9469055Z   test_compare_set (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:43.9683655Z ok (1.668s)
2022-05-18T04:34:43.9683853Z 
2022-05-18T04:34:43.9684368Z ----------------------------------------------------------------------
2022-05-18T04:34:43.9684754Z Ran 1 test in 1.669s
2022-05-18T04:34:43.9684930Z 
2022-05-18T04:34:43.9685054Z OK
2022-05-18T04:34:43.9685203Z 
2022-05-18T04:34:43.9689751Z Generating XML reports...
2022-05-18T04:34:43.9719392Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043442.xml
2022-05-18T04:34:45.1299964Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:45.1312443Z 
2022-05-18T04:34:45.1312681Z Running tests...
2022-05-18T04:34:45.1313166Z ----------------------------------------------------------------------
2022-05-18T04:34:46.7887115Z   test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:46.8073636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:46.8074824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T04:34:46.9221455Z ok (1.791s)
2022-05-18T04:34:46.9221795Z 
2022-05-18T04:34:46.9223528Z ----------------------------------------------------------------------
2022-05-18T04:34:46.9224229Z Ran 1 test in 1.791s
2022-05-18T04:34:46.9224556Z 
2022-05-18T04:34:46.9224706Z OK
2022-05-18T04:34:46.9224974Z 
2022-05-18T04:34:46.9225219Z Generating XML reports...
2022-05-18T04:34:46.9260353Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043445.xml
2022-05-18T04:34:48.1378332Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:48.1387531Z 
2022-05-18T04:34:48.1388287Z Running tests...
2022-05-18T04:34:48.1388784Z ----------------------------------------------------------------------
2022-05-18T04:34:49.7868977Z   test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:49.8108612Z ok (1.672s)
2022-05-18T04:34:49.8108771Z 
2022-05-18T04:34:49.8109555Z ----------------------------------------------------------------------
2022-05-18T04:34:49.8109928Z Ran 1 test in 1.672s
2022-05-18T04:34:49.8110104Z 
2022-05-18T04:34:49.8110202Z OK
2022-05-18T04:34:49.8110340Z 
2022-05-18T04:34:49.8110480Z Generating XML reports...
2022-05-18T04:34:49.8143217Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043448.xml
2022-05-18T04:34:51.0261180Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:51.0270744Z 
2022-05-18T04:34:51.0270993Z Running tests...
2022-05-18T04:34:51.0271466Z ----------------------------------------------------------------------
2022-05-18T04:34:52.6422890Z   test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:52.6612833Z ok (1.634s)
2022-05-18T04:34:52.6612999Z 
2022-05-18T04:34:52.6613905Z ----------------------------------------------------------------------
2022-05-18T04:34:52.6614281Z Ran 1 test in 1.634s
2022-05-18T04:34:52.6614449Z 
2022-05-18T04:34:52.6614570Z OK
2022-05-18T04:34:52.6614708Z 
2022-05-18T04:34:52.6614826Z Generating XML reports...
2022-05-18T04:34:52.6648541Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043451.xml
2022-05-18T04:34:53.8609624Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:53.8619797Z 
2022-05-18T04:34:53.8620075Z Running tests...
2022-05-18T04:34:53.8620544Z ----------------------------------------------------------------------
2022-05-18T04:34:55.5067736Z   test_multitenancy (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:34:55.5253620Z ok (1.663s)
2022-05-18T04:34:55.5253818Z 
2022-05-18T04:34:55.5254479Z ----------------------------------------------------------------------
2022-05-18T04:34:55.5254854Z Ran 1 test in 1.664s
2022-05-18T04:34:55.5255027Z 
2022-05-18T04:34:55.5255128Z OK
2022-05-18T04:34:55.5255274Z 
2022-05-18T04:34:55.5255410Z Generating XML reports...
2022-05-18T04:34:55.5289823Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043453.xml
2022-05-18T04:34:56.7247311Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:34:56.7257390Z 
2022-05-18T04:34:56.7257733Z Running tests...
2022-05-18T04:34:56.7258692Z ----------------------------------------------------------------------
2022-05-18T04:34:58.3533333Z   test_numkeys_delkeys (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:00.3999328Z ok (3.674s)
2022-05-18T04:35:00.3999519Z 
2022-05-18T04:35:00.3999943Z ----------------------------------------------------------------------
2022-05-18T04:35:00.4000288Z Ran 1 test in 3.674s
2022-05-18T04:35:00.4000458Z 
2022-05-18T04:35:00.4000561Z OK
2022-05-18T04:35:00.4000708Z 
2022-05-18T04:35:00.4000852Z Generating XML reports...
2022-05-18T04:35:00.4035993Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043456.xml
2022-05-18T04:35:01.6020339Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:35:01.6030221Z 
2022-05-18T04:35:01.6030668Z Running tests...
2022-05-18T04:35:01.6031152Z ----------------------------------------------------------------------
2022-05-18T04:35:03.2764156Z   test_set_get (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:03.2964941Z ok (1.693s)
2022-05-18T04:35:03.2965152Z 
2022-05-18T04:35:03.2965601Z ----------------------------------------------------------------------
2022-05-18T04:35:03.2965950Z Ran 1 test in 1.694s
2022-05-18T04:35:03.2966123Z 
2022-05-18T04:35:03.2966228Z OK
2022-05-18T04:35:03.2966371Z 
2022-05-18T04:35:03.2966485Z Generating XML reports...
2022-05-18T04:35:03.2998863Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043501.xml
2022-05-18T04:35:05.1305336Z 
2022-05-18T04:35:05.1305916Z real	1m36.056s
2022-05-18T04:35:05.1306279Z user	1m39.533s
2022-05-18T04:35:05.1306519Z sys	2m37.191s
2022-05-18T04:35:05.1307084Z + python test/run_test.py --verbose -i distributed/test_pg_wrapper
2022-05-18T04:35:14.7605176Z Ignoring disabled issues:  []
2022-05-18T04:35:14.7736876Z /var/lib/jenkins/workspace/test/run_test.py:894: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-05-18T04:35:14.7737644Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) == "11.6":
2022-05-18T04:35:14.7738009Z Selected tests:
2022-05-18T04:35:14.7738269Z  distributed/test_pg_wrapper
2022-05-18T04:35:14.7839220Z Prioritized test from test file changes.
2022-05-18T04:35:14.7839650Z reordering tests for PR:
2022-05-18T04:35:14.7839934Z prioritized: []
2022-05-18T04:35:14.7840426Z the rest: ['distributed/test_pg_wrapper']
2022-05-18T04:35:14.7840628Z 
2022-05-18T04:35:14.7849612Z Running distributed/test_pg_wrapper ... [2022-05-18 04:35:14.784518]
2022-05-18T04:35:14.7850394Z Executing ['/opt/conda/bin/python', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:35:14.784593]
2022-05-18T04:35:15.7217480Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:35:15.7218126Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:35:15.7220716Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-05-18T04:35:15.7222782Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:35:15.7223432Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:35:15.7223894Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:35:15.7224806Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:35:15.7225301Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:35:15.7225746Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:35:15.7226204Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:35:15.7226697Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:35:15.7227298Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:35:15.7228347Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-05-18T04:35:15.7229348Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T04:35:15.7229790Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T04:35:15.7230266Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T04:35:15.7230707Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T04:35:15.7231168Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T04:35:16.6591156Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:35:16.6602091Z 
2022-05-18T04:35:16.6602276Z Running tests...
2022-05-18T04:35:16.6602760Z ----------------------------------------------------------------------
2022-05-18T04:35:18.2898025Z   test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:18.3335953Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59164
2022-05-18T04:35:18.3441327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59165
2022-05-18T04:35:18.3558772Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59166
2022-05-18T04:35:18.3667901Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59167
2022-05-18T04:35:19.2770308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:35:19.2800044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:19.3345634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:19.3530092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:19.3690016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:19.3791777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:19.3895005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:35:19.3895806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:19.3896664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:19.3897370Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:19.3898107Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:19.3898807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:19.4768940Z [E ProcessGroupGloo.cpp:2791] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T04:35:19.4769438Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T04:35:19.4870811Z [E ProcessGroupGloo.cpp:136] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-05-18T04:35:19.4971927Z [E ProcessGroupGloo.cpp:136] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-05-18T04:35:19.7723415Z ok (3.112s)
2022-05-18T04:35:19.7723894Z 
2022-05-18T04:35:19.7724643Z ----------------------------------------------------------------------
2022-05-18T04:35:19.7724984Z Ran 1 test in 3.112s
2022-05-18T04:35:19.7725152Z 
2022-05-18T04:35:19.7725253Z OK
2022-05-18T04:35:19.7725395Z 
2022-05-18T04:35:19.7725563Z Generating XML reports...
2022-05-18T04:35:19.7769208Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043516.xml
2022-05-18T04:35:21.0290600Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:35:21.0300146Z 
2022-05-18T04:35:21.0300386Z Running tests...
2022-05-18T04:35:21.0300853Z ----------------------------------------------------------------------
2022-05-18T04:35:22.6795576Z   test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:22.7233667Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59531
2022-05-18T04:35:22.7347590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59532
2022-05-18T04:35:22.7467482Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59533
2022-05-18T04:35:22.7583126Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59534
2022-05-18T04:35:23.7273314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:23.7691815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:23.7692853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:23.7801248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:35:23.8112575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:23.8213659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:23.8317421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:35:23.8317958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:23.8318845Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:23.8319567Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:23.8418890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:23.8419630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:24.1634209Z ok (3.133s)
2022-05-18T04:35:24.1634446Z 
2022-05-18T04:35:24.1634890Z ----------------------------------------------------------------------
2022-05-18T04:35:24.1635242Z Ran 1 test in 3.133s
2022-05-18T04:35:24.1635414Z 
2022-05-18T04:35:24.1635513Z OK
2022-05-18T04:35:24.1635655Z 
2022-05-18T04:35:24.1635798Z Generating XML reports...
2022-05-18T04:35:24.1678849Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043521.xml
2022-05-18T04:35:25.3972418Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:35:25.3982848Z 
2022-05-18T04:35:25.3983136Z Running tests...
2022-05-18T04:35:25.3983626Z ----------------------------------------------------------------------
2022-05-18T04:35:27.0505268Z   test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:27.0963146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59898
2022-05-18T04:35:27.1083791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59899
2022-05-18T04:35:27.1211177Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59900
2022-05-18T04:35:27.1335922Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59901
2022-05-18T04:35:28.1194366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:28.1220184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:28.1339932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:28.1681924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:35:28.1855673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:28.1856216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:28.1961810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:35:28.1962392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:28.1963259Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:28.1963991Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:28.2060734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:28.2062370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:30.1427599Z ok (4.744s)
2022-05-18T04:35:30.1427815Z 
2022-05-18T04:35:30.1428287Z ----------------------------------------------------------------------
2022-05-18T04:35:30.1428650Z Ran 1 test in 4.744s
2022-05-18T04:35:30.1428831Z 
2022-05-18T04:35:30.1428930Z OK
2022-05-18T04:35:30.1429072Z 
2022-05-18T04:35:30.1429188Z Generating XML reports...
2022-05-18T04:35:30.1474231Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043525.xml
2022-05-18T04:35:31.4050525Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:35:31.4061191Z 
2022-05-18T04:35:31.4061470Z Running tests...
2022-05-18T04:35:31.4062464Z ----------------------------------------------------------------------
2022-05-18T04:35:33.0573101Z   test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:33.1013748Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60269
2022-05-18T04:35:33.1135697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60270
2022-05-18T04:35:33.1260127Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60271
2022-05-18T04:35:33.1371177Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60272
2022-05-18T04:35:34.0658469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:34.0886543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:34.1152319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:34.1170494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:35:34.1614339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:34.1716998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:34.1817970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:35:34.1818713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:34.1819551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:34.1820246Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:34.1922298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:34.1923041Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:34.2544804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:35:34.2647978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:35:34.2748765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-05-18T04:35:34.2749318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:35:34.2750068Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:35:34.2750773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:35:34.2751453Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:35:34.2851617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:35:36.2465693Z ok (4.840s)
2022-05-18T04:35:36.2466124Z 
2022-05-18T04:35:36.2466823Z ----------------------------------------------------------------------
2022-05-18T04:35:36.2467416Z Ran 1 test in 4.840s
2022-05-18T04:35:36.2467709Z 
2022-05-18T04:35:36.2467886Z OK
2022-05-18T04:35:36.2468121Z 
2022-05-18T04:35:36.2468351Z Generating XML reports...
2022-05-18T04:35:36.2514617Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043531.xml
2022-05-18T04:35:37.5050134Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:35:37.5060068Z 
2022-05-18T04:35:37.5060365Z Running tests...
2022-05-18T04:35:37.5061090Z ----------------------------------------------------------------------
2022-05-18T04:35:39.1192386Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:39.1631865Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60652
2022-05-18T04:35:39.1750871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60653
2022-05-18T04:35:39.1876814Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60654
2022-05-18T04:35:39.1983236Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60655
2022-05-18T04:35:40.1080326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:40.1464581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:40.1572764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:35:40.1636533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:40.2194283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:40.2296449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:40.2297138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:40.2297657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:35:40.2298508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:40.2299236Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:40.2399922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:40.2400687Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:40.3125160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:35:40.3223584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:35:40.3327022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-05-18T04:35:40.3327561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:35:40.3328310Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:35:40.3329002Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:35:40.3329749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:35:40.3429536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:35:40.7036857Z ok (3.197s)
2022-05-18T04:35:40.7037159Z 
2022-05-18T04:35:40.7037838Z ----------------------------------------------------------------------
2022-05-18T04:35:40.7038204Z Ran 1 test in 3.198s
2022-05-18T04:35:40.7038353Z 
2022-05-18T04:35:40.7038451Z OK
2022-05-18T04:35:40.7038594Z 
2022-05-18T04:35:40.7038739Z Generating XML reports...
2022-05-18T04:35:40.7083880Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043537.xml
2022-05-18T04:35:41.9631757Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:35:41.9641947Z 
2022-05-18T04:35:41.9642347Z Running tests...
2022-05-18T04:35:41.9642851Z ----------------------------------------------------------------------
2022-05-18T04:35:43.6256183Z   test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:43.6701788Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61031
2022-05-18T04:35:43.6823234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61032
2022-05-18T04:35:43.6948306Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61033
2022-05-18T04:35:43.7060036Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61034
2022-05-18T04:35:44.6797945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:44.7259989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:44.7414993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:44.7528678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:35:44.7641012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:44.7742365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:44.7843465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:35:44.7844008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:44.7844870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:44.7845590Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:44.7846271Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:44.7846978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:45.2115542Z ok (3.247s)
2022-05-18T04:35:45.2115765Z 
2022-05-18T04:35:45.2116214Z ----------------------------------------------------------------------
2022-05-18T04:35:45.2116582Z Ran 1 test in 3.247s
2022-05-18T04:35:45.2116753Z 
2022-05-18T04:35:45.2116829Z OK
2022-05-18T04:35:45.2116974Z 
2022-05-18T04:35:45.2117120Z Generating XML reports...
2022-05-18T04:35:45.2161472Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043541.xml
2022-05-18T04:35:46.4509111Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:35:46.4519484Z 
2022-05-18T04:35:46.4519813Z Running tests...
2022-05-18T04:35:46.4520468Z ----------------------------------------------------------------------
2022-05-18T04:35:48.0640865Z   test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:48.1078827Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61398
2022-05-18T04:35:48.1189614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61399
2022-05-18T04:35:48.1311628Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61400
2022-05-18T04:35:48.1420414Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61401
2022-05-18T04:35:49.0908925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:35:49.1219510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:49.1255658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:49.1260657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:49.1533141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:49.1636662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:49.1739268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:35:49.1740156Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:49.1740696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:49.1741658Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:49.1839601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:49.1840348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:51.2515509Z ok (4.799s)
2022-05-18T04:35:51.2515735Z 
2022-05-18T04:35:51.2516550Z ----------------------------------------------------------------------
2022-05-18T04:35:51.2517015Z Ran 1 test in 4.800s
2022-05-18T04:35:51.2517186Z 
2022-05-18T04:35:51.2517261Z OK
2022-05-18T04:35:51.2517405Z 
2022-05-18T04:35:51.2517548Z Generating XML reports...
2022-05-18T04:35:51.2560816Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043546.xml
2022-05-18T04:35:52.4853037Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:35:52.4863379Z 
2022-05-18T04:35:52.4863695Z Running tests...
2022-05-18T04:35:52.4864182Z ----------------------------------------------------------------------
2022-05-18T04:35:54.1488610Z   test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:35:54.1945749Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61769
2022-05-18T04:35:54.2066261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61770
2022-05-18T04:35:54.2175880Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61771
2022-05-18T04:35:54.2298572Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61772
2022-05-18T04:35:55.1449411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:35:55.1520003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:55.1884373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:55.1935782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:55.2582557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:55.2583259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:55.2684653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:35:55.2685531Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:55.2686097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:55.2686758Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:55.2786703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:55.2787451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:35:55.3511394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:35:55.3612544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:35:55.3613104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:35:55.3613627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-05-18T04:35:55.3614844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:35:55.3615545Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:35:55.3716389Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:35:55.3717140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:35:57.4398525Z ok (4.953s)
2022-05-18T04:35:57.4398960Z 
2022-05-18T04:35:57.4399896Z ----------------------------------------------------------------------
2022-05-18T04:35:57.4400269Z Ran 1 test in 4.954s
2022-05-18T04:35:57.4400440Z 
2022-05-18T04:35:57.4400516Z OK
2022-05-18T04:35:57.4400654Z 
2022-05-18T04:35:57.4400796Z Generating XML reports...
2022-05-18T04:35:57.4444092Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043552.xml
2022-05-18T04:35:58.6996683Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:35:58.7005887Z 
2022-05-18T04:35:58.7006057Z Running tests...
2022-05-18T04:35:58.7006566Z ----------------------------------------------------------------------
2022-05-18T04:36:00.3586518Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:00.4032077Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62152
2022-05-18T04:36:00.4154167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62153
2022-05-18T04:36:00.4276929Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62154
2022-05-18T04:36:00.4402660Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62155
2022-05-18T04:36:01.4637554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:01.4771700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:01.4786909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:01.5132914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:36:01.5817298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:01.5817848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:01.5818360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:36:01.5918638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:01.5920198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:01.5921566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:01.5922851Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:01.5923550Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:01.6744530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:36:01.6846056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:36:01.6846612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:36:01.6848591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-05-18T04:36:01.6849590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:36:01.6850292Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:36:01.6950625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:36:01.6951370Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:36:02.0462921Z ok (3.345s)
2022-05-18T04:36:02.0463577Z 
2022-05-18T04:36:02.0464030Z ----------------------------------------------------------------------
2022-05-18T04:36:02.0464356Z Ran 1 test in 3.345s
2022-05-18T04:36:02.0464524Z 
2022-05-18T04:36:02.0464620Z OK
2022-05-18T04:36:02.0465434Z 
2022-05-18T04:36:02.0465594Z Generating XML reports...
2022-05-18T04:36:02.0508725Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043558.xml
2022-05-18T04:36:03.3282106Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:03.3290688Z 
2022-05-18T04:36:03.3290878Z Running tests...
2022-05-18T04:36:03.3291870Z ----------------------------------------------------------------------
2022-05-18T04:36:04.9268619Z   test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:04.9699838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62531
2022-05-18T04:36:04.9817282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62532
2022-05-18T04:36:05.9382319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:05.9383099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:05.9800243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:05.9802781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:05.9803817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:05.9897286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:06.0212744Z [E ProcessGroupGloo.cpp:2791] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T04:36:06.0213259Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T04:36:06.1860603Z ok (2.857s)
2022-05-18T04:36:06.1860850Z 
2022-05-18T04:36:06.1861291Z ----------------------------------------------------------------------
2022-05-18T04:36:06.1861621Z Ran 1 test in 2.857s
2022-05-18T04:36:06.1861818Z 
2022-05-18T04:36:06.1862199Z OK
2022-05-18T04:36:06.1862349Z 
2022-05-18T04:36:06.1862488Z Generating XML reports...
2022-05-18T04:36:06.1904534Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043603.xml
2022-05-18T04:36:07.4313687Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:07.4323947Z 
2022-05-18T04:36:07.4324286Z Running tests...
2022-05-18T04:36:07.4324771Z ----------------------------------------------------------------------
2022-05-18T04:36:09.1022588Z   test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:09.1479865Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62740
2022-05-18T04:36:09.1602672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62741
2022-05-18T04:36:10.1051628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:10.1054671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:10.1523941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:10.1526202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:10.1527148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:10.1568512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:11.9687490Z ok (4.536s)
2022-05-18T04:36:11.9688017Z 
2022-05-18T04:36:11.9688838Z ----------------------------------------------------------------------
2022-05-18T04:36:11.9689270Z Ran 1 test in 4.536s
2022-05-18T04:36:11.9689436Z 
2022-05-18T04:36:11.9689554Z OK
2022-05-18T04:36:11.9689690Z 
2022-05-18T04:36:11.9689835Z Generating XML reports...
2022-05-18T04:36:11.9733533Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043607.xml
2022-05-18T04:36:13.1971907Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:13.1981317Z 
2022-05-18T04:36:13.1981640Z Running tests...
2022-05-18T04:36:13.1982585Z ----------------------------------------------------------------------
2022-05-18T04:36:14.8341045Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:14.8788873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62965
2022-05-18T04:36:14.8909640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62966
2022-05-18T04:36:15.8015637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:15.8371776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:15.8480703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:15.8481230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:15.8482095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:15.8482822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:15.8692354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:36:15.8692882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:36:15.8693623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:36:15.8694356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:36:17.5991462Z ok (4.401s)
2022-05-18T04:36:17.5991690Z 
2022-05-18T04:36:17.5992148Z ----------------------------------------------------------------------
2022-05-18T04:36:17.5992497Z Ran 1 test in 4.401s
2022-05-18T04:36:17.5992670Z 
2022-05-18T04:36:17.5992752Z OK
2022-05-18T04:36:17.5992891Z 
2022-05-18T04:36:17.5993037Z Generating XML reports...
2022-05-18T04:36:17.6038003Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043613.xml
2022-05-18T04:36:18.8517176Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:18.8527129Z 
2022-05-18T04:36:18.8527511Z Running tests...
2022-05-18T04:36:18.8528127Z ----------------------------------------------------------------------
2022-05-18T04:36:20.4862310Z   test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:20.5308665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63196
2022-05-18T04:36:20.5425517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63197
2022-05-18T04:36:21.4595450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:21.4599443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:21.4709741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:21.4711104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:21.4711954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:21.4805246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:23.2508740Z ok (4.398s)
2022-05-18T04:36:23.2508997Z 
2022-05-18T04:36:23.2509408Z ----------------------------------------------------------------------
2022-05-18T04:36:23.2509754Z Ran 1 test in 4.398s
2022-05-18T04:36:23.2509919Z 
2022-05-18T04:36:23.2510012Z OK
2022-05-18T04:36:23.2512572Z 
2022-05-18T04:36:23.2512802Z Generating XML reports...
2022-05-18T04:36:23.2555889Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043618.xml
2022-05-18T04:36:24.5116326Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:24.5125792Z 
2022-05-18T04:36:24.5125982Z Running tests...
2022-05-18T04:36:24.5126996Z ----------------------------------------------------------------------
2022-05-18T04:36:26.1675800Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:26.2103260Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63421
2022-05-18T04:36:26.2223043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63422
2022-05-18T04:36:27.1628124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:27.2166050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:27.2376986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:27.2377518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:27.2378388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:27.2379102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:36:27.2487831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:36:27.2488390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:36:27.2489102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:36:27.2489810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-05-18T04:36:29.1305958Z ok (4.618s)
2022-05-18T04:36:29.1306170Z 
2022-05-18T04:36:29.1306618Z ----------------------------------------------------------------------
2022-05-18T04:36:29.1306986Z Ran 1 test in 4.618s
2022-05-18T04:36:29.1307158Z 
2022-05-18T04:36:29.1307263Z OK
2022-05-18T04:36:29.1307406Z 
2022-05-18T04:36:29.1307523Z Generating XML reports...
2022-05-18T04:36:29.1351288Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043624.xml
2022-05-18T04:36:31.0185173Z 
2022-05-18T04:36:31.0185932Z real	1m25.888s
2022-05-18T04:36:31.0186252Z user	2m34.428s
2022-05-18T04:36:31.0186497Z sys	3m49.217s
2022-05-18T04:36:31.0187121Z + python test/run_test.py --verbose -i distributed/rpc/cuda/test_tensorpipe_agent
2022-05-18T04:36:40.7777025Z Ignoring disabled issues:  []
2022-05-18T04:36:40.7911166Z /var/lib/jenkins/workspace/test/run_test.py:894: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-05-18T04:36:40.7912079Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) == "11.6":
2022-05-18T04:36:40.7912454Z Selected tests:
2022-05-18T04:36:40.7912755Z  distributed/rpc/cuda/test_tensorpipe_agent
2022-05-18T04:36:40.8020840Z Prioritized test from test file changes.
2022-05-18T04:36:40.8021183Z reordering tests for PR:
2022-05-18T04:36:40.8021477Z prioritized: []
2022-05-18T04:36:40.8022266Z the rest: ['distributed/rpc/cuda/test_tensorpipe_agent']
2022-05-18T04:36:40.8022508Z 
2022-05-18T04:36:40.8031282Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2022-05-18 04:36:40.802575]
2022-05-18T04:36:40.8032066Z Executing ['/opt/conda/bin/python', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:36:40.802648]
2022-05-18T04:36:41.7792034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0h2kwjln
2022-05-18T04:36:41.7792688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0h2kwjln/_remote_module_non_scriptable.py
2022-05-18T04:36:42.2113977Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote_gpu>]>
2022-05-18T04:36:42.2114632Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest)
2022-05-18T04:36:42.2115492Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_simple>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]>
2022-05-18T04:36:42.2116377Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest)
2022-05-18T04:36:42.2117132Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest)
2022-05-18T04:36:42.2117986Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest)
2022-05-18T04:36:42.2119750Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]>
2022-05-18T04:36:42.2121360Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-05-18T04:36:42.2122188Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest)
2022-05-18T04:36:42.2122841Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest)
2022-05-18T04:36:42.2123277Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-05-18T04:36:42.2123870Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRpcTest testMethod=test_profiler_remote_cuda>]>
2022-05-18T04:36:42.2124334Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest)
2022-05-18T04:36:42.2125666Z <unittest.suite.TestSuite tests=[<__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]>
2022-05-18T04:36:42.2127389Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-05-18T04:36:42.2127865Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-05-18T04:36:42.2128307Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-05-18T04:36:42.2128724Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-05-18T04:36:42.2129267Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-05-18T04:36:42.2129714Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-05-18T04:36:42.2130135Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-05-18T04:36:42.2130546Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-05-18T04:36:42.2146087Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_nested_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_meta_multiple_tensors>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]>
2022-05-18T04:36:42.2160626Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2161177Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2161689Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2162240Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2162783Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2163358Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2163927Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2164506Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2165071Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2165577Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2226218Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2226744Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2227226Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2227728Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2228226Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2228711Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2229186Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2229643Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2230108Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2230572Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2231044Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2231529Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2232043Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2232533Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2233028Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2233705Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2234158Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2234622Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2235084Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2235530Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2236073Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2236546Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2237005Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2237458Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2237945Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2238426Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2238888Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2239364Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2239841Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2240322Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2240831Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2241310Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2241813Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2242319Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2242801Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2243280Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2243745Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2244225Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2244731Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2245236Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2245722Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2246192Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2246689Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2247203Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2247698Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2248219Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2248733Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2249250Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2249734Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2250211Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2250687Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2251211Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2251682Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2252169Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2252656Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2253293Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2253769Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2254316Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2254799Z test_meta_multiple_tensors (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2255297Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2255818Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2256331Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2256828Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2257331Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2257825Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2258301Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2258800Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2259286Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2259779Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2260260Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2260756Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2261412Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2262457Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2262980Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2263475Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2263976Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2264465Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2264965Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T04:36:42.2265881Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_device_maps_backward_pass>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]>
2022-05-18T04:36:42.2266938Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-05-18T04:36:42.2267426Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-05-18T04:36:42.2268108Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-05-18T04:36:43.1277110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5qod892_
2022-05-18T04:36:43.1277715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5qod892_/_remote_module_non_scriptable.py
2022-05-18T04:36:43.5423047Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:36:43.5434293Z 
2022-05-18T04:36:43.5435340Z Running tests...
2022-05-18T04:36:43.5435976Z ----------------------------------------------------------------------
2022-05-18T04:36:45.1789028Z   test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:45.2221403Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63787
2022-05-18T04:36:45.2340952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63788
2022-05-18T04:36:45.2468652Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63789
2022-05-18T04:36:45.2577747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63790
2022-05-18T04:36:46.1570114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12r42zii
2022-05-18T04:36:46.1570741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12r42zii/_remote_module_non_scriptable.py
2022-05-18T04:36:46.1678980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskxapx7r
2022-05-18T04:36:46.1679576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskxapx7r/_remote_module_non_scriptable.py
2022-05-18T04:36:46.1722556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_kt0ioc
2022-05-18T04:36:46.1723156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_kt0ioc/_remote_module_non_scriptable.py
2022-05-18T04:36:46.1883944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76pbof8j
2022-05-18T04:36:46.1885316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76pbof8j/_remote_module_non_scriptable.py
2022-05-18T04:36:46.5624222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:46.5738265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:36:46.5812770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:46.6029826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:46.9801695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:46.9903096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:46.9903693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:46.9904207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:36:46.9905059Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:46.9905772Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:47.0006905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:47.0007675Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:48.6576665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:36:48.6577242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:36:48.6577974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:36:48.6578921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:36:48.6607641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:36:48.6608183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:36:48.6608658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:36:48.6609592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:36:49.1691407Z ok (5.625s)
2022-05-18T04:36:49.1691613Z 
2022-05-18T04:36:49.1692064Z ----------------------------------------------------------------------
2022-05-18T04:36:49.1692429Z Ran 1 test in 5.626s
2022-05-18T04:36:49.1692596Z 
2022-05-18T04:36:49.1692689Z OK
2022-05-18T04:36:49.1692825Z 
2022-05-18T04:36:49.1692941Z Generating XML reports...
2022-05-18T04:36:49.1737253Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20220518043643.xml
2022-05-18T04:36:50.4190587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwhw_9qcb
2022-05-18T04:36:50.4191212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwhw_9qcb/_remote_module_non_scriptable.py
2022-05-18T04:36:50.8408256Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:36:50.8417685Z 
2022-05-18T04:36:50.8417994Z Running tests...
2022-05-18T04:36:50.8418505Z ----------------------------------------------------------------------
2022-05-18T04:36:52.4618301Z   test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:52.5042704Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64450
2022-05-18T04:36:52.5162300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64451
2022-05-18T04:36:52.5264698Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64452
2022-05-18T04:36:52.5381838Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64453
2022-05-18T04:36:53.4647586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpkwdr47
2022-05-18T04:36:53.4648224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpkwdr47/_remote_module_non_scriptable.py
2022-05-18T04:36:53.5006608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpih6vnlpa
2022-05-18T04:36:53.5007232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpih6vnlpa/_remote_module_non_scriptable.py
2022-05-18T04:36:53.5195657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5b6oe47y
2022-05-18T04:36:53.5196277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5b6oe47y/_remote_module_non_scriptable.py
2022-05-18T04:36:53.5227189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvuqhkmfa
2022-05-18T04:36:53.5227816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvuqhkmfa/_remote_module_non_scriptable.py
2022-05-18T04:36:53.8803235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:53.9091552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:53.9206625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:53.9294764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:36:56.1487932Z ok (5.307s)
2022-05-18T04:36:56.1488219Z 
2022-05-18T04:36:56.1489838Z ----------------------------------------------------------------------
2022-05-18T04:36:56.1490293Z Ran 1 test in 5.307s
2022-05-18T04:36:56.1490474Z 
2022-05-18T04:36:56.1490586Z OK
2022-05-18T04:36:56.1490734Z 
2022-05-18T04:36:56.1490854Z Generating XML reports...
2022-05-18T04:36:56.1533786Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220518043650.xml
2022-05-18T04:36:57.3797100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp86ar8jwi
2022-05-18T04:36:57.3797715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp86ar8jwi/_remote_module_non_scriptable.py
2022-05-18T04:36:57.8084721Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:36:57.8095753Z 
2022-05-18T04:36:57.8096078Z Running tests...
2022-05-18T04:36:57.8096544Z ----------------------------------------------------------------------
2022-05-18T04:36:59.4791889Z   test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:36:59.5230742Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65101
2022-05-18T04:36:59.5343105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65102
2022-05-18T04:36:59.5461383Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65103
2022-05-18T04:36:59.5580219Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65104
2022-05-18T04:37:00.5461042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmaqliggz
2022-05-18T04:37:00.5466576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmaqliggz/_remote_module_non_scriptable.py
2022-05-18T04:37:00.5526926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9yfu6z5k
2022-05-18T04:37:00.5527531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9yfu6z5k/_remote_module_non_scriptable.py
2022-05-18T04:37:00.5574058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_ftv8c8
2022-05-18T04:37:00.5574628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9b4mm2_v
2022-05-18T04:37:00.5575177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_ftv8c8/_remote_module_non_scriptable.py
2022-05-18T04:37:00.5577180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9b4mm2_v/_remote_module_non_scriptable.py
2022-05-18T04:37:00.9498586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:00.9601174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:37:00.9767380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:37:00.9779674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:03.2688772Z ok (5.459s)
2022-05-18T04:37:03.2688975Z 
2022-05-18T04:37:03.2689440Z ----------------------------------------------------------------------
2022-05-18T04:37:03.2689790Z Ran 1 test in 5.459s
2022-05-18T04:37:03.2689958Z 
2022-05-18T04:37:03.2690056Z OK
2022-05-18T04:37:03.2690197Z 
2022-05-18T04:37:03.2690338Z Generating XML reports...
2022-05-18T04:37:03.2735582Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220518043657.xml
2022-05-18T04:37:04.5536479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyv0ozyu_
2022-05-18T04:37:04.5537218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyv0ozyu_/_remote_module_non_scriptable.py
2022-05-18T04:37:04.9715298Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:37:04.9725981Z 
2022-05-18T04:37:04.9726310Z Running tests...
2022-05-18T04:37:04.9726778Z ----------------------------------------------------------------------
2022-05-18T04:37:06.6175108Z   test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:06.6597845Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65752
2022-05-18T04:37:06.6713973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65753
2022-05-18T04:37:06.6831140Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65754
2022-05-18T04:37:06.6948940Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65755
2022-05-18T04:37:07.6938204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphvhara7y
2022-05-18T04:37:07.6938851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphvhara7y/_remote_module_non_scriptable.py
2022-05-18T04:37:07.7079819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu52hanfd
2022-05-18T04:37:07.7080431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu52hanfd/_remote_module_non_scriptable.py
2022-05-18T04:37:07.7328342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvtj38g8n
2022-05-18T04:37:07.7329185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7il5hhy3
2022-05-18T04:37:07.7329747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvtj38g8n/_remote_module_non_scriptable.py
2022-05-18T04:37:07.7330306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7il5hhy3/_remote_module_non_scriptable.py
2022-05-18T04:37:08.0970267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:08.1197092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:37:08.1344223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:37:08.1365256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:10.4056929Z ok (5.433s)
2022-05-18T04:37:10.4057178Z 
2022-05-18T04:37:10.4057595Z ----------------------------------------------------------------------
2022-05-18T04:37:10.4058772Z Ran 1 test in 5.433s
2022-05-18T04:37:10.4058975Z 
2022-05-18T04:37:10.4059085Z OK
2022-05-18T04:37:10.4059231Z 
2022-05-18T04:37:10.4059364Z Generating XML reports...
2022-05-18T04:37:10.4104903Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220518043704.xml
2022-05-18T04:37:11.6772994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyy2dmuws
2022-05-18T04:37:11.6773611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyy2dmuws/_remote_module_non_scriptable.py
2022-05-18T04:37:12.1038372Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:37:12.1049317Z 
2022-05-18T04:37:12.1049657Z Running tests...
2022-05-18T04:37:12.1050157Z ----------------------------------------------------------------------
2022-05-18T04:37:13.7500803Z   test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:13.7949575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66403
2022-05-18T04:37:13.8073057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66404
2022-05-18T04:37:14.7692407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8rifvew
2022-05-18T04:37:14.7693164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8rifvew/_remote_module_non_scriptable.py
2022-05-18T04:37:14.7693736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawjqhyiv
2022-05-18T04:37:14.7698974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawjqhyiv/_remote_module_non_scriptable.py
2022-05-18T04:37:15.1847735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:15.1858086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:16.9156778Z ok (4.810s)
2022-05-18T04:37:16.9157036Z 
2022-05-18T04:37:16.9157467Z ----------------------------------------------------------------------
2022-05-18T04:37:16.9157810Z Ran 1 test in 4.811s
2022-05-18T04:37:16.9157981Z 
2022-05-18T04:37:16.9158057Z OK
2022-05-18T04:37:16.9159019Z 
2022-05-18T04:37:16.9161267Z Generating XML reports...
2022-05-18T04:37:16.9203392Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518043712.xml
2022-05-18T04:37:18.1621826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnudlrmly
2022-05-18T04:37:18.1622916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnudlrmly/_remote_module_non_scriptable.py
2022-05-18T04:37:18.5705792Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:37:18.5716764Z 
2022-05-18T04:37:18.5717039Z Running tests...
2022-05-18T04:37:18.5717781Z ----------------------------------------------------------------------
2022-05-18T04:37:20.1716977Z   test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:20.2146128Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66687
2022-05-18T04:37:20.2264551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66688
2022-05-18T04:37:21.1051585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xe5b45b
2022-05-18T04:37:21.1052185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xe5b45b/_remote_module_non_scriptable.py
2022-05-18T04:37:21.1518350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyg6j9z3o
2022-05-18T04:37:21.1519333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyg6j9z3o/_remote_module_non_scriptable.py
2022-05-18T04:37:21.5165776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:21.5640806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:21.7854327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyg6j9z3o/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T04:37:21.7855160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xe5b45b/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T04:37:21.7937663Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmp9xe5b45b/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T04:37:23.5357434Z ok (4.964s)
2022-05-18T04:37:23.5357657Z 
2022-05-18T04:37:23.5358094Z ----------------------------------------------------------------------
2022-05-18T04:37:23.5358445Z Ran 1 test in 4.964s
2022-05-18T04:37:23.5358615Z 
2022-05-18T04:37:23.5358730Z OK
2022-05-18T04:37:23.5358870Z 
2022-05-18T04:37:23.5358991Z Generating XML reports...
2022-05-18T04:37:23.5401884Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518043718.xml
2022-05-18T04:37:24.7740920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmn6u4lae
2022-05-18T04:37:24.7741569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmn6u4lae/_remote_module_non_scriptable.py
2022-05-18T04:37:25.1874761Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:37:25.1884913Z 
2022-05-18T04:37:25.1885369Z Running tests...
2022-05-18T04:37:25.1885864Z ----------------------------------------------------------------------
2022-05-18T04:37:26.8056050Z   test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:26.8505633Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67003
2022-05-18T04:37:26.8621452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67004
2022-05-18T04:37:27.8450682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65ienpg4
2022-05-18T04:37:27.8451264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9luvvqv6
2022-05-18T04:37:27.8452126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65ienpg4/_remote_module_non_scriptable.py
2022-05-18T04:37:27.8453570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9luvvqv6/_remote_module_non_scriptable.py
2022-05-18T04:37:28.2601751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:28.2711911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:28.4967772Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:37:28.4986042Z RuntimeError('CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from exchangeDevice at /var/lib/jenkins/workspace/c10/cuda/impl/CUDAGuardImpl.h:33 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7da60661bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #1: <unknown function> + 0x146b4 (0x7f7da62b86b4 in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10_cuda.so)\nframe #2: <unknown function> + 0xd8821d (0x7f7da727f21d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #3: <unknown function> + 0x2c59814 (0x7f7da9150814 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #4: <unknown function> + 0x2c598fb (0x7f7da91508fb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #5: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x10f (0x7f7db20aa51f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x1a8f8b5 (0x7f7db230b8b5 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #7: at::_ops::empty_strided::call(c10::ArrayRef<long>, c10::ArrayRef<long>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x174 (0x7f7db20e9314 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x12da (0x7f7db1b046ea in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x1c29a63 (0x7f7db24a5a63 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x10d (0x7f7db1e6b63d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: <unknown function> + 0x1a920d1 (0x7f7db230e0d1 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x10d (0x7f7db1e6b63d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: <unknown function> + 0x2a525ce (0x7f7db32ce5ce in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x2a52b4b (0x7f7db32ceb4b in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x202 (0x7f7db1ee05d2 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f7db1afb8de in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x1d1aa99 (0x7f7db2596a99 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x216 (0x7f7db1ff5676 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: <unknown function> + 0x3224b0 (0x7f7dbc1304b0 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #20: <unknown function> + 0x322965 (0x7f7dbc130965 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #21: <unknown function> + 0x1bfb9c (0x55e4cbcaab9c in /opt/conda/bin/python)\nframe #22: <unknown function> + 0xff72f (0x55e4cbbea72f in /opt/conda/bin/python)\nframe #23: <unknown function> + 0x196663 (0x55e4cbc81663 in /opt/conda/bin/python)\nframe #24: _PyFunction_Vectorcall + 0x1d4 (0x55e4cbc82354 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0xfdae6 (0x55e4cbbe8ae6 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x197bf9 (0x55e4cbc82bf9 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0xff755 (0x55e4cbbea755 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x196663 (0x55e4cbc81663 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x197ca4 (0x55e4cbc82ca4 in /opt/conda/bin/python)\nframe #30: <unknown function> + 0xff755 (0x55e4cbbea755 in /opt/conda/bin/python)\nframe #31: _PyFunction_Vectorcall + 0x104 (0x55e4cbc82284 in /opt/conda/bin/python)\nframe #32: _PyObject_Call + 0x1da (0x55e4cbc30a7a in /opt/conda/bin/python)\nframe #33: _PyEval_EvalFrameDefault + 0x2610 (0x55e4cbcc29f0 in /opt/conda/bin/python)\nframe #34: _PyFunction_Vectorcall + 0x104 (0x55e4cbc82284 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x1da (0x55e4cbc30a7a in /opt/conda/bin/python)\nframe #36: <unknown function> + 0x94774a (0x7f7dbc75574a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #37: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7dbc753a3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #38: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7dbc756b25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #39: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f7dbc7571e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f7db4563b44 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7dbc756915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #42: <unknown function> + 0x3ce0e43 (0x7f7db455ce43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7db455da38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7db45580b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #45: <unknown function> + 0x3d10b42 (0x7f7db458cb42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7da60545eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #47: <unknown function> + 0xc9039 (0x7f7dbf796039 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #48: <unknown function> + 0x76db (0x7f7df4d9b6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #49: clone + 0x3f (0x7f7df4ac461f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-05-18T04:37:28.4996273Z Traceback (most recent call last):
2022-05-18T04:37:28.4996834Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:37:28.4997449Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:37:28.4998045Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/nn/api/remote_module.py", line 89, in _create_module
2022-05-18T04:37:28.4998440Z     module.to(device)
2022-05-18T04:37:28.4998902Z   File "/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 927, in to
2022-05-18T04:37:28.4999259Z     return self._apply(convert)
2022-05-18T04:37:28.4999747Z   File "/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 602, in _apply
2022-05-18T04:37:28.5000123Z     param_applied = fn(param)
2022-05-18T04:37:28.5000582Z   File "/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 925, in convert
2022-05-18T04:37:28.5001051Z     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2022-05-18T04:37:28.5001461Z RuntimeError: CUDA error: invalid device ordinal
2022-05-18T04:37:28.5001902Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-05-18T04:37:28.5002360Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-05-18T04:37:28.5002841Z Exception raised from exchangeDevice at /var/lib/jenkins/workspace/c10/cuda/impl/CUDAGuardImpl.h:33 (most recent call first):
2022-05-18T04:37:28.5003708Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7da60661bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:37:28.5004434Z frame #1: <unknown function> + 0x146b4 (0x7f7da62b86b4 in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10_cuda.so)
2022-05-18T04:37:28.5005081Z frame #2: <unknown function> + 0xd8821d (0x7f7da727f21d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:37:28.5005735Z frame #3: <unknown function> + 0x2c59814 (0x7f7da9150814 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:37:28.5006369Z frame #4: <unknown function> + 0x2c598fb (0x7f7da91508fb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:37:28.5007375Z frame #5: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x10f (0x7f7db20aa51f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5008210Z frame #6: <unknown function> + 0x1a8f8b5 (0x7f7db230b8b5 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5009156Z frame #7: at::_ops::empty_strided::call(c10::ArrayRef<long>, c10::ArrayRef<long>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x174 (0x7f7db20e9314 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5010359Z frame #8: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x12da (0x7f7db1b046ea in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5011178Z frame #9: <unknown function> + 0x1c29a63 (0x7f7db24a5a63 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5012336Z frame #10: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x10d (0x7f7db1e6b63d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5013203Z frame #11: <unknown function> + 0x1a920d1 (0x7f7db230e0d1 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5014210Z frame #12: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x10d (0x7f7db1e6b63d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5015070Z frame #13: <unknown function> + 0x2a525ce (0x7f7db32ce5ce in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5015726Z frame #14: <unknown function> + 0x2a52b4b (0x7f7db32ceb4b in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5016692Z frame #15: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x202 (0x7f7db1ee05d2 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5017826Z frame #16: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f7db1afb8de in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5018630Z frame #17: <unknown function> + 0x1d1aa99 (0x7f7db2596a99 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5019619Z frame #18: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x216 (0x7f7db1ff5676 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5020458Z frame #19: <unknown function> + 0x3224b0 (0x7f7dbc1304b0 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5021110Z frame #20: <unknown function> + 0x322965 (0x7f7dbc130965 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5021583Z frame #21: <unknown function> + 0x1bfb9c (0x55e4cbcaab9c in /opt/conda/bin/python)
2022-05-18T04:37:28.5022448Z frame #22: <unknown function> + 0xff72f (0x55e4cbbea72f in /opt/conda/bin/python)
2022-05-18T04:37:28.5022866Z frame #23: <unknown function> + 0x196663 (0x55e4cbc81663 in /opt/conda/bin/python)
2022-05-18T04:37:28.5023281Z frame #24: _PyFunction_Vectorcall + 0x1d4 (0x55e4cbc82354 in /opt/conda/bin/python)
2022-05-18T04:37:28.5023696Z frame #25: <unknown function> + 0xfdae6 (0x55e4cbbe8ae6 in /opt/conda/bin/python)
2022-05-18T04:37:28.5024109Z frame #26: <unknown function> + 0x197bf9 (0x55e4cbc82bf9 in /opt/conda/bin/python)
2022-05-18T04:37:28.5024515Z frame #27: <unknown function> + 0xff755 (0x55e4cbbea755 in /opt/conda/bin/python)
2022-05-18T04:37:28.5024911Z frame #28: <unknown function> + 0x196663 (0x55e4cbc81663 in /opt/conda/bin/python)
2022-05-18T04:37:28.5025396Z frame #29: <unknown function> + 0x197ca4 (0x55e4cbc82ca4 in /opt/conda/bin/python)
2022-05-18T04:37:28.5025799Z frame #30: <unknown function> + 0xff755 (0x55e4cbbea755 in /opt/conda/bin/python)
2022-05-18T04:37:28.5026208Z frame #31: _PyFunction_Vectorcall + 0x104 (0x55e4cbc82284 in /opt/conda/bin/python)
2022-05-18T04:37:28.5026599Z frame #32: _PyObject_Call + 0x1da (0x55e4cbc30a7a in /opt/conda/bin/python)
2022-05-18T04:37:28.5027015Z frame #33: _PyEval_EvalFrameDefault + 0x2610 (0x55e4cbcc29f0 in /opt/conda/bin/python)
2022-05-18T04:37:28.5027442Z frame #34: _PyFunction_Vectorcall + 0x104 (0x55e4cbc82284 in /opt/conda/bin/python)
2022-05-18T04:37:28.5027914Z frame #35: _PyObject_Call + 0x1da (0x55e4cbc30a7a in /opt/conda/bin/python)
2022-05-18T04:37:28.5028519Z frame #36: <unknown function> + 0x94774a (0x7f7dbc75574a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5029320Z frame #37: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7dbc753a3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5030350Z frame #38: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7dbc756b25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5031520Z frame #39: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f7dbc7571e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5032751Z frame #40: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f7db4563b44 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5034051Z frame #41: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7dbc756915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5034935Z frame #42: <unknown function> + 0x3ce0e43 (0x7f7db455ce43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5035884Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7db455da38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5036960Z frame #44: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7db45580b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5037757Z frame #45: <unknown function> + 0x3d10b42 (0x7f7db458cb42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5038438Z frame #46: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7da60545eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:37:28.5038924Z frame #47: <unknown function> + 0xc9039 (0x7f7dbf796039 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-05-18T04:37:28.5039478Z frame #48: <unknown function> + 0x76db (0x7f7df4d9b6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-05-18T04:37:28.5039986Z frame #49: clone + 0x3f (0x7f7df4ac461f in /lib/x86_64-linux-gnu/libc.so.6)
2022-05-18T04:37:28.5040214Z 
2022-05-18T04:37:28.5040234Z 
2022-05-18T04:37:28.5040368Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:37:28.5076430Z RuntimeError('On WorkerInfo(id=1, name=worker1):\nRuntimeError(\'CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from exchangeDevice at /var/lib/jenkins/workspace/c10/cuda/impl/CUDAGuardImpl.h:33 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7da60661bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #1: <unknown function> + 0x146b4 (0x7f7da62b86b4 in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10_cuda.so)\nframe #2: <unknown function> + 0xd8821d (0x7f7da727f21d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #3: <unknown function> + 0x2c59814 (0x7f7da9150814 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #4: <unknown function> + 0x2c598fb (0x7f7da91508fb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #5: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x10f (0x7f7db20aa51f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x1a8f8b5 (0x7f7db230b8b5 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #7: at::_ops::empty_strided::call(c10::ArrayRef<long>, c10::ArrayRef<long>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x174 (0x7f7db20e9314 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x12da (0x7f7db1b046ea in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x1c29a63 (0x7f7db24a5a63 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x10d (0x7f7db1e6b63d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: <unknown function> + 0x1a920d1 (0x7f7db230e0d1 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x10d (0x7f7db1e6b63d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: <unknown function> + 0x2a525ce (0x7f7db32ce5ce in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x2a52b4b (0x7f7db32ceb4b in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x202 (0x7f7db1ee05d2 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f7db1afb8de in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x1d1aa99 (0x7f7db2596a99 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x216 (0x7f7db1ff5676 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: <unknown function> + 0x3224b0 (0x7f7dbc1304b0 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #20: <unknown function> + 0x322965 (0x7f7dbc130965 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #21: <unknown function> + 0x1bfb9c (0x55e4cbcaab9c in /opt/conda/bin/python)\nframe #22: <unknown function> + 0xff72f (0x55e4cbbea72f in /opt/conda/bin/python)\nframe #23: <unknown function> + 0x196663 (0x55e4cbc81663 in /opt/conda/bin/python)\nframe #24: _PyFunction_Vectorcall + 0x1d4 (0x55e4cbc82354 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0xfdae6 (0x55e4cbbe8ae6 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x197bf9 (0x55e4cbc82bf9 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0xff755 (0x55e4cbbea755 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x196663 (0x55e4cbc81663 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x197ca4 (0x55e4cbc82ca4 in /opt/conda/bin/python)\nframe #30: <unknown function> + 0xff755 (0x55e4cbbea755 in /opt/conda/bin/python)\nframe #31: _PyFunction_Vectorcall + 0x104 (0x55e4cbc82284 in /opt/conda/bin/python)\nframe #32: _PyObject_Call + 0x1da (0x55e4cbc30a7a in /opt/conda/bin/python)\nframe #33: _PyEval_EvalFrameDefault + 0x2610 (0x55e4cbcc29f0 in /opt/conda/bin/python)\nframe #34: _PyFunction_Vectorcall + 0x104 (0x55e4cbc82284 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x1da (0x55e4cbc30a7a in /opt/conda/bin/python)\nframe #36: <unknown function> + 0x94774a (0x7f7dbc75574a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #37: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7dbc753a3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #38: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7dbc756b25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #39: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f7dbc7571e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f7db4563b44 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7dbc756915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #42: <unknown function> + 0x3ce0e43 (0x7f7db455ce43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7db455da38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7db45580b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #45: <unknown function> + 0x3d10b42 (0x7f7db458cb42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7da60545eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #47: <unknown function> + 0xc9039 (0x7f7dbf796039 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #48: <unknown function> + 0x76db (0x7f7df4d9b6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #49: clone + 0x3f (0x7f7df4ac461f in /lib/x86_64-linux-gnu/libc.so.6)\n\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.9/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.9/site-packages/torch/distributed/nn/api/remote_module.py", line 89, in _create_module\n    module.to(device)\n  File "/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 927, in to\n    return self._apply(convert)\n  File "/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 602, in _apply\n    param_applied = fn(param)\n  File "/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 925, in convert\n    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)\nRuntimeError: CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from exchangeDevice at /var/lib/jenkins/workspace/c10/cuda/impl/CUDAGuardImpl.h:33 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7da60661bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #1: <unknown function> + 0x146b4 (0x7f7da62b86b4 in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10_cuda.so)\nframe #2: <unknown function> + 0xd8821d (0x7f7da727f21d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #3: <unknown function> + 0x2c59814 (0x7f7da9150814 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #4: <unknown function> + 0x2c598fb (0x7f7da91508fb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #5: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x10f (0x7f7db20aa51f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x1a8f8b5 (0x7f7db230b8b5 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #7: at::_ops::empty_strided::call(c10::ArrayRef<long>, c10::ArrayRef<long>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x174 (0x7f7db20e9314 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x12da (0x7f7db1b046ea in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x1c29a63 (0x7f7db24a5a63 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x10d (0x7f7db1e6b63d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: <unknown function> + 0x1a920d1 (0x7f7db230e0d1 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x10d (0x7f7db1e6b63d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: <unknown function> + 0x2a525ce (0x7f7db32ce5ce in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x2a52b4b (0x7f7db32ceb4b in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x202 (0x7f7db1ee05d2 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f7db1afb8de in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x1d1aa99 (0x7f7db2596a99 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x216 (0x7f7db1ff5676 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: <unknown function> + 0x3224b0 (0x7f7dbc1304b0 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #20: <unknown function> + 0x322965 (0x7f7dbc130965 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #21: <unknown function> + 0x1bfb9c (0x55e4cbcaab9c in /opt/conda/bin/python)\nframe #22: <unknown function> + 0xff72f (0x55e4cbbea72f in /opt/conda/bin/python)\nframe #23: <unknown function> + 0x196663 (0x55e4cbc81663 in /opt/conda/bin/python)\nframe #24: _PyFunction_Vectorcall + 0x1d4 (0x55e4cbc82354 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0xfdae6 (0x55e4cbbe8ae6 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x197bf9 (0x55e4cbc82bf9 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0xff755 (0x55e4cbbea755 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x196663 (0x55e4cbc81663 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x197ca4 (0x55e4cbc82ca4 in /opt/conda/bin/python)\nframe #30: <unknown function> + 0xff755 (0x55e4cbbea755 in /opt/conda/bin/python)\nframe #31: _PyFunction_Vectorcall + 0x104 (0x55e4cbc82284 in /opt/conda/bin/python)\nframe #32: _PyObject_Call + 0x1da (0x55e4cbc30a7a in /opt/conda/bin/python)\nframe #33: _PyEval_EvalFrameDefault + 0x2610 (0x55e4cbcc29f0 in /opt/conda/bin/python)\nframe #34: _PyFunction_Vectorcall + 0x104 (0x55e4cbc82284 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x1da (0x55e4cbc30a7a in /opt/conda/bin/python)\nframe #36: <unknown function> + 0x94774a (0x7f7dbc75574a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #37: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7dbc753a3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #38: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7dbc756b25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #39: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f7dbc7571e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f7db4563b44 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7dbc756915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #42: <unknown function> + 0x3ce0e43 (0x7f7db455ce43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7db455da38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7db45580b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #45: <unknown function> + 0x3d10b42 (0x7f7db458cb42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7da60545eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #47: <unknown function> + 0xc9039 (0x7f7dbf796039 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #48: <unknown function> + 0x76db (0x7f7df4d9b6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #49: clone + 0x3f (0x7f7df4ac461f in /lib/x86_64-linux-gnu/libc.so.6)\n\n')
2022-05-18T04:37:28.5097987Z Traceback (most recent call last):
2022-05-18T04:37:28.5098541Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:37:28.5099008Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:37:28.5099420Z   File "/tmp/tmpmn6u4lae/_remote_module_non_scriptable.py", line 47, in _remote_forward
2022-05-18T04:37:28.5099790Z     module = module_rref.local_value()
2022-05-18T04:37:28.5100323Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception
2022-05-18T04:37:28.5100886Z     raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape"))
2022-05-18T04:37:28.5101292Z RuntimeError: On WorkerInfo(id=1, name=worker1):
2022-05-18T04:37:28.5101697Z RuntimeError('CUDA error: invalid device ordinal
2022-05-18T04:37:28.5102376Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-05-18T04:37:28.5102818Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-05-18T04:37:28.5103301Z Exception raised from exchangeDevice at /var/lib/jenkins/workspace/c10/cuda/impl/CUDAGuardImpl.h:33 (most recent call first):
2022-05-18T04:37:28.5104171Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7da60661bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:37:28.5104915Z frame #1: <unknown function> + 0x146b4 (0x7f7da62b86b4 in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10_cuda.so)
2022-05-18T04:37:28.5105544Z frame #2: <unknown function> + 0xd8821d (0x7f7da727f21d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:37:28.5106183Z frame #3: <unknown function> + 0x2c59814 (0x7f7da9150814 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:37:28.5106815Z frame #4: <unknown function> + 0x2c598fb (0x7f7da91508fb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:37:28.5107821Z frame #5: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x10f (0x7f7db20aa51f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5108659Z frame #6: <unknown function> + 0x1a8f8b5 (0x7f7db230b8b5 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5109595Z frame #7: at::_ops::empty_strided::call(c10::ArrayRef<long>, c10::ArrayRef<long>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x174 (0x7f7db20e9314 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5110814Z frame #8: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x12da (0x7f7db1b046ea in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5111691Z frame #9: <unknown function> + 0x1c29a63 (0x7f7db24a5a63 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5112728Z frame #10: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x10d (0x7f7db1e6b63d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5113593Z frame #11: <unknown function> + 0x1a920d1 (0x7f7db230e0d1 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5114602Z frame #12: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x10d (0x7f7db1e6b63d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5115463Z frame #13: <unknown function> + 0x2a525ce (0x7f7db32ce5ce in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5116119Z frame #14: <unknown function> + 0x2a52b4b (0x7f7db32ceb4b in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5117089Z frame #15: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x202 (0x7f7db1ee05d2 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5118224Z frame #16: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f7db1afb8de in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5119025Z frame #17: <unknown function> + 0x1d1aa99 (0x7f7db2596a99 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5120009Z frame #18: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x216 (0x7f7db1ff5676 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5120849Z frame #19: <unknown function> + 0x3224b0 (0x7f7dbc1304b0 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5121502Z frame #20: <unknown function> + 0x322965 (0x7f7dbc130965 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5121972Z frame #21: <unknown function> + 0x1bfb9c (0x55e4cbcaab9c in /opt/conda/bin/python)
2022-05-18T04:37:28.5122368Z frame #22: <unknown function> + 0xff72f (0x55e4cbbea72f in /opt/conda/bin/python)
2022-05-18T04:37:28.5122770Z frame #23: <unknown function> + 0x196663 (0x55e4cbc81663 in /opt/conda/bin/python)
2022-05-18T04:37:28.5123177Z frame #24: _PyFunction_Vectorcall + 0x1d4 (0x55e4cbc82354 in /opt/conda/bin/python)
2022-05-18T04:37:28.5123577Z frame #25: <unknown function> + 0xfdae6 (0x55e4cbbe8ae6 in /opt/conda/bin/python)
2022-05-18T04:37:28.5123987Z frame #26: <unknown function> + 0x197bf9 (0x55e4cbc82bf9 in /opt/conda/bin/python)
2022-05-18T04:37:28.5124387Z frame #27: <unknown function> + 0xff755 (0x55e4cbbea755 in /opt/conda/bin/python)
2022-05-18T04:37:28.5124857Z frame #28: <unknown function> + 0x196663 (0x55e4cbc81663 in /opt/conda/bin/python)
2022-05-18T04:37:28.5125235Z frame #29: <unknown function> + 0x197ca4 (0x55e4cbc82ca4 in /opt/conda/bin/python)
2022-05-18T04:37:28.5125630Z frame #30: <unknown function> + 0xff755 (0x55e4cbbea755 in /opt/conda/bin/python)
2022-05-18T04:37:28.5126036Z frame #31: _PyFunction_Vectorcall + 0x104 (0x55e4cbc82284 in /opt/conda/bin/python)
2022-05-18T04:37:28.5126425Z frame #32: _PyObject_Call + 0x1da (0x55e4cbc30a7a in /opt/conda/bin/python)
2022-05-18T04:37:28.5126886Z frame #33: _PyEval_EvalFrameDefault + 0x2610 (0x55e4cbcc29f0 in /opt/conda/bin/python)
2022-05-18T04:37:28.5127317Z frame #34: _PyFunction_Vectorcall + 0x104 (0x55e4cbc82284 in /opt/conda/bin/python)
2022-05-18T04:37:28.5127724Z frame #35: _PyObject_Call + 0x1da (0x55e4cbc30a7a in /opt/conda/bin/python)
2022-05-18T04:37:28.5128306Z frame #36: <unknown function> + 0x94774a (0x7f7dbc75574a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5129106Z frame #37: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7dbc753a3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5130126Z frame #38: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7dbc756b25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5131273Z frame #39: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f7dbc7571e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5132513Z frame #40: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f7db4563b44 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5133786Z frame #41: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7dbc756915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5134694Z frame #42: <unknown function> + 0x3ce0e43 (0x7f7db455ce43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5135631Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7db455da38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5136703Z frame #44: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7db45580b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5137491Z frame #45: <unknown function> + 0x3d10b42 (0x7f7db458cb42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5138156Z frame #46: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7da60545eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:37:28.5138666Z frame #47: <unknown function> + 0xc9039 (0x7f7dbf796039 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-05-18T04:37:28.5139216Z frame #48: <unknown function> + 0x76db (0x7f7df4d9b6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-05-18T04:37:28.5139719Z frame #49: clone + 0x3f (0x7f7df4ac461f in /lib/x86_64-linux-gnu/libc.so.6)
2022-05-18T04:37:28.5140023Z ')
2022-05-18T04:37:28.5140272Z Traceback (most recent call last):
2022-05-18T04:37:28.5140861Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:37:28.5141302Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:37:28.5142125Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/nn/api/remote_module.py", line 89, in _create_module
2022-05-18T04:37:28.5142527Z     module.to(device)
2022-05-18T04:37:28.5142990Z   File "/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 927, in to
2022-05-18T04:37:28.5143344Z     return self._apply(convert)
2022-05-18T04:37:28.5143899Z   File "/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 602, in _apply
2022-05-18T04:37:28.5144284Z     param_applied = fn(param)
2022-05-18T04:37:28.5144750Z   File "/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 925, in convert
2022-05-18T04:37:28.5145271Z     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2022-05-18T04:37:28.5145675Z RuntimeError: CUDA error: invalid device ordinal
2022-05-18T04:37:28.5146130Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-05-18T04:37:28.5146572Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-05-18T04:37:28.5147047Z Exception raised from exchangeDevice at /var/lib/jenkins/workspace/c10/cuda/impl/CUDAGuardImpl.h:33 (most recent call first):
2022-05-18T04:37:28.5147903Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7da60661bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:37:28.5148647Z frame #1: <unknown function> + 0x146b4 (0x7f7da62b86b4 in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10_cuda.so)
2022-05-18T04:37:28.5149271Z frame #2: <unknown function> + 0xd8821d (0x7f7da727f21d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:37:28.5149918Z frame #3: <unknown function> + 0x2c59814 (0x7f7da9150814 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:37:28.5150549Z frame #4: <unknown function> + 0x2c598fb (0x7f7da91508fb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:37:28.5151552Z frame #5: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x10f (0x7f7db20aa51f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5152379Z frame #6: <unknown function> + 0x1a8f8b5 (0x7f7db230b8b5 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5153325Z frame #7: at::_ops::empty_strided::call(c10::ArrayRef<long>, c10::ArrayRef<long>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x174 (0x7f7db20e9314 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5154440Z frame #8: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x12da (0x7f7db1b046ea in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5155252Z frame #9: <unknown function> + 0x1c29a63 (0x7f7db24a5a63 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5157268Z frame #10: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x10d (0x7f7db1e6b63d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5158821Z frame #11: <unknown function> + 0x1a920d1 (0x7f7db230e0d1 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5160948Z frame #12: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x10d (0x7f7db1e6b63d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5162155Z frame #13: <unknown function> + 0x2a525ce (0x7f7db32ce5ce in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5162885Z frame #14: <unknown function> + 0x2a52b4b (0x7f7db32ceb4b in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5163870Z frame #15: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x202 (0x7f7db1ee05d2 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5165006Z frame #16: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f7db1afb8de in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5165811Z frame #17: <unknown function> + 0x1d1aa99 (0x7f7db2596a99 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5166801Z frame #18: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x216 (0x7f7db1ff5676 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5167638Z frame #19: <unknown function> + 0x3224b0 (0x7f7dbc1304b0 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5168287Z frame #20: <unknown function> + 0x322965 (0x7f7dbc130965 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5168740Z frame #21: <unknown function> + 0x1bfb9c (0x55e4cbcaab9c in /opt/conda/bin/python)
2022-05-18T04:37:28.5169158Z frame #22: <unknown function> + 0xff72f (0x55e4cbbea72f in /opt/conda/bin/python)
2022-05-18T04:37:28.5169561Z frame #23: <unknown function> + 0x196663 (0x55e4cbc81663 in /opt/conda/bin/python)
2022-05-18T04:37:28.5169963Z frame #24: _PyFunction_Vectorcall + 0x1d4 (0x55e4cbc82354 in /opt/conda/bin/python)
2022-05-18T04:37:28.5170365Z frame #25: <unknown function> + 0xfdae6 (0x55e4cbbe8ae6 in /opt/conda/bin/python)
2022-05-18T04:37:28.5170774Z frame #26: <unknown function> + 0x197bf9 (0x55e4cbc82bf9 in /opt/conda/bin/python)
2022-05-18T04:37:28.5171170Z frame #27: <unknown function> + 0xff755 (0x55e4cbbea755 in /opt/conda/bin/python)
2022-05-18T04:37:28.5171551Z frame #28: <unknown function> + 0x196663 (0x55e4cbc81663 in /opt/conda/bin/python)
2022-05-18T04:37:28.5171953Z frame #29: <unknown function> + 0x197ca4 (0x55e4cbc82ca4 in /opt/conda/bin/python)
2022-05-18T04:37:28.5172355Z frame #30: <unknown function> + 0xff755 (0x55e4cbbea755 in /opt/conda/bin/python)
2022-05-18T04:37:28.5172762Z frame #31: _PyFunction_Vectorcall + 0x104 (0x55e4cbc82284 in /opt/conda/bin/python)
2022-05-18T04:37:28.5173150Z frame #32: _PyObject_Call + 0x1da (0x55e4cbc30a7a in /opt/conda/bin/python)
2022-05-18T04:37:28.5173567Z frame #33: _PyEval_EvalFrameDefault + 0x2610 (0x55e4cbcc29f0 in /opt/conda/bin/python)
2022-05-18T04:37:28.5173998Z frame #34: _PyFunction_Vectorcall + 0x104 (0x55e4cbc82284 in /opt/conda/bin/python)
2022-05-18T04:37:28.5174397Z frame #35: _PyObject_Call + 0x1da (0x55e4cbc30a7a in /opt/conda/bin/python)
2022-05-18T04:37:28.5174982Z frame #36: <unknown function> + 0x94774a (0x7f7dbc75574a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5175777Z frame #37: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7dbc753a3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5176885Z frame #38: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7dbc756b25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5178077Z frame #39: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f7dbc7571e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5179326Z frame #40: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f7db4563b44 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5180592Z frame #41: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7dbc756915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:37:28.5181491Z frame #42: <unknown function> + 0x3ce0e43 (0x7f7db455ce43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5182799Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7db455da38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5183879Z frame #44: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7db45580b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5184680Z frame #45: <unknown function> + 0x3d10b42 (0x7f7db458cb42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:37:28.5185345Z frame #46: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7da60545eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:37:28.5185852Z frame #47: <unknown function> + 0xc9039 (0x7f7dbf796039 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-05-18T04:37:28.5186406Z frame #48: <unknown function> + 0x76db (0x7f7df4d9b6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-05-18T04:37:28.5186910Z frame #49: clone + 0x3f (0x7f7df4ac461f in /lib/x86_64-linux-gnu/libc.so.6)
2022-05-18T04:37:28.5187138Z 
2022-05-18T04:37:28.5187159Z 
2022-05-18T04:37:28.5187182Z 
2022-05-18T04:37:28.7680315Z ok (3.579s)
2022-05-18T04:37:28.7680531Z 
2022-05-18T04:37:28.7681004Z ----------------------------------------------------------------------
2022-05-18T04:37:28.7681379Z Ran 1 test in 3.580s
2022-05-18T04:37:28.7681552Z 
2022-05-18T04:37:28.7681657Z OK
2022-05-18T04:37:28.7681803Z 
2022-05-18T04:37:28.7681923Z Generating XML reports...
2022-05-18T04:37:28.7727136Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518043725.xml
2022-05-18T04:37:30.0611473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_cu33j6_
2022-05-18T04:37:30.0612141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_cu33j6_/_remote_module_non_scriptable.py
2022-05-18T04:37:30.4890856Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:37:30.4902214Z 
2022-05-18T04:37:30.4902659Z Running tests...
2022-05-18T04:37:30.4903613Z ----------------------------------------------------------------------
2022-05-18T04:37:32.1566847Z   test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:32.2013226Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67286
2022-05-18T04:37:32.2137257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67287
2022-05-18T04:37:33.1912048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6srdiugp
2022-05-18T04:37:33.1912648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6srdiugp/_remote_module_non_scriptable.py
2022-05-18T04:37:33.1916228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zbhcmio
2022-05-18T04:37:33.1916801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zbhcmio/_remote_module_non_scriptable.py
2022-05-18T04:37:33.6055154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:33.6124406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:35.3217800Z ok (4.831s)
2022-05-18T04:37:35.3218009Z 
2022-05-18T04:37:35.3218470Z ----------------------------------------------------------------------
2022-05-18T04:37:35.3218814Z Ran 1 test in 4.832s
2022-05-18T04:37:35.3218979Z 
2022-05-18T04:37:35.3219075Z OK
2022-05-18T04:37:35.3219213Z 
2022-05-18T04:37:35.3219333Z Generating XML reports...
2022-05-18T04:37:35.3263540Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518043730.xml
2022-05-18T04:37:36.5888110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzh90ub5
2022-05-18T04:37:36.5888730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzh90ub5/_remote_module_non_scriptable.py
2022-05-18T04:37:37.0195926Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:37:37.0207329Z 
2022-05-18T04:37:37.0207641Z Running tests...
2022-05-18T04:37:37.0208113Z ----------------------------------------------------------------------
2022-05-18T04:37:38.6792021Z   test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:38.7238324Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67570
2022-05-18T04:37:38.7362415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67571
2022-05-18T04:37:38.7476828Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67572
2022-05-18T04:37:38.7599414Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67573
2022-05-18T04:37:39.6975472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wa5eg4m
2022-05-18T04:37:39.6976085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wa5eg4m/_remote_module_non_scriptable.py
2022-05-18T04:37:39.7287849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdfoy_m06
2022-05-18T04:37:39.7288580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdfoy_m06/_remote_module_non_scriptable.py
2022-05-18T04:37:39.7384742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_esvjao
2022-05-18T04:37:39.7386670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_esvjao/_remote_module_non_scriptable.py
2022-05-18T04:37:39.7399372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wfui0d0
2022-05-18T04:37:39.7399969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wfui0d0/_remote_module_non_scriptable.py
2022-05-18T04:37:40.0982772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:40.1332772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:37:40.1494047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:37:40.1508541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:44.4760188Z ok (7.455s)
2022-05-18T04:37:44.4760408Z 
2022-05-18T04:37:44.4760839Z ----------------------------------------------------------------------
2022-05-18T04:37:44.4761181Z Ran 1 test in 7.455s
2022-05-18T04:37:44.4761357Z 
2022-05-18T04:37:44.4761452Z OK
2022-05-18T04:37:44.4761589Z 
2022-05-18T04:37:44.4761733Z Generating XML reports...
2022-05-18T04:37:44.4806216Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20220518043737.xml
2022-05-18T04:37:45.7379721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp815emyll
2022-05-18T04:37:45.7380394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp815emyll/_remote_module_non_scriptable.py
2022-05-18T04:37:46.1608212Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:37:46.1620411Z 
2022-05-18T04:37:46.1620868Z Running tests...
2022-05-18T04:37:46.1621344Z ----------------------------------------------------------------------
2022-05-18T04:37:47.8041063Z   test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:47.8466809Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68073
2022-05-18T04:37:47.8582988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68074
2022-05-18T04:37:48.7910276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdf_aryld
2022-05-18T04:37:48.7910910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdf_aryld/_remote_module_non_scriptable.py
2022-05-18T04:37:48.7962798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0nqq38d
2022-05-18T04:37:48.7964637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0nqq38d/_remote_module_non_scriptable.py
2022-05-18T04:37:49.2000000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:49.2075959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:49.4271669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:49.4274607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:49.4275507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:49.4373282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:52.1006331Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:37:52.1007057Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:37:52.1095241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:37:52.1095760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:37:52.5704533Z ok (6.408s)
2022-05-18T04:37:52.5704751Z 
2022-05-18T04:37:52.5705180Z ----------------------------------------------------------------------
2022-05-18T04:37:52.5705517Z Ran 1 test in 6.408s
2022-05-18T04:37:52.5705695Z 
2022-05-18T04:37:52.5705798Z OK
2022-05-18T04:37:52.5707310Z 
2022-05-18T04:37:52.5707873Z Generating XML reports...
2022-05-18T04:37:52.5750548Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043746.xml
2022-05-18T04:37:53.8034774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpykevqxyr
2022-05-18T04:37:53.8035721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpykevqxyr/_remote_module_non_scriptable.py
2022-05-18T04:37:54.2348286Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:37:54.2359408Z 
2022-05-18T04:37:54.2359782Z Running tests...
2022-05-18T04:37:54.2360277Z ----------------------------------------------------------------------
2022-05-18T04:37:55.9021421Z   test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:37:55.9466206Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68378
2022-05-18T04:37:55.9592234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68379
2022-05-18T04:37:56.8883281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8mq1eox
2022-05-18T04:37:56.8883915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbk2kdlz_
2022-05-18T04:37:56.8884514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8mq1eox/_remote_module_non_scriptable.py
2022-05-18T04:37:56.8885083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbk2kdlz_/_remote_module_non_scriptable.py
2022-05-18T04:37:57.3012756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:37:57.3123257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:37:57.5213276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:37:57.5213843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:57.5214682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:37:57.5215400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:00.2206161Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:00.2206919Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:00.2284411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:00.2284954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:00.6720233Z ok (6.436s)
2022-05-18T04:38:00.6720453Z 
2022-05-18T04:38:00.6720987Z ----------------------------------------------------------------------
2022-05-18T04:38:00.6721350Z Ran 1 test in 6.436s
2022-05-18T04:38:00.6721517Z 
2022-05-18T04:38:00.6721617Z OK
2022-05-18T04:38:00.6721756Z 
2022-05-18T04:38:00.6721869Z Generating XML reports...
2022-05-18T04:38:00.6767768Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043754.xml
2022-05-18T04:38:01.9460521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8rhhpuj
2022-05-18T04:38:01.9461128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8rhhpuj/_remote_module_non_scriptable.py
2022-05-18T04:38:02.3773808Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:38:02.3783918Z 
2022-05-18T04:38:02.3784112Z Running tests...
2022-05-18T04:38:02.3784604Z ----------------------------------------------------------------------
2022-05-18T04:38:04.0418975Z   test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:04.0876727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68683
2022-05-18T04:38:04.0987752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68684
2022-05-18T04:38:05.0558911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppc3ftz8d
2022-05-18T04:38:05.0559867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppc3ftz8d/_remote_module_non_scriptable.py
2022-05-18T04:38:05.0849920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo06b2m0y
2022-05-18T04:38:05.0850582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo06b2m0y/_remote_module_non_scriptable.py
2022-05-18T04:38:05.4643219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:05.5047642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:05.7191167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:05.7192671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:05.7193507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:05.7293774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:08.4758565Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:08.4759251Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:08.4764097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:08.4764623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:08.9106507Z ok (6.532s)
2022-05-18T04:38:08.9106897Z 
2022-05-18T04:38:08.9107624Z ----------------------------------------------------------------------
2022-05-18T04:38:08.9108297Z Ran 1 test in 6.532s
2022-05-18T04:38:08.9108611Z 
2022-05-18T04:38:08.9108782Z OK
2022-05-18T04:38:08.9109059Z 
2022-05-18T04:38:08.9109320Z Generating XML reports...
2022-05-18T04:38:08.9153975Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043802.xml
2022-05-18T04:38:10.1424499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0oljyno
2022-05-18T04:38:10.1425098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0oljyno/_remote_module_non_scriptable.py
2022-05-18T04:38:10.5594985Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:38:10.5606350Z 
2022-05-18T04:38:10.5606613Z Running tests...
2022-05-18T04:38:10.5607114Z ----------------------------------------------------------------------
2022-05-18T04:38:12.1701651Z   test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:12.2145229Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68988
2022-05-18T04:38:12.2270413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68989
2022-05-18T04:38:13.1418170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpquoa0irf
2022-05-18T04:38:13.1418802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpquoa0irf/_remote_module_non_scriptable.py
2022-05-18T04:38:13.1545887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ejhlpq2
2022-05-18T04:38:13.1546496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ejhlpq2/_remote_module_non_scriptable.py
2022-05-18T04:38:13.5594156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:13.5648574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:13.7812668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:13.7813624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:13.7814479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:13.7815157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:16.4397103Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:16.4398190Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:16.9397618Z ok (6.379s)
2022-05-18T04:38:16.9397845Z 
2022-05-18T04:38:16.9398885Z ----------------------------------------------------------------------
2022-05-18T04:38:16.9399244Z Ran 1 test in 6.379s
2022-05-18T04:38:16.9399432Z 
2022-05-18T04:38:16.9399529Z OK
2022-05-18T04:38:16.9399685Z 
2022-05-18T04:38:16.9399830Z Generating XML reports...
2022-05-18T04:38:16.9443333Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043810.xml
2022-05-18T04:38:18.2473246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbp6xst3b
2022-05-18T04:38:18.2473883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbp6xst3b/_remote_module_non_scriptable.py
2022-05-18T04:38:18.6733951Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:38:18.6746255Z 
2022-05-18T04:38:18.6746547Z Running tests...
2022-05-18T04:38:18.6747029Z ----------------------------------------------------------------------
2022-05-18T04:38:20.3258414Z   test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:20.3690740Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69293
2022-05-18T04:38:20.3814547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69294
2022-05-18T04:38:21.2980568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptx7h9h6u
2022-05-18T04:38:21.2981214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptx7h9h6u/_remote_module_non_scriptable.py
2022-05-18T04:38:21.3322472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxy1410v_
2022-05-18T04:38:21.3323106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxy1410v_/_remote_module_non_scriptable.py
2022-05-18T04:38:21.7155111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:21.7466536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:21.9407740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:21.9408342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:21.9409232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:21.9409953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:24.7140296Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:24.7140993Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:24.7222144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:24.7222983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:25.1943689Z ok (6.519s)
2022-05-18T04:38:25.1943991Z 
2022-05-18T04:38:25.1945013Z ----------------------------------------------------------------------
2022-05-18T04:38:25.1945387Z Ran 1 test in 6.520s
2022-05-18T04:38:25.1945553Z 
2022-05-18T04:38:25.1945653Z OK
2022-05-18T04:38:25.1945794Z 
2022-05-18T04:38:25.1945909Z Generating XML reports...
2022-05-18T04:38:25.1991645Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043818.xml
2022-05-18T04:38:26.4805759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3mkxay6x
2022-05-18T04:38:26.4806625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3mkxay6x/_remote_module_non_scriptable.py
2022-05-18T04:38:26.9113919Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:38:26.9124456Z 
2022-05-18T04:38:26.9124747Z Running tests...
2022-05-18T04:38:26.9125231Z ----------------------------------------------------------------------
2022-05-18T04:38:28.5819101Z   test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:28.6255239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69608
2022-05-18T04:38:28.6378458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69609
2022-05-18T04:38:29.5761517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_56utuh
2022-05-18T04:38:29.5762154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_56utuh/_remote_module_non_scriptable.py
2022-05-18T04:38:29.5887328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfdshzf0i
2022-05-18T04:38:29.5888349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfdshzf0i/_remote_module_non_scriptable.py
2022-05-18T04:38:29.9870575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:30.0072721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:30.2248275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:30.2249510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:30.2250353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:30.2251057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:33.0194872Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:33.0195541Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:33.0263885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:33.0264437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:33.5507096Z ok (6.638s)
2022-05-18T04:38:33.5507303Z 
2022-05-18T04:38:33.5507729Z ----------------------------------------------------------------------
2022-05-18T04:38:33.5508102Z Ran 1 test in 6.638s
2022-05-18T04:38:33.5508274Z 
2022-05-18T04:38:33.5508375Z OK
2022-05-18T04:38:33.5508492Z 
2022-05-18T04:38:33.5508630Z Generating XML reports...
2022-05-18T04:38:33.5554509Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043826.xml
2022-05-18T04:38:34.8200524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdctww5a
2022-05-18T04:38:34.8201156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdctww5a/_remote_module_non_scriptable.py
2022-05-18T04:38:35.2503677Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:38:35.2514940Z 
2022-05-18T04:38:35.2515219Z Running tests...
2022-05-18T04:38:35.2515697Z ----------------------------------------------------------------------
2022-05-18T04:38:36.9118945Z   test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:36.9548451Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69923
2022-05-18T04:38:36.9665316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69924
2022-05-18T04:38:37.8964069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53_gnr0p
2022-05-18T04:38:37.8964940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53_gnr0p/_remote_module_non_scriptable.py
2022-05-18T04:38:37.9121187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiqnrafyy
2022-05-18T04:38:37.9121793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiqnrafyy/_remote_module_non_scriptable.py
2022-05-18T04:38:38.3171214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:38.3192460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:38.5688431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:38.5689353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:38.5689911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:38.5690613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:41.3841790Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:41.3842499Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:41.3846278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:41.3846823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:38:41.8794739Z ok (6.627s)
2022-05-18T04:38:41.8795153Z 
2022-05-18T04:38:41.8797697Z ----------------------------------------------------------------------
2022-05-18T04:38:41.8798064Z Ran 1 test in 6.628s
2022-05-18T04:38:41.8798249Z 
2022-05-18T04:38:41.8798347Z OK
2022-05-18T04:38:41.8798491Z 
2022-05-18T04:38:41.8798628Z Generating XML reports...
2022-05-18T04:38:41.8840239Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043835.xml
2022-05-18T04:38:43.1341159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuixholo2
2022-05-18T04:38:43.1341802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuixholo2/_remote_module_non_scriptable.py
2022-05-18T04:38:43.5494768Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:38:43.5504870Z 
2022-05-18T04:38:43.5505155Z Running tests...
2022-05-18T04:38:43.5505643Z ----------------------------------------------------------------------
2022-05-18T04:38:45.1554051Z   test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:45.1983305Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70238
2022-05-18T04:38:45.2099377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70239
2022-05-18T04:38:46.1121313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2wlcsyh
2022-05-18T04:38:46.1121959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2wlcsyh/_remote_module_non_scriptable.py
2022-05-18T04:38:46.1495955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkuvq_3k
2022-05-18T04:38:46.1496549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkuvq_3k/_remote_module_non_scriptable.py
2022-05-18T04:38:46.5237415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:46.5644473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:46.7569112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:38:46.7570016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:38:46.7571120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:46.7571820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:38:49.5352864Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:49.5353634Z [W logger.cpp:316] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-05-18T04:38:50.0222672Z ok (6.471s)
2022-05-18T04:38:50.0222931Z 
2022-05-18T04:38:50.0223450Z ----------------------------------------------------------------------
2022-05-18T04:38:50.0224013Z Ran 1 test in 6.472s
2022-05-18T04:38:50.0224315Z 
2022-05-18T04:38:50.0224488Z OK
2022-05-18T04:38:50.0224786Z 
2022-05-18T04:38:50.0225043Z Generating XML reports...
2022-05-18T04:38:50.0268229Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043843.xml
2022-05-18T04:38:51.2669200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprv93n2vw
2022-05-18T04:38:51.2669853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprv93n2vw/_remote_module_non_scriptable.py
2022-05-18T04:38:51.6794520Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:38:51.6804860Z 
2022-05-18T04:38:51.6805158Z Running tests...
2022-05-18T04:38:51.6805649Z ----------------------------------------------------------------------
2022-05-18T04:38:53.3116750Z   test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:38:53.3545522Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70553
2022-05-18T04:38:53.3665207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70554
2022-05-18T04:38:53.3790388Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70555
2022-05-18T04:38:53.3911618Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70556
2022-05-18T04:38:54.3441984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ew6udqj
2022-05-18T04:38:54.3442717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ew6udqj/_remote_module_non_scriptable.py
2022-05-18T04:38:54.3585915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1l1_7jbk
2022-05-18T04:38:54.3586490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1l1_7jbk/_remote_module_non_scriptable.py
2022-05-18T04:38:54.3638996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxo9791nk
2022-05-18T04:38:54.3639608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxo9791nk/_remote_module_non_scriptable.py
2022-05-18T04:38:54.4115558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ua6p0ea
2022-05-18T04:38:54.4116159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ua6p0ea/_remote_module_non_scriptable.py
2022-05-18T04:38:54.7510423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:38:54.7574497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:38:54.7704389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:38:54.8137734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:38:59.4067189Z ok (7.726s)
2022-05-18T04:38:59.4067433Z 
2022-05-18T04:38:59.4068446Z ----------------------------------------------------------------------
2022-05-18T04:38:59.4068790Z Ran 1 test in 7.726s
2022-05-18T04:38:59.4068980Z 
2022-05-18T04:38:59.4069495Z OK
2022-05-18T04:38:59.4069681Z 
2022-05-18T04:38:59.4069828Z Generating XML reports...
2022-05-18T04:38:59.4112442Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043851.xml
2022-05-18T04:39:00.6996569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9z5tznh
2022-05-18T04:39:00.7000162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9z5tznh/_remote_module_non_scriptable.py
2022-05-18T04:39:01.1329267Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:39:01.1340359Z 
2022-05-18T04:39:01.1340677Z Running tests...
2022-05-18T04:39:01.1341449Z ----------------------------------------------------------------------
2022-05-18T04:39:02.8140853Z   test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:02.8577910Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71056
2022-05-18T04:39:02.8697770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71057
2022-05-18T04:39:02.8823824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71058
2022-05-18T04:39:02.8945344Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71059
2022-05-18T04:39:03.8883581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpykgxl64t
2022-05-18T04:39:03.8884196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpykgxl64t/_remote_module_non_scriptable.py
2022-05-18T04:39:03.8985079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_szy9mic
2022-05-18T04:39:03.8986222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_szy9mic/_remote_module_non_scriptable.py
2022-05-18T04:39:03.9131612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk76jkq4w
2022-05-18T04:39:03.9132519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk76jkq4w/_remote_module_non_scriptable.py
2022-05-18T04:39:03.9337671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpghx6w2_s
2022-05-18T04:39:03.9338286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpghx6w2_s/_remote_module_non_scriptable.py
2022-05-18T04:39:04.2904841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:39:04.3139040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:04.3293925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:04.3371515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:39:11.6153887Z ok (10.481s)
2022-05-18T04:39:11.6154130Z 
2022-05-18T04:39:11.6154593Z ----------------------------------------------------------------------
2022-05-18T04:39:11.6154946Z Ran 1 test in 10.481s
2022-05-18T04:39:11.6155115Z 
2022-05-18T04:39:11.6155210Z OK
2022-05-18T04:39:11.6155346Z 
2022-05-18T04:39:11.6155487Z Generating XML reports...
2022-05-18T04:39:11.6199857Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043901.xml
2022-05-18T04:39:12.8861801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mivsviy
2022-05-18T04:39:12.8862744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mivsviy/_remote_module_non_scriptable.py
2022-05-18T04:39:13.3169863Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:39:13.3181924Z 
2022-05-18T04:39:13.3182126Z Running tests...
2022-05-18T04:39:13.3182956Z ----------------------------------------------------------------------
2022-05-18T04:39:14.9623164Z   test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:15.0062856Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71559
2022-05-18T04:39:15.0183207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71560
2022-05-18T04:39:15.0312362Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71561
2022-05-18T04:39:15.0425606Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71562
2022-05-18T04:39:15.9706828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmerc8ej6
2022-05-18T04:39:15.9708015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmerc8ej6/_remote_module_non_scriptable.py
2022-05-18T04:39:15.9919039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpko81ct7d
2022-05-18T04:39:15.9920095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpko81ct7d/_remote_module_non_scriptable.py
2022-05-18T04:39:16.0186220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppesbe2l4
2022-05-18T04:39:16.0187424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppesbe2l4/_remote_module_non_scriptable.py
2022-05-18T04:39:16.0533880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmlnd92ny
2022-05-18T04:39:16.0535069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmlnd92ny/_remote_module_non_scriptable.py
2022-05-18T04:39:16.3954604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:16.4011298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:39:16.4386117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:16.4635487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:39:21.3591378Z ok (8.041s)
2022-05-18T04:39:21.3591612Z 
2022-05-18T04:39:21.3592064Z ----------------------------------------------------------------------
2022-05-18T04:39:21.3592430Z Ran 1 test in 8.041s
2022-05-18T04:39:21.3592605Z 
2022-05-18T04:39:21.3592703Z OK
2022-05-18T04:39:21.3592846Z 
2022-05-18T04:39:21.3592996Z Generating XML reports...
2022-05-18T04:39:21.3637192Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043913.xml
2022-05-18T04:39:22.5979251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpvlzpny
2022-05-18T04:39:22.5979893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpvlzpny/_remote_module_non_scriptable.py
2022-05-18T04:39:23.0142600Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:39:23.0153942Z 
2022-05-18T04:39:23.0154256Z Running tests...
2022-05-18T04:39:23.0154767Z ----------------------------------------------------------------------
2022-05-18T04:39:24.6160368Z   test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:24.6579045Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71898
2022-05-18T04:39:24.6695012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71899
2022-05-18T04:39:24.6814083Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71900
2022-05-18T04:39:24.6932176Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71901
2022-05-18T04:39:25.6802008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd55xntl4
2022-05-18T04:39:25.6802654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd55xntl4/_remote_module_non_scriptable.py
2022-05-18T04:39:25.7006646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7zgafpk
2022-05-18T04:39:25.7007527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7zgafpk/_remote_module_non_scriptable.py
2022-05-18T04:39:25.7008578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgxr3g2na
2022-05-18T04:39:25.7011316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgxr3g2na/_remote_module_non_scriptable.py
2022-05-18T04:39:25.7522742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3c8qi794
2022-05-18T04:39:25.7523363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3c8qi794/_remote_module_non_scriptable.py
2022-05-18T04:39:26.0794064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:26.1024567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:39:26.1134192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:26.1646535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:39:31.7104437Z ok (8.695s)
2022-05-18T04:39:31.7104669Z 
2022-05-18T04:39:31.7105108Z ----------------------------------------------------------------------
2022-05-18T04:39:31.7105457Z Ran 1 test in 8.695s
2022-05-18T04:39:31.7105649Z 
2022-05-18T04:39:31.7105747Z OK
2022-05-18T04:39:31.7105890Z 
2022-05-18T04:39:31.7106032Z Generating XML reports...
2022-05-18T04:39:31.7150409Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043923.xml
2022-05-18T04:39:32.9813994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77odbjvv
2022-05-18T04:39:32.9814635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77odbjvv/_remote_module_non_scriptable.py
2022-05-18T04:39:33.4047061Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:39:33.4058238Z 
2022-05-18T04:39:33.4058515Z Running tests...
2022-05-18T04:39:33.4058997Z ----------------------------------------------------------------------
2022-05-18T04:39:35.0378021Z   test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:35.0813212Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72357
2022-05-18T04:39:35.0930362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72358
2022-05-18T04:39:35.1051160Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72359
2022-05-18T04:39:35.1171981Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72360
2022-05-18T04:39:36.1012096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mixgyns
2022-05-18T04:39:36.1012762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mixgyns/_remote_module_non_scriptable.py
2022-05-18T04:39:36.1336185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjkbbr09
2022-05-18T04:39:36.1339347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjkbbr09/_remote_module_non_scriptable.py
2022-05-18T04:39:36.1415331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3k8jodbq
2022-05-18T04:39:36.1416211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3k8jodbq/_remote_module_non_scriptable.py
2022-05-18T04:39:36.1450506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphjsyk79y
2022-05-18T04:39:36.1451354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphjsyk79y/_remote_module_non_scriptable.py
2022-05-18T04:39:36.5195215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:36.5441084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:39:36.5500307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:39:36.5543861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:42.2345355Z ok (8.828s)
2022-05-18T04:39:42.2345633Z 
2022-05-18T04:39:42.2346088Z ----------------------------------------------------------------------
2022-05-18T04:39:42.2346437Z Ran 1 test in 8.829s
2022-05-18T04:39:42.2346602Z 
2022-05-18T04:39:42.2346679Z OK
2022-05-18T04:39:42.2346868Z 
2022-05-18T04:39:42.2347008Z Generating XML reports...
2022-05-18T04:39:42.2392481Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043933.xml
2022-05-18T04:39:43.5054291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69yrigph
2022-05-18T04:39:43.5054947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69yrigph/_remote_module_non_scriptable.py
2022-05-18T04:39:43.9317822Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:39:43.9328126Z 
2022-05-18T04:39:43.9328449Z Running tests...
2022-05-18T04:39:43.9328914Z ----------------------------------------------------------------------
2022-05-18T04:39:45.5890936Z   test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:45.6325727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72692
2022-05-18T04:39:45.6446069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72693
2022-05-18T04:39:45.6568170Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72694
2022-05-18T04:39:45.6681754Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72695
2022-05-18T04:39:46.6242395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfxq7ruq
2022-05-18T04:39:46.6243039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfxq7ruq/_remote_module_non_scriptable.py
2022-05-18T04:39:46.6489257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbaz78j2
2022-05-18T04:39:46.6489873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbaz78j2/_remote_module_non_scriptable.py
2022-05-18T04:39:46.6525238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzlsxe98
2022-05-18T04:39:46.6525850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzlsxe98/_remote_module_non_scriptable.py
2022-05-18T04:39:46.6687117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgd1kfp_
2022-05-18T04:39:46.6687718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgd1kfp_/_remote_module_non_scriptable.py
2022-05-18T04:39:47.0240920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:47.0498921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:47.0550035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:39:47.0860305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:39:52.6855985Z ok (8.752s)
2022-05-18T04:39:52.6856200Z 
2022-05-18T04:39:52.6856825Z ----------------------------------------------------------------------
2022-05-18T04:39:52.6857181Z Ran 1 test in 8.753s
2022-05-18T04:39:52.6857362Z 
2022-05-18T04:39:52.6857465Z OK
2022-05-18T04:39:52.6857606Z 
2022-05-18T04:39:52.6857728Z Generating XML reports...
2022-05-18T04:39:52.6901132Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043943.xml
2022-05-18T04:39:53.9387123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcg89a7o
2022-05-18T04:39:53.9387834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcg89a7o/_remote_module_non_scriptable.py
2022-05-18T04:39:54.3664266Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:39:54.3675024Z 
2022-05-18T04:39:54.3675384Z Running tests...
2022-05-18T04:39:54.3675872Z ----------------------------------------------------------------------
2022-05-18T04:39:56.0119296Z   test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:39:56.0568091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73155
2022-05-18T04:39:56.0695633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73156
2022-05-18T04:39:56.0827386Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73157
2022-05-18T04:39:56.0954311Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73158
2022-05-18T04:39:57.0453219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoyod83c8
2022-05-18T04:39:57.0453859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoyod83c8/_remote_module_non_scriptable.py
2022-05-18T04:39:57.0959468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm63lki93
2022-05-18T04:39:57.0960591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm63lki93/_remote_module_non_scriptable.py
2022-05-18T04:39:57.1360009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadabkhnh
2022-05-18T04:39:57.1360615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadabkhnh/_remote_module_non_scriptable.py
2022-05-18T04:39:57.1501432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0t2t7zdj
2022-05-18T04:39:57.1502301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0t2t7zdj/_remote_module_non_scriptable.py
2022-05-18T04:39:57.4516571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:39:57.4996129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:39:57.5555804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:39:57.5583476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:40:03.1127387Z ok (8.745s)
2022-05-18T04:40:03.1127621Z 
2022-05-18T04:40:03.1128726Z ----------------------------------------------------------------------
2022-05-18T04:40:03.1129087Z Ran 1 test in 8.745s
2022-05-18T04:40:03.1129270Z 
2022-05-18T04:40:03.1129350Z OK
2022-05-18T04:40:03.1129493Z 
2022-05-18T04:40:03.1129632Z Generating XML reports...
2022-05-18T04:40:03.1173766Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043954.xml
2022-05-18T04:40:04.3669840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lduzhaw
2022-05-18T04:40:04.3670540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lduzhaw/_remote_module_non_scriptable.py
2022-05-18T04:40:04.7993224Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:40:04.8003500Z 
2022-05-18T04:40:04.8003969Z Running tests...
2022-05-18T04:40:04.8004735Z ----------------------------------------------------------------------
2022-05-18T04:40:06.4907723Z   test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:06.5365265Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73494
2022-05-18T04:40:06.5490093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73495
2022-05-18T04:40:06.5619321Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73496
2022-05-18T04:40:06.5736600Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73497
2022-05-18T04:40:07.4970431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjfsmxt75
2022-05-18T04:40:07.4971079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjfsmxt75/_remote_module_non_scriptable.py
2022-05-18T04:40:07.5518202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdztibfb_
2022-05-18T04:40:07.5518903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrq2pwjy
2022-05-18T04:40:07.5519474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdztibfb_/_remote_module_non_scriptable.py
2022-05-18T04:40:07.5520334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrq2pwjy/_remote_module_non_scriptable.py
2022-05-18T04:40:07.5708881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppnsv0lvl
2022-05-18T04:40:07.5709463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppnsv0lvl/_remote_module_non_scriptable.py
2022-05-18T04:40:07.9187724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:07.9520612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:40:07.9609903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:07.9801315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:40:12.7892410Z ok (7.989s)
2022-05-18T04:40:12.7892705Z 
2022-05-18T04:40:12.7893161Z ----------------------------------------------------------------------
2022-05-18T04:40:12.7893521Z Ran 1 test in 7.989s
2022-05-18T04:40:12.7893688Z 
2022-05-18T04:40:12.7893790Z OK
2022-05-18T04:40:12.7893911Z 
2022-05-18T04:40:12.7894071Z Generating XML reports...
2022-05-18T04:40:12.7937953Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044004.xml
2022-05-18T04:40:14.0393817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1h64l4zm
2022-05-18T04:40:14.0394468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1h64l4zm/_remote_module_non_scriptable.py
2022-05-18T04:40:14.4690415Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:40:14.4700996Z 
2022-05-18T04:40:14.4701361Z Running tests...
2022-05-18T04:40:14.4702037Z ----------------------------------------------------------------------
2022-05-18T04:40:16.1189694Z   test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:16.1635494Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73953
2022-05-18T04:40:16.1760994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73954
2022-05-18T04:40:16.1893225Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73955
2022-05-18T04:40:16.2021226Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73956
2022-05-18T04:40:17.1725222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_ykcchd
2022-05-18T04:40:17.1725868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_ykcchd/_remote_module_non_scriptable.py
2022-05-18T04:40:17.2231316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5vmtmdf
2022-05-18T04:40:17.2231897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5vmtmdf/_remote_module_non_scriptable.py
2022-05-18T04:40:17.2503901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_34g5j5
2022-05-18T04:40:17.2506057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_34g5j5/_remote_module_non_scriptable.py
2022-05-18T04:40:17.3239173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpff27tt2u
2022-05-18T04:40:17.3239854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpff27tt2u/_remote_module_non_scriptable.py
2022-05-18T04:40:17.5752059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:17.6417133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:17.6632160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:40:17.7366508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:40:23.2197487Z ok (8.749s)
2022-05-18T04:40:23.2197741Z 
2022-05-18T04:40:23.2198180Z ----------------------------------------------------------------------
2022-05-18T04:40:23.2198533Z Ran 1 test in 8.750s
2022-05-18T04:40:23.2198728Z 
2022-05-18T04:40:23.2198829Z OK
2022-05-18T04:40:23.2198971Z 
2022-05-18T04:40:23.2201984Z Generating XML reports...
2022-05-18T04:40:23.2245286Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044014.xml
2022-05-18T04:40:24.4659809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpng14ji0t
2022-05-18T04:40:24.4660450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpng14ji0t/_remote_module_non_scriptable.py
2022-05-18T04:40:24.8975202Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:40:24.8986158Z 
2022-05-18T04:40:24.8986445Z Running tests...
2022-05-18T04:40:24.8986944Z ----------------------------------------------------------------------
2022-05-18T04:40:26.5861241Z   test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:26.6313640Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74288
2022-05-18T04:40:26.6438966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74289
2022-05-18T04:40:26.6554699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74290
2022-05-18T04:40:26.6680006Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74291
2022-05-18T04:40:27.5809387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05vmxjnr
2022-05-18T04:40:27.5810077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05vmxjnr/_remote_module_non_scriptable.py
2022-05-18T04:40:27.5812010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0220sysr
2022-05-18T04:40:27.5812988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0220sysr/_remote_module_non_scriptable.py
2022-05-18T04:40:27.6239584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbsfptq58
2022-05-18T04:40:27.6240175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbsfptq58/_remote_module_non_scriptable.py
2022-05-18T04:40:27.6269275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdqx_71ct
2022-05-18T04:40:27.6270241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdqx_71ct/_remote_module_non_scriptable.py
2022-05-18T04:40:27.9851728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:27.9854767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:40:28.0271453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:28.0396660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:40:28.2738966Z ok (3.375s)
2022-05-18T04:40:28.2739208Z 
2022-05-18T04:40:28.2740127Z ----------------------------------------------------------------------
2022-05-18T04:40:28.2740589Z Ran 1 test in 3.375s
2022-05-18T04:40:28.2740762Z 
2022-05-18T04:40:28.2740838Z OK
2022-05-18T04:40:28.2740980Z 
2022-05-18T04:40:28.2741121Z Generating XML reports...
2022-05-18T04:40:28.2784739Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044024.xml
2022-05-18T04:40:29.5378057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvq96vwjm
2022-05-18T04:40:29.5378693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvq96vwjm/_remote_module_non_scriptable.py
2022-05-18T04:40:29.9767947Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:40:29.9778629Z 
2022-05-18T04:40:29.9779210Z Running tests...
2022-05-18T04:40:29.9779675Z ----------------------------------------------------------------------
2022-05-18T04:40:31.6520559Z   test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:31.6980633Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74619
2022-05-18T04:40:31.7106083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74620
2022-05-18T04:40:31.7237957Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74621
2022-05-18T04:40:31.7353850Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74622
2022-05-18T04:40:32.6913873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkuv4cxv3
2022-05-18T04:40:32.6914525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkuv4cxv3/_remote_module_non_scriptable.py
2022-05-18T04:40:32.7374370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm26pum4v
2022-05-18T04:40:32.7375023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm26pum4v/_remote_module_non_scriptable.py
2022-05-18T04:40:32.7451762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4jlu9qg
2022-05-18T04:40:32.7452359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4jlu9qg/_remote_module_non_scriptable.py
2022-05-18T04:40:32.7506353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1e7d2te6
2022-05-18T04:40:32.7507164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1e7d2te6/_remote_module_non_scriptable.py
2022-05-18T04:40:33.0909599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:40:33.1442236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:33.1451311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:33.1573712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:40:33.3413499Z ok (3.363s)
2022-05-18T04:40:33.3413766Z 
2022-05-18T04:40:33.3414195Z ----------------------------------------------------------------------
2022-05-18T04:40:33.3414579Z Ran 1 test in 3.364s
2022-05-18T04:40:33.3414751Z 
2022-05-18T04:40:33.3414849Z OK
2022-05-18T04:40:33.3414966Z 
2022-05-18T04:40:33.3415516Z Generating XML reports...
2022-05-18T04:40:33.3459276Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044029.xml
2022-05-18T04:40:34.6082916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplp_d6zo2
2022-05-18T04:40:34.6083537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplp_d6zo2/_remote_module_non_scriptable.py
2022-05-18T04:40:35.0423024Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:40:35.0434412Z 
2022-05-18T04:40:35.0434832Z Running tests...
2022-05-18T04:40:35.0435608Z ----------------------------------------------------------------------
2022-05-18T04:40:36.7312441Z   test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:36.7771553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74950
2022-05-18T04:40:36.7897435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74951
2022-05-18T04:40:36.8035448Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74952
2022-05-18T04:40:36.8163867Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74953
2022-05-18T04:40:37.8167408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_emj2lq
2022-05-18T04:40:37.8168032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_emj2lq/_remote_module_non_scriptable.py
2022-05-18T04:40:37.8263850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyqsf9vk
2022-05-18T04:40:37.8264470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyqsf9vk/_remote_module_non_scriptable.py
2022-05-18T04:40:37.8283607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4r5zahno
2022-05-18T04:40:37.8284455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4r5zahno/_remote_module_non_scriptable.py
2022-05-18T04:40:37.8726575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpclb3lqi9
2022-05-18T04:40:37.8727142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpclb3lqi9/_remote_module_non_scriptable.py
2022-05-18T04:40:38.2290167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:38.2303928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:40:38.2310688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:38.2914585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:40:38.5226553Z ok (3.479s)
2022-05-18T04:40:38.5226822Z 
2022-05-18T04:40:38.5227241Z ----------------------------------------------------------------------
2022-05-18T04:40:38.5227610Z Ran 1 test in 3.479s
2022-05-18T04:40:38.5227804Z 
2022-05-18T04:40:38.5227907Z OK
2022-05-18T04:40:38.5228057Z 
2022-05-18T04:40:38.5228202Z Generating XML reports...
2022-05-18T04:40:38.5273248Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044035.xml
2022-05-18T04:40:39.8231908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6ji6yrn
2022-05-18T04:40:39.8232532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6ji6yrn/_remote_module_non_scriptable.py
2022-05-18T04:40:40.2567887Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:40:40.2579952Z 
2022-05-18T04:40:40.2580271Z Running tests...
2022-05-18T04:40:40.2580731Z ----------------------------------------------------------------------
2022-05-18T04:40:41.9217402Z   test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:41.9658400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75281
2022-05-18T04:40:41.9779949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75282
2022-05-18T04:40:41.9895948Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75283
2022-05-18T04:40:42.0008713Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75284
2022-05-18T04:40:42.9581074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm14thlgt
2022-05-18T04:40:42.9582276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm14thlgt/_remote_module_non_scriptable.py
2022-05-18T04:40:43.0030144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1hrq3hg
2022-05-18T04:40:43.0030745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1hrq3hg/_remote_module_non_scriptable.py
2022-05-18T04:40:43.0039735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kmmbtcd
2022-05-18T04:40:43.0040346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kmmbtcd/_remote_module_non_scriptable.py
2022-05-18T04:40:43.0129721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpueuqx8n7
2022-05-18T04:40:43.0130313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpueuqx8n7/_remote_module_non_scriptable.py
2022-05-18T04:40:43.3766031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:43.4086485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:40:43.4092490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:43.4358146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:40:43.6068715Z ok (3.348s)
2022-05-18T04:40:43.6068937Z 
2022-05-18T04:40:43.6069408Z ----------------------------------------------------------------------
2022-05-18T04:40:43.6069769Z Ran 1 test in 3.349s
2022-05-18T04:40:43.6069942Z 
2022-05-18T04:40:43.6070021Z OK
2022-05-18T04:40:43.6070160Z 
2022-05-18T04:40:43.6070300Z Generating XML reports...
2022-05-18T04:40:43.6114892Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044040.xml
2022-05-18T04:40:44.8871873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnw73gqyn
2022-05-18T04:40:44.8872571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnw73gqyn/_remote_module_non_scriptable.py
2022-05-18T04:40:45.3249441Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:40:45.3260655Z 
2022-05-18T04:40:45.3261005Z Running tests...
2022-05-18T04:40:45.3261648Z ----------------------------------------------------------------------
2022-05-18T04:40:47.0010634Z   test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:47.0456588Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75612
2022-05-18T04:40:47.0581739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75613
2022-05-18T04:40:47.0709197Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75614
2022-05-18T04:40:47.0835023Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75615
2022-05-18T04:40:48.0464082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz91tquy4
2022-05-18T04:40:48.0464691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz91tquy4/_remote_module_non_scriptable.py
2022-05-18T04:40:48.1133119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqnzsf5z
2022-05-18T04:40:48.1133731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqnzsf5z/_remote_module_non_scriptable.py
2022-05-18T04:40:48.1134597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmuyrhmr
2022-05-18T04:40:48.1136384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmuyrhmr/_remote_module_non_scriptable.py
2022-05-18T04:40:48.1276468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8ctxh5t
2022-05-18T04:40:48.1277061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8ctxh5t/_remote_module_non_scriptable.py
2022-05-18T04:40:48.4638490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:48.5199917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:40:48.5200480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:48.5413683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:40:50.1928838Z ok (4.866s)
2022-05-18T04:40:50.1929060Z 
2022-05-18T04:40:50.1929502Z ----------------------------------------------------------------------
2022-05-18T04:40:50.1929846Z Ran 1 test in 4.867s
2022-05-18T04:40:50.1930012Z 
2022-05-18T04:40:50.1930109Z OK
2022-05-18T04:40:50.1930250Z 
2022-05-18T04:40:50.1932073Z Generating XML reports...
2022-05-18T04:40:50.1974111Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044045.xml
2022-05-18T04:40:51.4465560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5x9x4qmm
2022-05-18T04:40:51.4466168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5x9x4qmm/_remote_module_non_scriptable.py
2022-05-18T04:40:51.8636991Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:40:51.8648521Z 
2022-05-18T04:40:51.8648929Z Running tests...
2022-05-18T04:40:51.8649908Z ----------------------------------------------------------------------
2022-05-18T04:40:53.5060027Z   test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:40:53.5506541Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75947
2022-05-18T04:40:53.5631680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75948
2022-05-18T04:40:53.5759949Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75949
2022-05-18T04:40:53.5874756Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75950
2022-05-18T04:40:54.5515469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp252kt6x4
2022-05-18T04:40:54.5516171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp252kt6x4/_remote_module_non_scriptable.py
2022-05-18T04:40:54.5789904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafnpm2jr
2022-05-18T04:40:54.5790525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafnpm2jr/_remote_module_non_scriptable.py
2022-05-18T04:40:54.5889792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kk__l2b
2022-05-18T04:40:54.5890372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kk__l2b/_remote_module_non_scriptable.py
2022-05-18T04:40:54.5911030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfdcw517r
2022-05-18T04:40:54.5911626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfdcw517r/_remote_module_non_scriptable.py
2022-05-18T04:40:54.9628741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:40:54.9814164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:40:54.9933086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:40:54.9945777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:40:56.7972506Z ok (4.932s)
2022-05-18T04:40:56.7972736Z 
2022-05-18T04:40:56.7973200Z ----------------------------------------------------------------------
2022-05-18T04:40:56.7973555Z Ran 1 test in 4.932s
2022-05-18T04:40:56.7973838Z 
2022-05-18T04:40:56.7973934Z OK
2022-05-18T04:40:56.7974074Z 
2022-05-18T04:40:56.7974193Z Generating XML reports...
2022-05-18T04:40:56.8019488Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044051.xml
2022-05-18T04:40:58.0633151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_i4b9c2
2022-05-18T04:40:58.0633782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_i4b9c2/_remote_module_non_scriptable.py
2022-05-18T04:40:58.4951281Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:40:58.4963270Z 
2022-05-18T04:40:58.4963570Z Running tests...
2022-05-18T04:40:58.4964060Z ----------------------------------------------------------------------
2022-05-18T04:41:00.1644487Z   test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:00.2090875Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76282
2022-05-18T04:41:00.2213211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76283
2022-05-18T04:41:00.2340046Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76284
2022-05-18T04:41:00.2465311Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76285
2022-05-18T04:41:01.2195188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxqn1gm9
2022-05-18T04:41:01.2203408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxqn1gm9/_remote_module_non_scriptable.py
2022-05-18T04:41:01.2471184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5sogwwf7
2022-05-18T04:41:01.2471775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5sogwwf7/_remote_module_non_scriptable.py
2022-05-18T04:41:01.2650656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1121h8m
2022-05-18T04:41:01.2651261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1121h8m/_remote_module_non_scriptable.py
2022-05-18T04:41:01.2902841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthtk7w39
2022-05-18T04:41:01.2903463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthtk7w39/_remote_module_non_scriptable.py
2022-05-18T04:41:01.6371496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:41:01.6636203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:01.6682250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:01.6934598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:41:08.1662853Z ok (9.670s)
2022-05-18T04:41:08.1663124Z 
2022-05-18T04:41:08.1663560Z ----------------------------------------------------------------------
2022-05-18T04:41:08.1663916Z Ran 1 test in 9.670s
2022-05-18T04:41:08.1664088Z 
2022-05-18T04:41:08.1664165Z OK
2022-05-18T04:41:08.1664309Z 
2022-05-18T04:41:08.1664460Z Generating XML reports...
2022-05-18T04:41:08.1707899Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044058.xml
2022-05-18T04:41:09.3890045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyufy8_oy
2022-05-18T04:41:09.3890678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyufy8_oy/_remote_module_non_scriptable.py
2022-05-18T04:41:09.8068726Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:41:09.8078964Z 
2022-05-18T04:41:09.8079229Z Running tests...
2022-05-18T04:41:09.8079718Z ----------------------------------------------------------------------
2022-05-18T04:41:11.4224139Z   test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:11.4650813Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76621
2022-05-18T04:41:11.4766322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76622
2022-05-18T04:41:11.4885217Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76623
2022-05-18T04:41:11.5001305Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76624
2022-05-18T04:41:12.5048183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqfmmb7wj
2022-05-18T04:41:12.5048854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqfmmb7wj/_remote_module_non_scriptable.py
2022-05-18T04:41:12.5057166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbh11i970
2022-05-18T04:41:12.5058042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbh11i970/_remote_module_non_scriptable.py
2022-05-18T04:41:12.5106256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wv84pmy
2022-05-18T04:41:12.5107107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wv84pmy/_remote_module_non_scriptable.py
2022-05-18T04:41:12.5613313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_xg663o
2022-05-18T04:41:12.5613922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_xg663o/_remote_module_non_scriptable.py
2022-05-18T04:41:12.9070600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:12.9084969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:41:12.9184364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:12.9784882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:41:20.6227470Z ok (10.814s)
2022-05-18T04:41:20.6227745Z 
2022-05-18T04:41:20.6228192Z ----------------------------------------------------------------------
2022-05-18T04:41:20.6228525Z Ran 1 test in 10.815s
2022-05-18T04:41:20.6228693Z 
2022-05-18T04:41:20.6228793Z OK
2022-05-18T04:41:20.6228929Z 
2022-05-18T04:41:20.6229099Z Generating XML reports...
2022-05-18T04:41:20.6273259Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044109.xml
2022-05-18T04:41:21.8935146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa90w2q1d
2022-05-18T04:41:21.8935819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa90w2q1d/_remote_module_non_scriptable.py
2022-05-18T04:41:22.3217727Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:41:22.3229183Z 
2022-05-18T04:41:22.3229586Z Running tests...
2022-05-18T04:41:22.3230379Z ----------------------------------------------------------------------
2022-05-18T04:41:23.9769245Z   test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:24.0224564Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77136
2022-05-18T04:41:24.0351732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77137
2022-05-18T04:41:24.0465277Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77138
2022-05-18T04:41:24.0588201Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77139
2022-05-18T04:41:25.0430522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5nxyhph7
2022-05-18T04:41:25.0431175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5nxyhph7/_remote_module_non_scriptable.py
2022-05-18T04:41:25.0437809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpagzgar_e
2022-05-18T04:41:25.0438364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbff9z830
2022-05-18T04:41:25.0438940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpagzgar_e/_remote_module_non_scriptable.py
2022-05-18T04:41:25.0439769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbff9z830/_remote_module_non_scriptable.py
2022-05-18T04:41:25.0692599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpibw752og
2022-05-18T04:41:25.0693173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpibw752og/_remote_module_non_scriptable.py
2022-05-18T04:41:25.4474362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:25.4492191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:25.4498822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:41:25.4680799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:41:39.5953394Z ok (17.272s)
2022-05-18T04:41:39.5953635Z 
2022-05-18T04:41:39.5955265Z ----------------------------------------------------------------------
2022-05-18T04:41:39.5955745Z Ran 1 test in 17.273s
2022-05-18T04:41:39.5955927Z 
2022-05-18T04:41:39.5956034Z OK
2022-05-18T04:41:39.5956195Z 
2022-05-18T04:41:39.5956344Z Generating XML reports...
2022-05-18T04:41:39.6001059Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044122.xml
2022-05-18T04:41:40.8533069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7cchrrgg
2022-05-18T04:41:40.8533689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7cchrrgg/_remote_module_non_scriptable.py
2022-05-18T04:41:41.2857106Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:41:41.2868860Z 
2022-05-18T04:41:41.2869066Z Running tests...
2022-05-18T04:41:41.2869541Z ----------------------------------------------------------------------
2022-05-18T04:41:42.9712260Z   test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:43.0165863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77651
2022-05-18T04:41:43.0290651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77652
2022-05-18T04:41:43.0423136Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77653
2022-05-18T04:41:43.0541154Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77654
2022-05-18T04:41:44.0364023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd0urpan6
2022-05-18T04:41:44.0364699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd0urpan6/_remote_module_non_scriptable.py
2022-05-18T04:41:44.0478972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgo669mww
2022-05-18T04:41:44.0479562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgo669mww/_remote_module_non_scriptable.py
2022-05-18T04:41:44.0540528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnxp8lsnn
2022-05-18T04:41:44.0541131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnxp8lsnn/_remote_module_non_scriptable.py
2022-05-18T04:41:44.0555443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wge6pvu
2022-05-18T04:41:44.0556344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wge6pvu/_remote_module_non_scriptable.py
2022-05-18T04:41:44.4407505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:44.4524137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:41:44.4531905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:41:44.4585189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:53.3797291Z ok (12.092s)
2022-05-18T04:41:53.3797530Z 
2022-05-18T04:41:53.3798581Z ----------------------------------------------------------------------
2022-05-18T04:41:53.3799196Z Ran 1 test in 12.093s
2022-05-18T04:41:53.3799464Z 
2022-05-18T04:41:53.3799770Z OK
2022-05-18T04:41:53.3799952Z 
2022-05-18T04:41:53.3800097Z Generating XML reports...
2022-05-18T04:41:53.3844166Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044141.xml
2022-05-18T04:41:54.6607515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprimk82mh
2022-05-18T04:41:54.6608182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprimk82mh/_remote_module_non_scriptable.py
2022-05-18T04:41:55.0707962Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:41:55.0718690Z 
2022-05-18T04:41:55.0718977Z Running tests...
2022-05-18T04:41:55.0719453Z ----------------------------------------------------------------------
2022-05-18T04:41:56.6663747Z   test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:41:56.7091479Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78166
2022-05-18T04:41:56.7214656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78167
2022-05-18T04:41:56.7340725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 78168
2022-05-18T04:41:56.7463093Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 78169
2022-05-18T04:41:57.6682867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpror53e6a
2022-05-18T04:41:57.6683450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwk8w1tds
2022-05-18T04:41:57.6684019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpror53e6a/_remote_module_non_scriptable.py
2022-05-18T04:41:57.6684592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwk8w1tds/_remote_module_non_scriptable.py
2022-05-18T04:41:57.7076497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptcc3blyn
2022-05-18T04:41:57.7077087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptcc3blyn/_remote_module_non_scriptable.py
2022-05-18T04:41:57.7212653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfrm1sp3y
2022-05-18T04:41:57.7213235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfrm1sp3y/_remote_module_non_scriptable.py
2022-05-18T04:41:58.0760080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:41:58.0780056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:41:58.1208680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:41:58.1267546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:42:05.8695800Z ok (10.797s)
2022-05-18T04:42:05.8696122Z 
2022-05-18T04:42:05.8696583Z ----------------------------------------------------------------------
2022-05-18T04:42:05.8696945Z Ran 1 test in 10.798s
2022-05-18T04:42:05.8697121Z 
2022-05-18T04:42:05.8697230Z OK
2022-05-18T04:42:05.8697378Z 
2022-05-18T04:42:05.8697807Z Generating XML reports...
2022-05-18T04:42:05.8740757Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044155.xml
2022-05-18T04:42:07.1130177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsc48b0f
2022-05-18T04:42:07.5289967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsc48b0f/_remote_module_non_scriptable.py
2022-05-18T04:42:07.5290791Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:42:07.5301139Z 
2022-05-18T04:42:07.5301431Z Running tests...
2022-05-18T04:42:07.5302615Z ----------------------------------------------------------------------
2022-05-18T04:42:09.1482827Z   test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:09.1930386Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78676
2022-05-18T04:42:09.2063716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78677
2022-05-18T04:42:09.2190909Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 78678
2022-05-18T04:42:09.2308929Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 78679
2022-05-18T04:42:10.1482490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3wm148t
2022-05-18T04:42:10.1483125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3wm148t/_remote_module_non_scriptable.py
2022-05-18T04:42:10.1550090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiivxldmu
2022-05-18T04:42:10.1550664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiivxldmu/_remote_module_non_scriptable.py
2022-05-18T04:42:10.1695967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjv6e8jbj
2022-05-18T04:42:10.1696550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjv6e8jbj/_remote_module_non_scriptable.py
2022-05-18T04:42:10.2094650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_op1bru
2022-05-18T04:42:10.2095231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_op1bru/_remote_module_non_scriptable.py
2022-05-18T04:42:10.5582153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:10.5727023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:10.5762429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:42:10.6166455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:42:11.2377379Z ok (3.707s)
2022-05-18T04:42:11.2377635Z 
2022-05-18T04:42:11.2378044Z ----------------------------------------------------------------------
2022-05-18T04:42:11.2378399Z Ran 1 test in 3.708s
2022-05-18T04:42:11.2378589Z 
2022-05-18T04:42:11.2378695Z OK
2022-05-18T04:42:11.2378841Z 
2022-05-18T04:42:11.2378987Z Generating XML reports...
2022-05-18T04:42:11.2423324Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044207.xml
2022-05-18T04:42:12.5017315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp778u6nfq
2022-05-18T04:42:12.5017958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp778u6nfq/_remote_module_non_scriptable.py
2022-05-18T04:42:12.9163490Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:42:12.9174073Z 
2022-05-18T04:42:12.9174270Z Running tests...
2022-05-18T04:42:12.9174766Z ----------------------------------------------------------------------
2022-05-18T04:42:14.5370626Z   test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:14.5819090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79175
2022-05-18T04:42:14.5947218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79176
2022-05-18T04:42:14.6071940Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79177
2022-05-18T04:42:14.6183387Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79178
2022-05-18T04:42:15.5820005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_g0gly_
2022-05-18T04:42:15.5820880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_g0gly_/_remote_module_non_scriptable.py
2022-05-18T04:42:15.6287689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_o6jk3iw
2022-05-18T04:42:15.6288268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_o6jk3iw/_remote_module_non_scriptable.py
2022-05-18T04:42:15.6299940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3gccw_i
2022-05-18T04:42:15.6300554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3gccw_i/_remote_module_non_scriptable.py
2022-05-18T04:42:15.6545308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvceumne
2022-05-18T04:42:15.6545890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvceumne/_remote_module_non_scriptable.py
2022-05-18T04:42:15.9851440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:16.0311394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:42:16.0344438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:16.0565141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:42:19.4310041Z ok (6.513s)
2022-05-18T04:42:19.4310299Z 
2022-05-18T04:42:19.4310771Z ----------------------------------------------------------------------
2022-05-18T04:42:19.4311136Z Ran 1 test in 6.514s
2022-05-18T04:42:19.4311311Z 
2022-05-18T04:42:19.4311414Z OK
2022-05-18T04:42:19.4311564Z 
2022-05-18T04:42:19.4311718Z Generating XML reports...
2022-05-18T04:42:19.4354719Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044212.xml
2022-05-18T04:42:20.7039912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwqq9y0d
2022-05-18T04:42:20.7040554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwqq9y0d/_remote_module_non_scriptable.py
2022-05-18T04:42:21.1398985Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:42:21.1409039Z 
2022-05-18T04:42:21.1409341Z Running tests...
2022-05-18T04:42:21.1409805Z ----------------------------------------------------------------------
2022-05-18T04:42:22.8266513Z   test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:22.8763887Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79682
2022-05-18T04:42:22.8914689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79683
2022-05-18T04:42:22.9066925Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79684
2022-05-18T04:42:22.9204933Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79685
2022-05-18T04:42:23.8376202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2e44up1o
2022-05-18T04:42:23.8376814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2e44up1o/_remote_module_non_scriptable.py
2022-05-18T04:42:23.8525901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6f9lp7r
2022-05-18T04:42:23.8527404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6f9lp7r/_remote_module_non_scriptable.py
2022-05-18T04:42:23.8760903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88h1bqg8
2022-05-18T04:42:23.8761530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88h1bqg8/_remote_module_non_scriptable.py
2022-05-18T04:42:23.8973942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1dra398
2022-05-18T04:42:23.8974552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1dra398/_remote_module_non_scriptable.py
2022-05-18T04:42:24.2460011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:24.2630779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:24.2734462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:42:24.3085656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:42:27.6331564Z ok (6.492s)
2022-05-18T04:42:27.6331955Z 
2022-05-18T04:42:27.6332415Z ----------------------------------------------------------------------
2022-05-18T04:42:27.6332756Z Ran 1 test in 6.492s
2022-05-18T04:42:27.6332932Z 
2022-05-18T04:42:27.6333033Z OK
2022-05-18T04:42:27.6334420Z 
2022-05-18T04:42:27.6334605Z Generating XML reports...
2022-05-18T04:42:27.6378103Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044221.xml
2022-05-18T04:42:28.9351143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7xtif6e
2022-05-18T04:42:28.9351783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7xtif6e/_remote_module_non_scriptable.py
2022-05-18T04:42:29.3639373Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:42:29.3650399Z 
2022-05-18T04:42:29.3650606Z Running tests...
2022-05-18T04:42:29.3651504Z ----------------------------------------------------------------------
2022-05-18T04:42:31.0223532Z   test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:31.0668301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80189
2022-05-18T04:42:31.0797211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80190
2022-05-18T04:42:31.0935592Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80191
2022-05-18T04:42:31.1054503Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80192
2022-05-18T04:42:32.0524873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp094jfwq7
2022-05-18T04:42:32.0525507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp094jfwq7/_remote_module_non_scriptable.py
2022-05-18T04:42:32.0819113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9h_k9we
2022-05-18T04:42:32.0819731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9h_k9we/_remote_module_non_scriptable.py
2022-05-18T04:42:32.0900111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_f6ln4n
2022-05-18T04:42:32.0900698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_f6ln4n/_remote_module_non_scriptable.py
2022-05-18T04:42:32.0929224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4qkdkdw
2022-05-18T04:42:32.0929821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4qkdkdw/_remote_module_non_scriptable.py
2022-05-18T04:42:32.4526382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:32.4912902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:42:32.4976243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:32.5020149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:42:35.8186008Z ok (6.453s)
2022-05-18T04:42:35.8186234Z 
2022-05-18T04:42:35.8186676Z ----------------------------------------------------------------------
2022-05-18T04:42:35.8187042Z Ran 1 test in 6.454s
2022-05-18T04:42:35.8187191Z 
2022-05-18T04:42:35.8187291Z OK
2022-05-18T04:42:35.8187429Z 
2022-05-18T04:42:35.8187570Z Generating XML reports...
2022-05-18T04:42:35.8233066Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044229.xml
2022-05-18T04:42:37.0866584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qtxfn0y
2022-05-18T04:42:37.0867267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qtxfn0y/_remote_module_non_scriptable.py
2022-05-18T04:42:37.5073611Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:42:37.5083579Z 
2022-05-18T04:42:37.5083864Z Running tests...
2022-05-18T04:42:37.5084330Z ----------------------------------------------------------------------
2022-05-18T04:42:39.1206184Z   test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:39.1665637Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80692
2022-05-18T04:42:39.1801578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80693
2022-05-18T04:42:39.1940520Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80694
2022-05-18T04:42:39.2062754Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80695
2022-05-18T04:42:40.1327623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3uowuwc
2022-05-18T04:42:40.1349110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3uowuwc/_remote_module_non_scriptable.py
2022-05-18T04:42:40.1349705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpza7d8uaq
2022-05-18T04:42:40.1350271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpza7d8uaq/_remote_module_non_scriptable.py
2022-05-18T04:42:40.1368053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqr897me
2022-05-18T04:42:40.1368653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqr897me/_remote_module_non_scriptable.py
2022-05-18T04:42:40.1687932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz1d3g4_s
2022-05-18T04:42:40.1688546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz1d3g4_s/_remote_module_non_scriptable.py
2022-05-18T04:42:40.5370236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:42:40.5432606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:42:40.5484137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:40.5781794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:46.2238666Z ok (8.715s)
2022-05-18T04:42:46.2239172Z 
2022-05-18T04:42:46.2239936Z ----------------------------------------------------------------------
2022-05-18T04:42:46.2240577Z Ran 1 test in 8.716s
2022-05-18T04:42:46.2240883Z 
2022-05-18T04:42:46.2241133Z OK
2022-05-18T04:42:46.2241344Z 
2022-05-18T04:42:46.2241602Z Generating XML reports...
2022-05-18T04:42:46.2288360Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044237.xml
2022-05-18T04:42:47.5236872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppnkhvw42
2022-05-18T04:42:47.9437757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppnkhvw42/_remote_module_non_scriptable.py
2022-05-18T04:42:47.9439057Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:42:47.9447770Z 
2022-05-18T04:42:47.9448132Z Running tests...
2022-05-18T04:42:47.9448625Z ----------------------------------------------------------------------
2022-05-18T04:42:49.5761032Z   test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:42:49.6223638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81207
2022-05-18T04:42:49.6357138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81208
2022-05-18T04:42:49.6492039Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81209
2022-05-18T04:42:49.6612460Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81210
2022-05-18T04:42:50.5696322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8w0doxr
2022-05-18T04:42:50.5696976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8w0doxr/_remote_module_non_scriptable.py
2022-05-18T04:42:50.6285553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mlpdpao
2022-05-18T04:42:50.6286169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mlpdpao/_remote_module_non_scriptable.py
2022-05-18T04:42:50.6327872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfu66wgm6
2022-05-18T04:42:50.6328463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfu66wgm6/_remote_module_non_scriptable.py
2022-05-18T04:42:50.6418172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfplt0nku
2022-05-18T04:42:50.6418780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfplt0nku/_remote_module_non_scriptable.py
2022-05-18T04:42:50.9829760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:42:51.0307176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:42:51.0409573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:42:51.0435729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:42:56.7796191Z ok (8.834s)
2022-05-18T04:42:56.7796660Z 
2022-05-18T04:42:56.7797153Z ----------------------------------------------------------------------
2022-05-18T04:42:56.7797521Z Ran 1 test in 8.835s
2022-05-18T04:42:56.7797709Z 
2022-05-18T04:42:56.7797809Z OK
2022-05-18T04:42:56.7800135Z 
2022-05-18T04:42:56.7800729Z Generating XML reports...
2022-05-18T04:42:56.7842309Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044247.xml
2022-05-18T04:42:58.0300117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxws99vw
2022-05-18T04:42:58.0300762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxws99vw/_remote_module_non_scriptable.py
2022-05-18T04:42:58.4643021Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:42:58.4653902Z 
2022-05-18T04:42:58.4654093Z Running tests...
2022-05-18T04:42:58.4654581Z ----------------------------------------------------------------------
2022-05-18T04:43:00.1263025Z   test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:00.1718142Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81714
2022-05-18T04:43:00.1851067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81715
2022-05-18T04:43:00.1985079Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81716
2022-05-18T04:43:00.2106335Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81717
2022-05-18T04:43:01.1358618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpalon4avm
2022-05-18T04:43:01.1359772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpalon4avm/_remote_module_non_scriptable.py
2022-05-18T04:43:01.1383170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvs8ewqrw
2022-05-18T04:43:01.1384217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvs8ewqrw/_remote_module_non_scriptable.py
2022-05-18T04:43:01.1778291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqviexqq
2022-05-18T04:43:01.1779741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqviexqq/_remote_module_non_scriptable.py
2022-05-18T04:43:01.2274443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpjp2lbc
2022-05-18T04:43:01.2279026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpjp2lbc/_remote_module_non_scriptable.py
2022-05-18T04:43:01.5472471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:01.5580393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:01.5852931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:43:01.6583701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:43:07.2293727Z ok (8.764s)
2022-05-18T04:43:07.2293996Z 
2022-05-18T04:43:07.2294429Z ----------------------------------------------------------------------
2022-05-18T04:43:07.2294810Z Ran 1 test in 8.764s
2022-05-18T04:43:07.2294983Z 
2022-05-18T04:43:07.2295078Z OK
2022-05-18T04:43:07.2295215Z 
2022-05-18T04:43:07.2295333Z Generating XML reports...
2022-05-18T04:43:07.2339752Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044258.xml
2022-05-18T04:43:08.5103458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ozuq2r_
2022-05-18T04:43:08.5104063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ozuq2r_/_remote_module_non_scriptable.py
2022-05-18T04:43:08.9468176Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:43:08.9478794Z 
2022-05-18T04:43:08.9479139Z Running tests...
2022-05-18T04:43:08.9479620Z ----------------------------------------------------------------------
2022-05-18T04:43:10.6042335Z   test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:10.6487289Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82221
2022-05-18T04:43:10.6619660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82222
2022-05-18T04:43:10.6752776Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82223
2022-05-18T04:43:10.6886442Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82224
2022-05-18T04:43:11.6497589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp36_wm7gf
2022-05-18T04:43:11.6498622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp36_wm7gf/_remote_module_non_scriptable.py
2022-05-18T04:43:11.6995307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpduqcdgvl
2022-05-18T04:43:11.6996466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpduqcdgvl/_remote_module_non_scriptable.py
2022-05-18T04:43:11.7167578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvy9q3yz
2022-05-18T04:43:11.7168249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvy9q3yz/_remote_module_non_scriptable.py
2022-05-18T04:43:11.7300950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtr6t0c4
2022-05-18T04:43:11.7302250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtr6t0c4/_remote_module_non_scriptable.py
2022-05-18T04:43:12.0528581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:12.1054089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:12.1245970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:43:12.1372936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:43:17.8069326Z ok (8.859s)
2022-05-18T04:43:17.8069599Z 
2022-05-18T04:43:17.8070364Z ----------------------------------------------------------------------
2022-05-18T04:43:17.8070750Z Ran 1 test in 8.859s
2022-05-18T04:43:17.8070927Z 
2022-05-18T04:43:17.8073331Z OK
2022-05-18T04:43:17.8073539Z 
2022-05-18T04:43:17.8073699Z Generating XML reports...
2022-05-18T04:43:17.8114343Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044308.xml
2022-05-18T04:43:19.0526409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvyd13bd0
2022-05-18T04:43:19.0527808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvyd13bd0/_remote_module_non_scriptable.py
2022-05-18T04:43:19.4887298Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:43:19.4898027Z 
2022-05-18T04:43:19.4898442Z Running tests...
2022-05-18T04:43:19.4899446Z ----------------------------------------------------------------------
2022-05-18T04:43:21.1696748Z   test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:21.2172812Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82728
2022-05-18T04:43:21.2309406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82729
2022-05-18T04:43:21.2443487Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82730
2022-05-18T04:43:21.2570101Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82731
2022-05-18T04:43:22.2596584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6ahfz4m
2022-05-18T04:43:22.2597280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6ahfz4m/_remote_module_non_scriptable.py
2022-05-18T04:43:22.2606879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdxtfzay
2022-05-18T04:43:22.2607497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdxtfzay/_remote_module_non_scriptable.py
2022-05-18T04:43:22.2654255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ed7s7dh
2022-05-18T04:43:22.2654853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ed7s7dh/_remote_module_non_scriptable.py
2022-05-18T04:43:22.2681626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwmfh15k4
2022-05-18T04:43:22.2682215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwmfh15k4/_remote_module_non_scriptable.py
2022-05-18T04:43:22.6634200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:43:22.6640931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:43:22.6700412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:22.6751325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:28.2763157Z ok (8.786s)
2022-05-18T04:43:28.2763478Z 
2022-05-18T04:43:28.2763954Z ----------------------------------------------------------------------
2022-05-18T04:43:28.2764320Z Ran 1 test in 8.786s
2022-05-18T04:43:28.2764496Z 
2022-05-18T04:43:28.2764600Z OK
2022-05-18T04:43:28.2764721Z 
2022-05-18T04:43:28.2765260Z Generating XML reports...
2022-05-18T04:43:28.2807802Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044319.xml
2022-05-18T04:43:29.5496051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppt6ga81q
2022-05-18T04:43:29.5496685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppt6ga81q/_remote_module_non_scriptable.py
2022-05-18T04:43:29.9811133Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:43:29.9821366Z 
2022-05-18T04:43:29.9821658Z Running tests...
2022-05-18T04:43:29.9822807Z ----------------------------------------------------------------------
2022-05-18T04:43:31.6436223Z   test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:31.6905922Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83235
2022-05-18T04:43:31.7039700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83236
2022-05-18T04:43:31.7174730Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83237
2022-05-18T04:43:31.7298636Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83238
2022-05-18T04:43:32.7293468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3tgmswcs
2022-05-18T04:43:32.7294094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3tgmswcs/_remote_module_non_scriptable.py
2022-05-18T04:43:32.7304960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa62x5f5d
2022-05-18T04:43:32.7305955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa62x5f5d/_remote_module_non_scriptable.py
2022-05-18T04:43:32.7442482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmy8q_md
2022-05-18T04:43:32.7443092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmy8q_md/_remote_module_non_scriptable.py
2022-05-18T04:43:32.7485442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4h27zbi
2022-05-18T04:43:32.7486013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4h27zbi/_remote_module_non_scriptable.py
2022-05-18T04:43:33.1334920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:43:33.1415464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:33.1493246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:43:33.1573785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:38.8485637Z ok (8.866s)
2022-05-18T04:43:38.8485996Z 
2022-05-18T04:43:38.8486905Z ----------------------------------------------------------------------
2022-05-18T04:43:38.8487348Z Ran 1 test in 8.866s
2022-05-18T04:43:38.8487536Z 
2022-05-18T04:43:38.8487630Z OK
2022-05-18T04:43:38.8487769Z 
2022-05-18T04:43:38.8487887Z Generating XML reports...
2022-05-18T04:43:38.8531299Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044329.xml
2022-05-18T04:43:40.1237190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbiyqxswa
2022-05-18T04:43:40.1237798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbiyqxswa/_remote_module_non_scriptable.py
2022-05-18T04:43:40.5574470Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:43:40.5585333Z 
2022-05-18T04:43:40.5585804Z Running tests...
2022-05-18T04:43:40.5586306Z ----------------------------------------------------------------------
2022-05-18T04:43:42.2427966Z   test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:42.2892374Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83750
2022-05-18T04:43:42.3028380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83751
2022-05-18T04:43:42.3168192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83752
2022-05-18T04:43:42.3291180Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83753
2022-05-18T04:43:43.2531398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3xd_wps
2022-05-18T04:43:43.2532299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3xd_wps/_remote_module_non_scriptable.py
2022-05-18T04:43:43.2570487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpva1z5gg9
2022-05-18T04:43:43.2571305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpva1z5gg9/_remote_module_non_scriptable.py
2022-05-18T04:43:43.2897564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3it611o2
2022-05-18T04:43:43.2898279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3it611o2/_remote_module_non_scriptable.py
2022-05-18T04:43:43.2915223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rgbhxm3
2022-05-18T04:43:43.2915936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rgbhxm3/_remote_module_non_scriptable.py
2022-05-18T04:43:43.6529991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:43.6710505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:43.6937966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:43:43.7043335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:43:49.3484094Z ok (8.789s)
2022-05-18T04:43:49.3484331Z 
2022-05-18T04:43:49.3484795Z ----------------------------------------------------------------------
2022-05-18T04:43:49.3485143Z Ran 1 test in 8.790s
2022-05-18T04:43:49.3485291Z 
2022-05-18T04:43:49.3485385Z OK
2022-05-18T04:43:49.3485526Z 
2022-05-18T04:43:49.3485659Z Generating XML reports...
2022-05-18T04:43:49.3530311Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044340.xml
2022-05-18T04:43:50.5810504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpklidn29j
2022-05-18T04:43:50.5811132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpklidn29j/_remote_module_non_scriptable.py
2022-05-18T04:43:51.0095719Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:43:51.0106751Z 
2022-05-18T04:43:51.0107058Z Running tests...
2022-05-18T04:43:51.0107523Z ----------------------------------------------------------------------
2022-05-18T04:43:52.6932044Z   test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:43:52.7408686Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84265
2022-05-18T04:43:52.7542574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84266
2022-05-18T04:43:52.7676337Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84267
2022-05-18T04:43:52.7799129Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84268
2022-05-18T04:43:53.8264112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gnd1xxi
2022-05-18T04:43:53.8264757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gnd1xxi/_remote_module_non_scriptable.py
2022-05-18T04:43:53.8428818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjr0lqff
2022-05-18T04:43:53.8429417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjr0lqff/_remote_module_non_scriptable.py
2022-05-18T04:43:53.8485402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4zrcf65
2022-05-18T04:43:53.8486021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4zrcf65/_remote_module_non_scriptable.py
2022-05-18T04:43:53.8503103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjp5bla2
2022-05-18T04:43:53.8504478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjp5bla2/_remote_module_non_scriptable.py
2022-05-18T04:43:54.2325216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:43:54.2466433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:43:54.2601801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:43:54.2634187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:43:59.7991382Z ok (8.788s)
2022-05-18T04:43:59.7991638Z 
2022-05-18T04:43:59.7992082Z ----------------------------------------------------------------------
2022-05-18T04:43:59.7992432Z Ran 1 test in 8.788s
2022-05-18T04:43:59.7992599Z 
2022-05-18T04:43:59.7992686Z OK
2022-05-18T04:43:59.7992824Z 
2022-05-18T04:43:59.7992970Z Generating XML reports...
2022-05-18T04:43:59.8036360Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044351.xml
2022-05-18T04:44:01.0713255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqfjf5di3
2022-05-18T04:44:01.0713922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqfjf5di3/_remote_module_non_scriptable.py
2022-05-18T04:44:01.5030784Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:44:01.5040858Z 
2022-05-18T04:44:01.5041383Z Running tests...
2022-05-18T04:44:01.5041916Z ----------------------------------------------------------------------
2022-05-18T04:44:03.1746370Z   test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:03.2199617Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84780
2022-05-18T04:44:03.2330536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84781
2022-05-18T04:44:03.2469431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84782
2022-05-18T04:44:03.2606078Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84783
2022-05-18T04:44:04.2552211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpha9yim6f
2022-05-18T04:44:04.2552846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpha9yim6f/_remote_module_non_scriptable.py
2022-05-18T04:44:04.2736713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgg467dv
2022-05-18T04:44:04.2737331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgg467dv/_remote_module_non_scriptable.py
2022-05-18T04:44:04.2750382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35q04pr3
2022-05-18T04:44:04.2750982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35q04pr3/_remote_module_non_scriptable.py
2022-05-18T04:44:04.3265750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppisjy076
2022-05-18T04:44:04.3266350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppisjy076/_remote_module_non_scriptable.py
2022-05-18T04:44:04.6564154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:04.6806830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:44:04.6810604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:04.7328673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:44:10.3794056Z ok (8.875s)
2022-05-18T04:44:10.3794295Z 
2022-05-18T04:44:10.3794735Z ----------------------------------------------------------------------
2022-05-18T04:44:10.3795087Z Ran 1 test in 8.875s
2022-05-18T04:44:10.3795255Z 
2022-05-18T04:44:10.3796852Z OK
2022-05-18T04:44:10.3796985Z 
2022-05-18T04:44:10.3797126Z Generating XML reports...
2022-05-18T04:44:10.3840636Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044401.xml
2022-05-18T04:44:11.6507628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgf15n3_l
2022-05-18T04:44:11.6508288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgf15n3_l/_remote_module_non_scriptable.py
2022-05-18T04:44:12.0806058Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:44:12.0818675Z 
2022-05-18T04:44:12.0819019Z Running tests...
2022-05-18T04:44:12.0819510Z ----------------------------------------------------------------------
2022-05-18T04:44:13.7441323Z   test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:13.7916687Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85295
2022-05-18T04:44:13.8053925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85296
2022-05-18T04:44:13.8198259Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85297
2022-05-18T04:44:13.8336286Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85298
2022-05-18T04:44:14.8224675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4sc549rs
2022-05-18T04:44:14.8225889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4sc549rs/_remote_module_non_scriptable.py
2022-05-18T04:44:14.8258354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpanq41pdt
2022-05-18T04:44:14.8259525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpanq41pdt/_remote_module_non_scriptable.py
2022-05-18T04:44:14.8653972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpovbtq044
2022-05-18T04:44:14.8655125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpovbtq044/_remote_module_non_scriptable.py
2022-05-18T04:44:14.8677905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp316k96ef
2022-05-18T04:44:14.8679059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp316k96ef/_remote_module_non_scriptable.py
2022-05-18T04:44:15.2246801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:15.2464825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:15.2683256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:44:15.2743349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:44:20.9520438Z ok (8.870s)
2022-05-18T04:44:20.9520680Z 
2022-05-18T04:44:20.9521138Z ----------------------------------------------------------------------
2022-05-18T04:44:20.9521474Z Ran 1 test in 8.870s
2022-05-18T04:44:20.9521642Z 
2022-05-18T04:44:20.9521743Z OK
2022-05-18T04:44:20.9522427Z 
2022-05-18T04:44:20.9522663Z Generating XML reports...
2022-05-18T04:44:20.9567711Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044412.xml
2022-05-18T04:44:22.2546306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpev3w_g5z
2022-05-18T04:44:22.2546933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpev3w_g5z/_remote_module_non_scriptable.py
2022-05-18T04:44:22.6713749Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:44:22.6724539Z 
2022-05-18T04:44:22.6724958Z Running tests...
2022-05-18T04:44:22.6725925Z ----------------------------------------------------------------------
2022-05-18T04:44:24.3111869Z   test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:24.3577012Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85802
2022-05-18T04:44:24.3716403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85803
2022-05-18T04:44:24.3861018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85804
2022-05-18T04:44:24.3998917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85805
2022-05-18T04:44:25.3517617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw564wqgw
2022-05-18T04:44:25.3518233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw564wqgw/_remote_module_non_scriptable.py
2022-05-18T04:44:25.4038127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7857_4ng
2022-05-18T04:44:25.4038716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7857_4ng/_remote_module_non_scriptable.py
2022-05-18T04:44:25.4181849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgd7let8
2022-05-18T04:44:25.4182719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgd7let8/_remote_module_non_scriptable.py
2022-05-18T04:44:25.4913473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9b25g3c4
2022-05-18T04:44:25.4914066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9b25g3c4/_remote_module_non_scriptable.py
2022-05-18T04:44:25.7637909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:25.8089285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:44:25.8190112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:25.9112576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:44:31.5193101Z ok (8.847s)
2022-05-18T04:44:31.5193364Z 
2022-05-18T04:44:31.5193812Z ----------------------------------------------------------------------
2022-05-18T04:44:31.5194184Z Ran 1 test in 8.847s
2022-05-18T04:44:31.5194358Z 
2022-05-18T04:44:31.5194460Z OK
2022-05-18T04:44:31.5194578Z 
2022-05-18T04:44:31.5194741Z Generating XML reports...
2022-05-18T04:44:31.5242060Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044422.xml
2022-05-18T04:44:32.8056558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7pl57ac
2022-05-18T04:44:32.8057215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7pl57ac/_remote_module_non_scriptable.py
2022-05-18T04:44:33.2206067Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:44:33.2216494Z 
2022-05-18T04:44:33.2216916Z Running tests...
2022-05-18T04:44:33.2217378Z ----------------------------------------------------------------------
2022-05-18T04:44:34.8427732Z   test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:34.8888632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86309
2022-05-18T04:44:34.9017365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86310
2022-05-18T04:44:34.9162452Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86311
2022-05-18T04:44:34.9284829Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86312
2022-05-18T04:44:35.8520689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mu5xggd
2022-05-18T04:44:35.8521324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mu5xggd/_remote_module_non_scriptable.py
2022-05-18T04:44:35.8736717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_hqrm3x
2022-05-18T04:44:35.8738155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_hqrm3x/_remote_module_non_scriptable.py
2022-05-18T04:44:35.9082906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0k54rz98
2022-05-18T04:44:35.9083817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0k54rz98/_remote_module_non_scriptable.py
2022-05-18T04:44:35.9095084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptq50rhos
2022-05-18T04:44:35.9095673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptq50rhos/_remote_module_non_scriptable.py
2022-05-18T04:44:36.2645016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:36.2747142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:36.3072085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:44:36.3219905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:44:41.8472340Z ok (8.625s)
2022-05-18T04:44:41.8472586Z 
2022-05-18T04:44:41.8473026Z ----------------------------------------------------------------------
2022-05-18T04:44:41.8473382Z Ran 1 test in 8.626s
2022-05-18T04:44:41.8473567Z 
2022-05-18T04:44:41.8473662Z OK
2022-05-18T04:44:41.8473805Z 
2022-05-18T04:44:41.8473944Z Generating XML reports...
2022-05-18T04:44:41.8517450Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044433.xml
2022-05-18T04:44:43.1263992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhkm86nx
2022-05-18T04:44:43.1264635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhkm86nx/_remote_module_non_scriptable.py
2022-05-18T04:44:43.5520281Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:44:43.5530777Z 
2022-05-18T04:44:43.5530942Z Running tests...
2022-05-18T04:44:45.2117964Z ----------------------------------------------------------------------
2022-05-18T04:44:45.2118632Z   test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:45.2580425Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86816
2022-05-18T04:44:45.2714002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86817
2022-05-18T04:44:45.2852899Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86818
2022-05-18T04:44:45.2984791Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86819
2022-05-18T04:44:46.2472769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vl53hjs
2022-05-18T04:44:46.2473428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vl53hjs/_remote_module_non_scriptable.py
2022-05-18T04:44:46.2845419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk5xsqs20
2022-05-18T04:44:46.2846075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk5xsqs20/_remote_module_non_scriptable.py
2022-05-18T04:44:46.2980300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvz3lqji
2022-05-18T04:44:46.2980907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvz3lqji/_remote_module_non_scriptable.py
2022-05-18T04:44:46.3556853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwi0o8khi
2022-05-18T04:44:46.3557707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwi0o8khi/_remote_module_non_scriptable.py
2022-05-18T04:44:46.6555052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:44:46.6859253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:46.7148350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:46.7602911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:44:52.4175118Z ok (8.864s)
2022-05-18T04:44:52.4175364Z 
2022-05-18T04:44:52.4177131Z ----------------------------------------------------------------------
2022-05-18T04:44:52.4177650Z Ran 1 test in 8.864s
2022-05-18T04:44:52.4177829Z 
2022-05-18T04:44:52.4177929Z OK
2022-05-18T04:44:52.4178398Z 
2022-05-18T04:44:52.4178861Z Generating XML reports...
2022-05-18T04:44:52.4219619Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044443.xml
2022-05-18T04:44:53.6297832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1783jdo2
2022-05-18T04:44:53.6298988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1783jdo2/_remote_module_non_scriptable.py
2022-05-18T04:44:54.0456329Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:44:54.0468696Z 
2022-05-18T04:44:54.0469108Z Running tests...
2022-05-18T04:44:54.0469628Z ----------------------------------------------------------------------
2022-05-18T04:44:55.6821298Z   test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:44:55.7280069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87323
2022-05-18T04:44:55.7409646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87324
2022-05-18T04:44:55.7541895Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87325
2022-05-18T04:44:55.7661357Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87326
2022-05-18T04:44:56.7227039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2w7pvni
2022-05-18T04:44:56.7227658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2w7pvni/_remote_module_non_scriptable.py
2022-05-18T04:44:56.7500342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mspcq36
2022-05-18T04:44:56.7501172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mspcq36/_remote_module_non_scriptable.py
2022-05-18T04:44:56.7990187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlauezun
2022-05-18T04:44:56.7990789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlauezun/_remote_module_non_scriptable.py
2022-05-18T04:44:56.8052951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjekupgo2
2022-05-18T04:44:56.8053533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjekupgo2/_remote_module_non_scriptable.py
2022-05-18T04:44:57.1246654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:44:57.1560395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:44:57.2015797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:44:57.2026053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:45:02.7848875Z ok (8.738s)
2022-05-18T04:45:02.7849120Z 
2022-05-18T04:45:02.7849555Z ----------------------------------------------------------------------
2022-05-18T04:45:02.7849909Z Ran 1 test in 8.738s
2022-05-18T04:45:02.7850077Z 
2022-05-18T04:45:02.7850176Z OK
2022-05-18T04:45:02.7850314Z 
2022-05-18T04:45:02.7850744Z Generating XML reports...
2022-05-18T04:45:02.7900675Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044454.xml
2022-05-18T04:45:04.0292308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22xmpyce
2022-05-18T04:45:04.0292956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22xmpyce/_remote_module_non_scriptable.py
2022-05-18T04:45:04.4440421Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:45:04.4451587Z 
2022-05-18T04:45:04.4451848Z Running tests...
2022-05-18T04:45:04.4452968Z ----------------------------------------------------------------------
2022-05-18T04:45:06.0786099Z   test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:06.1260234Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87830
2022-05-18T04:45:06.1385512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87831
2022-05-18T04:45:06.1521581Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87832
2022-05-18T04:45:06.1668620Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87833
2022-05-18T04:45:07.0707996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpopwp7zmi
2022-05-18T04:45:07.0708624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpopwp7zmi/_remote_module_non_scriptable.py
2022-05-18T04:45:07.0931960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ufdvqql
2022-05-18T04:45:07.0932555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ufdvqql/_remote_module_non_scriptable.py
2022-05-18T04:45:07.1096771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozu2j6uf
2022-05-18T04:45:07.1097367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozu2j6uf/_remote_module_non_scriptable.py
2022-05-18T04:45:07.1220995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ymrnnq2
2022-05-18T04:45:07.1221602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ymrnnq2/_remote_module_non_scriptable.py
2022-05-18T04:45:07.4877051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:07.4929570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:45:07.5178618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:07.5242173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:45:13.0843652Z ok (8.639s)
2022-05-18T04:45:13.0843921Z 
2022-05-18T04:45:13.0844347Z ----------------------------------------------------------------------
2022-05-18T04:45:13.0844704Z Ran 1 test in 8.639s
2022-05-18T04:45:13.0844907Z 
2022-05-18T04:45:13.0845003Z OK
2022-05-18T04:45:13.0847295Z 
2022-05-18T04:45:13.0847672Z Generating XML reports...
2022-05-18T04:45:13.0890159Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044504.xml
2022-05-18T04:45:14.3348330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuk159e_
2022-05-18T04:45:14.3348983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuk159e_/_remote_module_non_scriptable.py
2022-05-18T04:45:14.7616320Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:45:14.7627054Z 
2022-05-18T04:45:14.7627400Z Running tests...
2022-05-18T04:45:14.7627868Z ----------------------------------------------------------------------
2022-05-18T04:45:16.4355350Z   test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:16.4825062Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88337
2022-05-18T04:45:16.4957190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88338
2022-05-18T04:45:16.5091762Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88339
2022-05-18T04:45:16.5224190Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88340
2022-05-18T04:45:17.4993355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps5x_3zt3
2022-05-18T04:45:17.4994257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps5x_3zt3/_remote_module_non_scriptable.py
2022-05-18T04:45:17.5105155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5as8_78
2022-05-18T04:45:17.5105756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5as8_78/_remote_module_non_scriptable.py
2022-05-18T04:45:17.5396922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmggwntlz
2022-05-18T04:45:17.5397543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmggwntlz/_remote_module_non_scriptable.py
2022-05-18T04:45:17.5959456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcljr647r
2022-05-18T04:45:17.5960030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcljr647r/_remote_module_non_scriptable.py
2022-05-18T04:45:17.9012136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:45:17.9164150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:17.9445681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:17.9987896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:45:23.6399845Z ok (8.877s)
2022-05-18T04:45:23.6400128Z 
2022-05-18T04:45:23.6400602Z ----------------------------------------------------------------------
2022-05-18T04:45:23.6400954Z Ran 1 test in 8.877s
2022-05-18T04:45:23.6401103Z 
2022-05-18T04:45:23.6401200Z OK
2022-05-18T04:45:23.6401337Z 
2022-05-18T04:45:23.6401481Z Generating XML reports...
2022-05-18T04:45:23.6445323Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044514.xml
2022-05-18T04:45:24.9096372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnqm9j2h
2022-05-18T04:45:24.9097048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnqm9j2h/_remote_module_non_scriptable.py
2022-05-18T04:45:25.3210226Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:45:25.3221080Z 
2022-05-18T04:45:25.3221369Z Running tests...
2022-05-18T04:45:25.3222171Z ----------------------------------------------------------------------
2022-05-18T04:45:26.9409367Z   test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:26.9876037Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88844
2022-05-18T04:45:27.0010847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88845
2022-05-18T04:45:27.0156613Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88846
2022-05-18T04:45:27.0281226Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88847
2022-05-18T04:45:27.9358705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpglflcof8
2022-05-18T04:45:27.9359334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpglflcof8/_remote_module_non_scriptable.py
2022-05-18T04:45:27.9410649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq50k5scj
2022-05-18T04:45:27.9411240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq50k5scj/_remote_module_non_scriptable.py
2022-05-18T04:45:27.9485904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6hlw8an
2022-05-18T04:45:27.9486473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6hlw8an/_remote_module_non_scriptable.py
2022-05-18T04:45:27.9945686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zthziqo
2022-05-18T04:45:27.9946280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zthziqo/_remote_module_non_scriptable.py
2022-05-18T04:45:28.3459210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:28.3532055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:28.3532600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:45:28.3964082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:45:34.0468736Z ok (8.724s)
2022-05-18T04:45:34.0469205Z 
2022-05-18T04:45:34.0469957Z ----------------------------------------------------------------------
2022-05-18T04:45:34.0470289Z Ran 1 test in 8.725s
2022-05-18T04:45:34.0470459Z 
2022-05-18T04:45:34.0470555Z OK
2022-05-18T04:45:34.0470693Z 
2022-05-18T04:45:34.0470834Z Generating XML reports...
2022-05-18T04:45:34.0514991Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044525.xml
2022-05-18T04:45:35.3225294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jd3izfu
2022-05-18T04:45:35.3225931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jd3izfu/_remote_module_non_scriptable.py
2022-05-18T04:45:35.7506268Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:45:35.7518093Z 
2022-05-18T04:45:35.7518421Z Running tests...
2022-05-18T04:45:35.7518916Z ----------------------------------------------------------------------
2022-05-18T04:45:37.4273084Z   test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:37.4744154Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89351
2022-05-18T04:45:37.4880338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89352
2022-05-18T04:45:37.5021351Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89353
2022-05-18T04:45:37.5144747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89354
2022-05-18T04:45:38.4843037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzrim5ebx
2022-05-18T04:45:38.4844249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzrim5ebx/_remote_module_non_scriptable.py
2022-05-18T04:45:38.5126274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__cugczq
2022-05-18T04:45:38.5127382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__cugczq/_remote_module_non_scriptable.py
2022-05-18T04:45:38.5143576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8c01ul6e
2022-05-18T04:45:38.5145778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8c01ul6e/_remote_module_non_scriptable.py
2022-05-18T04:45:38.5326698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2ft5qfh
2022-05-18T04:45:38.5327819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2ft5qfh/_remote_module_non_scriptable.py
2022-05-18T04:45:38.8928185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:45:38.9205754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:45:38.9301579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:38.9529694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:42.3280820Z ok (6.576s)
2022-05-18T04:45:42.3281059Z 
2022-05-18T04:45:42.3281500Z ----------------------------------------------------------------------
2022-05-18T04:45:42.3281855Z Ran 1 test in 6.576s
2022-05-18T04:45:42.3282030Z 
2022-05-18T04:45:42.3282137Z OK
2022-05-18T04:45:42.3282281Z 
2022-05-18T04:45:42.3282424Z Generating XML reports...
2022-05-18T04:45:42.3326844Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044535.xml
2022-05-18T04:45:43.5659314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpru3xltsi
2022-05-18T04:45:43.5659954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpru3xltsi/_remote_module_non_scriptable.py
2022-05-18T04:45:43.9991034Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:45:44.0001124Z 
2022-05-18T04:45:44.0001472Z Running tests...
2022-05-18T04:45:44.0002094Z ----------------------------------------------------------------------
2022-05-18T04:45:45.6670425Z   test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:45.7128430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89854
2022-05-18T04:45:45.7263751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89855
2022-05-18T04:45:45.7399359Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89856
2022-05-18T04:45:45.7521467Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89857
2022-05-18T04:45:46.6946700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpne4_5yvf
2022-05-18T04:45:46.6947324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpne4_5yvf/_remote_module_non_scriptable.py
2022-05-18T04:45:46.7245963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa37j9zi3
2022-05-18T04:45:46.7246578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa37j9zi3/_remote_module_non_scriptable.py
2022-05-18T04:45:46.7306345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyay92k9
2022-05-18T04:45:46.7307017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyay92k9/_remote_module_non_scriptable.py
2022-05-18T04:45:46.7425143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxpnd_tu
2022-05-18T04:45:46.7427163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxpnd_tu/_remote_module_non_scriptable.py
2022-05-18T04:45:47.1128264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:45:47.1374149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:45:47.1432030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:45:47.1592648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:52.7712056Z ok (8.771s)
2022-05-18T04:45:52.7712313Z 
2022-05-18T04:45:52.7712758Z ----------------------------------------------------------------------
2022-05-18T04:45:52.7713108Z Ran 1 test in 8.771s
2022-05-18T04:45:52.7713278Z 
2022-05-18T04:45:52.7713375Z OK
2022-05-18T04:45:52.7713513Z 
2022-05-18T04:45:52.7713642Z Generating XML reports...
2022-05-18T04:45:52.7757595Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044543.xml
2022-05-18T04:45:54.0095424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5gsm0tn
2022-05-18T04:45:54.0096057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5gsm0tn/_remote_module_non_scriptable.py
2022-05-18T04:45:54.4388226Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:45:54.4398776Z 
2022-05-18T04:45:54.4399194Z Running tests...
2022-05-18T04:45:54.4399685Z ----------------------------------------------------------------------
2022-05-18T04:45:56.1290257Z   test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:45:56.1765133Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90369
2022-05-18T04:45:56.1902765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90370
2022-05-18T04:45:56.2039685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90371
2022-05-18T04:45:56.2166809Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90372
2022-05-18T04:45:57.1908479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzs76xxg8
2022-05-18T04:45:57.1909800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzs76xxg8/_remote_module_non_scriptable.py
2022-05-18T04:45:57.1944019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_t7u6t3
2022-05-18T04:45:57.1945084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_t7u6t3/_remote_module_non_scriptable.py
2022-05-18T04:45:57.1978548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4f3dpmni
2022-05-18T04:45:57.1979649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4f3dpmni/_remote_module_non_scriptable.py
2022-05-18T04:45:57.2006553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3gyfpzl
2022-05-18T04:45:57.2007709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3gyfpzl/_remote_module_non_scriptable.py
2022-05-18T04:45:57.5926309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:45:57.6050912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:45:57.6051428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:45:57.6051917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:01.0296793Z ok (6.589s)
2022-05-18T04:46:01.0297030Z 
2022-05-18T04:46:01.0297768Z ----------------------------------------------------------------------
2022-05-18T04:46:01.0298166Z Ran 1 test in 6.590s
2022-05-18T04:46:01.0298338Z 
2022-05-18T04:46:01.0298439Z OK
2022-05-18T04:46:01.0298595Z 
2022-05-18T04:46:01.0298755Z Generating XML reports...
2022-05-18T04:46:01.0342583Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044554.xml
2022-05-18T04:46:02.3329622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0idone_
2022-05-18T04:46:02.3330562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0idone_/_remote_module_non_scriptable.py
2022-05-18T04:46:02.7696154Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:46:02.7706184Z 
2022-05-18T04:46:02.7706640Z Running tests...
2022-05-18T04:46:02.7707548Z ----------------------------------------------------------------------
2022-05-18T04:46:04.4271156Z   test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:04.4734832Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90876
2022-05-18T04:46:04.4869060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90877
2022-05-18T04:46:04.5003181Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90878
2022-05-18T04:46:04.5126520Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90879
2022-05-18T04:46:05.4799754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplt97zpny
2022-05-18T04:46:05.4800387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplt97zpny/_remote_module_non_scriptable.py
2022-05-18T04:46:05.4966959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19wt2o08
2022-05-18T04:46:05.4967539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19wt2o08/_remote_module_non_scriptable.py
2022-05-18T04:46:05.5454697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhyrxin7
2022-05-18T04:46:05.5455771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhyrxin7/_remote_module_non_scriptable.py
2022-05-18T04:46:05.5879146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9erb9m7
2022-05-18T04:46:05.5879769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9erb9m7/_remote_module_non_scriptable.py
2022-05-18T04:46:05.8808093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:05.9025446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:05.9520172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:46:06.0053360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:46:09.3256205Z ok (6.555s)
2022-05-18T04:46:09.3256431Z 
2022-05-18T04:46:09.3256887Z ----------------------------------------------------------------------
2022-05-18T04:46:09.3257237Z Ran 1 test in 6.555s
2022-05-18T04:46:09.3257407Z 
2022-05-18T04:46:09.3257507Z OK
2022-05-18T04:46:09.3257648Z 
2022-05-18T04:46:09.3257794Z Generating XML reports...
2022-05-18T04:46:09.3303095Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044602.xml
2022-05-18T04:46:10.5895333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26soa4i6
2022-05-18T04:46:10.5895974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26soa4i6/_remote_module_non_scriptable.py
2022-05-18T04:46:11.0200732Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:46:11.0211813Z 
2022-05-18T04:46:11.0212513Z Running tests...
2022-05-18T04:46:11.0213002Z ----------------------------------------------------------------------
2022-05-18T04:46:12.6911276Z   test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:12.7383464Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91383
2022-05-18T04:46:12.7516455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91384
2022-05-18T04:46:12.7658063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91385
2022-05-18T04:46:12.7795557Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91386
2022-05-18T04:46:13.7335100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3cb9t74
2022-05-18T04:46:13.8143521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3cb9t74/_remote_module_non_scriptable.py
2022-05-18T04:46:13.8144110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4xahjzr
2022-05-18T04:46:13.8144679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4xahjzr/_remote_module_non_scriptable.py
2022-05-18T04:46:13.8443494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3efsnqt
2022-05-18T04:46:13.8444089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatamwerh
2022-05-18T04:46:13.8444656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3efsnqt/_remote_module_non_scriptable.py
2022-05-18T04:46:13.8445545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatamwerh/_remote_module_non_scriptable.py
2022-05-18T04:46:14.1334465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:14.2124175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:46:14.2524454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:14.2529840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:46:19.7975434Z ok (8.776s)
2022-05-18T04:46:19.7975688Z 
2022-05-18T04:46:19.7976484Z ----------------------------------------------------------------------
2022-05-18T04:46:19.7976863Z Ran 1 test in 8.776s
2022-05-18T04:46:19.7977030Z 
2022-05-18T04:46:19.7977133Z OK
2022-05-18T04:46:19.7977280Z 
2022-05-18T04:46:19.7977426Z Generating XML reports...
2022-05-18T04:46:19.8023092Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044611.xml
2022-05-18T04:46:21.0626153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtittzkq
2022-05-18T04:46:21.0626780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtittzkq/_remote_module_non_scriptable.py
2022-05-18T04:46:21.4727657Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:46:21.4737945Z 
2022-05-18T04:46:21.4738275Z Running tests...
2022-05-18T04:46:21.4739428Z ----------------------------------------------------------------------
2022-05-18T04:46:23.0860898Z   test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:23.1328148Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91898
2022-05-18T04:46:23.1467759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91899
2022-05-18T04:46:23.1604863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91900
2022-05-18T04:46:23.1730360Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91901
2022-05-18T04:46:24.1090486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7ogf7fx
2022-05-18T04:46:24.1091138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7ogf7fx/_remote_module_non_scriptable.py
2022-05-18T04:46:24.1227833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnv74iiie
2022-05-18T04:46:24.1228431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnv74iiie/_remote_module_non_scriptable.py
2022-05-18T04:46:24.1303437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcfvey3k
2022-05-18T04:46:24.1304358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcfvey3k/_remote_module_non_scriptable.py
2022-05-18T04:46:24.1518606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxm7wd78
2022-05-18T04:46:24.1519175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxm7wd78/_remote_module_non_scriptable.py
2022-05-18T04:46:24.5154063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:24.5278980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:24.5319887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:46:24.5626320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:46:30.1916528Z ok (8.718s)
2022-05-18T04:46:30.1916802Z 
2022-05-18T04:46:30.1917240Z ----------------------------------------------------------------------
2022-05-18T04:46:30.1917574Z Ran 1 test in 8.718s
2022-05-18T04:46:30.1917738Z 
2022-05-18T04:46:30.1917834Z OK
2022-05-18T04:46:30.1918367Z 
2022-05-18T04:46:30.1918506Z Generating XML reports...
2022-05-18T04:46:30.1963490Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044621.xml
2022-05-18T04:46:31.4645454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqyp5vbpl
2022-05-18T04:46:31.4646095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqyp5vbpl/_remote_module_non_scriptable.py
2022-05-18T04:46:31.8903458Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:46:31.8915047Z 
2022-05-18T04:46:31.8915699Z Running tests...
2022-05-18T04:46:31.8916195Z ----------------------------------------------------------------------
2022-05-18T04:46:33.5528486Z   test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:33.5984546Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92413
2022-05-18T04:46:33.6117411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92414
2022-05-18T04:46:33.6250236Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92415
2022-05-18T04:46:33.6371427Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92416
2022-05-18T04:46:34.6397838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp450lvej3
2022-05-18T04:46:34.6398633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp450lvej3/_remote_module_non_scriptable.py
2022-05-18T04:46:34.6569051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_s80p2w
2022-05-18T04:46:34.6569871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_s80p2w/_remote_module_non_scriptable.py
2022-05-18T04:46:34.6925237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegrr27ct
2022-05-18T04:46:34.6925828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegrr27ct/_remote_module_non_scriptable.py
2022-05-18T04:46:34.7087529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgsq9duo8
2022-05-18T04:46:34.7088131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgsq9duo8/_remote_module_non_scriptable.py
2022-05-18T04:46:35.0378276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:35.0600567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:35.1004627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:46:35.1163257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:46:35.3427092Z ok (3.451s)
2022-05-18T04:46:35.3427379Z 
2022-05-18T04:46:35.3427914Z ----------------------------------------------------------------------
2022-05-18T04:46:35.3428349Z Ran 1 test in 3.451s
2022-05-18T04:46:35.3428522Z 
2022-05-18T04:46:35.3428630Z OK
2022-05-18T04:46:35.3428747Z 
2022-05-18T04:46:35.3428886Z Generating XML reports...
2022-05-18T04:46:35.3473978Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044631.xml
2022-05-18T04:46:36.6041694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9lq29se
2022-05-18T04:46:36.6042507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9lq29se/_remote_module_non_scriptable.py
2022-05-18T04:46:37.0334272Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:46:37.0346109Z 
2022-05-18T04:46:37.0346610Z Running tests...
2022-05-18T04:46:37.0347148Z ----------------------------------------------------------------------
2022-05-18T04:46:38.6969252Z   test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:38.7410539Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92756
2022-05-18T04:46:38.7543310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92757
2022-05-18T04:46:38.7681271Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92758
2022-05-18T04:46:38.7818053Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92759
2022-05-18T04:46:39.7620048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp912_hd0
2022-05-18T04:46:39.7620909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp912_hd0/_remote_module_non_scriptable.py
2022-05-18T04:46:39.7808003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfiyexe_9
2022-05-18T04:46:39.7808608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfiyexe_9/_remote_module_non_scriptable.py
2022-05-18T04:46:39.7828669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppuafrwk7
2022-05-18T04:46:39.7829254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppuafrwk7/_remote_module_non_scriptable.py
2022-05-18T04:46:39.8231718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lr96pl9
2022-05-18T04:46:39.8232324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lr96pl9/_remote_module_non_scriptable.py
2022-05-18T04:46:40.1755153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:40.1834289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:46:40.1859426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:40.2242205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:46:40.4877702Z ok (3.453s)
2022-05-18T04:46:40.4877950Z 
2022-05-18T04:46:40.4878366Z ----------------------------------------------------------------------
2022-05-18T04:46:40.4878698Z Ran 1 test in 3.453s
2022-05-18T04:46:40.4878865Z 
2022-05-18T04:46:40.4878963Z OK
2022-05-18T04:46:40.4879260Z 
2022-05-18T04:46:40.4879417Z Generating XML reports...
2022-05-18T04:46:40.4923069Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044637.xml
2022-05-18T04:46:41.7188591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_agrqog
2022-05-18T04:46:41.7189207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_agrqog/_remote_module_non_scriptable.py
2022-05-18T04:46:42.1480850Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:46:42.1492761Z 
2022-05-18T04:46:42.1493016Z Running tests...
2022-05-18T04:46:42.1493515Z ----------------------------------------------------------------------
2022-05-18T04:46:43.8152278Z   test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:43.8627941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93099
2022-05-18T04:46:43.8763592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93100
2022-05-18T04:46:43.8917148Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93101
2022-05-18T04:46:43.9062417Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93102
2022-05-18T04:46:44.8435138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7l31ibty
2022-05-18T04:46:44.8435798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7l31ibty/_remote_module_non_scriptable.py
2022-05-18T04:46:44.9085817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvr7oasny
2022-05-18T04:46:44.9086764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvr7oasny/_remote_module_non_scriptable.py
2022-05-18T04:46:44.9109922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4l78dhk0
2022-05-18T04:46:44.9110523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4l78dhk0/_remote_module_non_scriptable.py
2022-05-18T04:46:44.9199713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2lhqlsrv
2022-05-18T04:46:44.9200289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2lhqlsrv/_remote_module_non_scriptable.py
2022-05-18T04:46:45.2444876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:46:45.3219324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:46:45.3224585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:45.3237163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:45.5121889Z ok (3.363s)
2022-05-18T04:46:45.5122148Z 
2022-05-18T04:46:45.5122612Z ----------------------------------------------------------------------
2022-05-18T04:46:45.5122984Z Ran 1 test in 3.363s
2022-05-18T04:46:45.5123167Z 
2022-05-18T04:46:45.5123274Z OK
2022-05-18T04:46:45.5123394Z 
2022-05-18T04:46:45.5123595Z Generating XML reports...
2022-05-18T04:46:45.5167891Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044642.xml
2022-05-18T04:46:46.8067779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdulzw_i5
2022-05-18T04:46:46.8068418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdulzw_i5/_remote_module_non_scriptable.py
2022-05-18T04:46:47.2378889Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:46:47.2390685Z 
2022-05-18T04:46:47.2391320Z Running tests...
2022-05-18T04:46:47.2391815Z ----------------------------------------------------------------------
2022-05-18T04:46:48.9107057Z   test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:48.9568711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93430
2022-05-18T04:46:48.9703534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93431
2022-05-18T04:46:48.9840400Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93432
2022-05-18T04:46:48.9961393Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93433
2022-05-18T04:46:49.9256027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_316r905
2022-05-18T04:46:49.9256642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_316r905/_remote_module_non_scriptable.py
2022-05-18T04:46:49.9602077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsq1im33f
2022-05-18T04:46:49.9602662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsq1im33f/_remote_module_non_scriptable.py
2022-05-18T04:46:50.0005533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjy6laww1
2022-05-18T04:46:50.0006129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjy6laww1/_remote_module_non_scriptable.py
2022-05-18T04:46:50.0060303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyf6zboiy
2022-05-18T04:46:50.0060902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyf6zboiy/_remote_module_non_scriptable.py
2022-05-18T04:46:50.3307289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:50.3632747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:46:50.4117595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:50.4138219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:46:50.7025200Z ok (3.463s)
2022-05-18T04:46:50.7025422Z 
2022-05-18T04:46:50.7025843Z ----------------------------------------------------------------------
2022-05-18T04:46:50.7026189Z Ran 1 test in 3.463s
2022-05-18T04:46:50.7026358Z 
2022-05-18T04:46:50.7026457Z OK
2022-05-18T04:46:50.7026594Z 
2022-05-18T04:46:50.7026728Z Generating XML reports...
2022-05-18T04:46:50.7072489Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044647.xml
2022-05-18T04:46:51.9828496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahuk90_c
2022-05-18T04:46:51.9829192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahuk90_c/_remote_module_non_scriptable.py
2022-05-18T04:46:52.4152171Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:46:52.4162481Z 
2022-05-18T04:46:52.4162934Z Running tests...
2022-05-18T04:46:52.4163451Z ----------------------------------------------------------------------
2022-05-18T04:46:54.0861059Z   test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:46:54.1337583Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93773
2022-05-18T04:46:54.1476646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93774
2022-05-18T04:46:54.1621909Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93775
2022-05-18T04:46:54.1748656Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93776
2022-05-18T04:46:55.0970788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmparzlargk
2022-05-18T04:46:55.0971414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmparzlargk/_remote_module_non_scriptable.py
2022-05-18T04:46:55.1617069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_vb1t1s
2022-05-18T04:46:55.1617655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_vb1t1s/_remote_module_non_scriptable.py
2022-05-18T04:46:55.1658705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgb5vpj58
2022-05-18T04:46:55.1659305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgb5vpj58/_remote_module_non_scriptable.py
2022-05-18T04:46:55.1659854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__ubev_i
2022-05-18T04:46:55.1665333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__ubev_i/_remote_module_non_scriptable.py
2022-05-18T04:46:55.5137610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:46:55.5651952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:46:55.5688294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:46:55.5694687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:46:57.7854617Z ok (5.369s)
2022-05-18T04:46:57.7854848Z 
2022-05-18T04:46:57.7855291Z ----------------------------------------------------------------------
2022-05-18T04:46:57.7855658Z Ran 1 test in 5.369s
2022-05-18T04:46:57.7855832Z 
2022-05-18T04:46:57.7855919Z OK
2022-05-18T04:46:57.7856063Z 
2022-05-18T04:46:57.7856208Z Generating XML reports...
2022-05-18T04:46:57.7900016Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044652.xml
2022-05-18T04:46:59.0365470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwq_qd67p
2022-05-18T04:46:59.0366129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwq_qd67p/_remote_module_non_scriptable.py
2022-05-18T04:46:59.4638574Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:46:59.4649479Z 
2022-05-18T04:46:59.4649829Z Running tests...
2022-05-18T04:46:59.4650315Z ----------------------------------------------------------------------
2022-05-18T04:47:01.1232094Z   test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:01.1693223Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94276
2022-05-18T04:47:01.1835796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94277
2022-05-18T04:47:01.1982748Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94278
2022-05-18T04:47:01.2111640Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94279
2022-05-18T04:47:02.1290825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjfdwl8v
2022-05-18T04:47:02.1291487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjfdwl8v/_remote_module_non_scriptable.py
2022-05-18T04:47:02.1304917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphj2glsv1
2022-05-18T04:47:02.1305528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphj2glsv1/_remote_module_non_scriptable.py
2022-05-18T04:47:02.1811470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpjgp8km
2022-05-18T04:47:02.1812095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpjgp8km/_remote_module_non_scriptable.py
2022-05-18T04:47:02.1853135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgb4z5toi
2022-05-18T04:47:02.1853712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgb4z5toi/_remote_module_non_scriptable.py
2022-05-18T04:47:02.5388222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:47:02.5442223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:02.5851505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:47:02.5987848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:04.9217360Z ok (5.456s)
2022-05-18T04:47:04.9217615Z 
2022-05-18T04:47:04.9218072Z ----------------------------------------------------------------------
2022-05-18T04:47:04.9218424Z Ran 1 test in 5.457s
2022-05-18T04:47:04.9218596Z 
2022-05-18T04:47:04.9218720Z OK
2022-05-18T04:47:04.9218865Z 
2022-05-18T04:47:04.9219016Z Generating XML reports...
2022-05-18T04:47:04.9264668Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044659.xml
2022-05-18T04:47:06.2081613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprukspehb
2022-05-18T04:47:06.2082280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprukspehb/_remote_module_non_scriptable.py
2022-05-18T04:47:06.6412144Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:47:06.6422754Z 
2022-05-18T04:47:06.6422948Z Running tests...
2022-05-18T04:47:06.6423425Z ----------------------------------------------------------------------
2022-05-18T04:47:08.3168560Z   test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:08.3631139Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94779
2022-05-18T04:47:08.3762410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94780
2022-05-18T04:47:08.3884120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94781
2022-05-18T04:47:08.4017206Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94782
2022-05-18T04:47:09.3423316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpil4jcfk7
2022-05-18T04:47:09.3423954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpil4jcfk7/_remote_module_non_scriptable.py
2022-05-18T04:47:09.3996072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvzuiaj17
2022-05-18T04:47:09.3996680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvzuiaj17/_remote_module_non_scriptable.py
2022-05-18T04:47:09.4083530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpx62rgm
2022-05-18T04:47:09.4084156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpx62rgm/_remote_module_non_scriptable.py
2022-05-18T04:47:09.4178213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy034dnna
2022-05-18T04:47:09.4178870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy034dnna/_remote_module_non_scriptable.py
2022-05-18T04:47:09.7465851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:09.8017256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:09.8279494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:47:09.8366102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:47:12.1124428Z ok (5.470s)
2022-05-18T04:47:12.1124667Z 
2022-05-18T04:47:12.1125128Z ----------------------------------------------------------------------
2022-05-18T04:47:12.1125459Z Ran 1 test in 5.470s
2022-05-18T04:47:12.1125637Z 
2022-05-18T04:47:12.1125738Z OK
2022-05-18T04:47:12.1125880Z 
2022-05-18T04:47:12.1126021Z Generating XML reports...
2022-05-18T04:47:12.1171349Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044706.xml
2022-05-18T04:47:13.3832730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_147mbp
2022-05-18T04:47:13.3833382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_147mbp/_remote_module_non_scriptable.py
2022-05-18T04:47:13.8091213Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:47:13.8101793Z 
2022-05-18T04:47:13.8102168Z Running tests...
2022-05-18T04:47:13.8103032Z ----------------------------------------------------------------------
2022-05-18T04:47:15.4735816Z   test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:15.5186234Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95282
2022-05-18T04:47:15.5321492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95283
2022-05-18T04:47:15.5460331Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95284
2022-05-18T04:47:15.5580261Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95285
2022-05-18T04:47:16.4631806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hdjhrm8
2022-05-18T04:47:16.4632697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hdjhrm8/_remote_module_non_scriptable.py
2022-05-18T04:47:16.5267240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4kis4n1
2022-05-18T04:47:16.5267913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4kis4n1/_remote_module_non_scriptable.py
2022-05-18T04:47:16.5310977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurrbs7ny
2022-05-18T04:47:16.5311596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurrbs7ny/_remote_module_non_scriptable.py
2022-05-18T04:47:16.5406706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdebd3z76
2022-05-18T04:47:16.5407608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdebd3z76/_remote_module_non_scriptable.py
2022-05-18T04:47:16.8724579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:16.9306518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:47:16.9378481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:16.9503808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:47:19.2694677Z ok (5.459s)
2022-05-18T04:47:19.2694959Z 
2022-05-18T04:47:19.2695396Z ----------------------------------------------------------------------
2022-05-18T04:47:19.2695743Z Ran 1 test in 5.459s
2022-05-18T04:47:19.2695909Z 
2022-05-18T04:47:19.2695986Z OK
2022-05-18T04:47:19.2696122Z 
2022-05-18T04:47:19.2696273Z Generating XML reports...
2022-05-18T04:47:19.2740602Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044713.xml
2022-05-18T04:47:20.5235972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqojhpha
2022-05-18T04:47:20.5236627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqojhpha/_remote_module_non_scriptable.py
2022-05-18T04:47:20.9382483Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:47:20.9395263Z 
2022-05-18T04:47:20.9395587Z Running tests...
2022-05-18T04:47:20.9396091Z ----------------------------------------------------------------------
2022-05-18T04:47:22.5949440Z   test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:22.6401486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95785
2022-05-18T04:47:22.6532880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95786
2022-05-18T04:47:22.6669132Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95787
2022-05-18T04:47:22.6802810Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95788
2022-05-18T04:47:23.6197081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2r2w6a_
2022-05-18T04:47:23.6197716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2r2w6a_/_remote_module_non_scriptable.py
2022-05-18T04:47:23.6625202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgeqly3_d
2022-05-18T04:47:23.6626782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgeqly3_d/_remote_module_non_scriptable.py
2022-05-18T04:47:23.6693198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcb5yv3mp
2022-05-18T04:47:23.6693805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcb5yv3mp/_remote_module_non_scriptable.py
2022-05-18T04:47:23.7012554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69b4io4r
2022-05-18T04:47:23.7013133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69b4io4r/_remote_module_non_scriptable.py
2022-05-18T04:47:24.0281434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:24.0698065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:47:24.0814380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:24.1024152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:47:26.2911432Z ok (5.351s)
2022-05-18T04:47:26.2911715Z 
2022-05-18T04:47:26.2912159Z ----------------------------------------------------------------------
2022-05-18T04:47:26.2912509Z Ran 1 test in 5.352s
2022-05-18T04:47:26.2913072Z 
2022-05-18T04:47:26.2913169Z OK
2022-05-18T04:47:26.2913306Z 
2022-05-18T04:47:26.2913448Z Generating XML reports...
2022-05-18T04:47:26.2958011Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044720.xml
2022-05-18T04:47:27.5684666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3qnpsh3
2022-05-18T04:47:27.5685331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3qnpsh3/_remote_module_non_scriptable.py
2022-05-18T04:47:28.0019235Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:47:28.0030193Z 
2022-05-18T04:47:28.0030514Z Running tests...
2022-05-18T04:47:28.0093839Z ----------------------------------------------------------------------
2022-05-18T04:47:29.6673463Z   test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:29.7145040Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96288
2022-05-18T04:47:29.7279580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96289
2022-05-18T04:47:29.7427214Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96290
2022-05-18T04:47:29.7552994Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96291
2022-05-18T04:47:30.6771636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajc1hw19
2022-05-18T04:47:30.6772287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajc1hw19/_remote_module_non_scriptable.py
2022-05-18T04:47:30.6821386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8lfmhe2p
2022-05-18T04:47:30.6822369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8lfmhe2p/_remote_module_non_scriptable.py
2022-05-18T04:47:30.6915242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpja4flt2t
2022-05-18T04:47:30.6915845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxow6ogg1
2022-05-18T04:47:30.6916505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpja4flt2t/_remote_module_non_scriptable.py
2022-05-18T04:47:30.6917083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxow6ogg1/_remote_module_non_scriptable.py
2022-05-18T04:47:31.0871954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:31.0917020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:47:31.0976491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:31.1003673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:47:33.3663459Z ok (5.363s)
2022-05-18T04:47:33.3663697Z 
2022-05-18T04:47:33.3664304Z ----------------------------------------------------------------------
2022-05-18T04:47:33.3664728Z Ran 1 test in 5.363s
2022-05-18T04:47:33.3664895Z 
2022-05-18T04:47:33.3665002Z OK
2022-05-18T04:47:33.3665140Z 
2022-05-18T04:47:33.3665280Z Generating XML reports...
2022-05-18T04:47:33.3711225Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044727.xml
2022-05-18T04:47:34.6664291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7iyw3wie
2022-05-18T04:47:34.6664963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7iyw3wie/_remote_module_non_scriptable.py
2022-05-18T04:47:35.0984390Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:47:35.0996038Z 
2022-05-18T04:47:35.0996330Z Running tests...
2022-05-18T04:47:35.0996795Z ----------------------------------------------------------------------
2022-05-18T04:47:36.7642479Z   test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:36.8105503Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96791
2022-05-18T04:47:36.8239775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96792
2022-05-18T04:47:36.8376548Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96793
2022-05-18T04:47:36.8500859Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96794
2022-05-18T04:47:37.7744712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqukvlp14
2022-05-18T04:47:37.7745369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqukvlp14/_remote_module_non_scriptable.py
2022-05-18T04:47:37.8244654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2k_bg43g
2022-05-18T04:47:37.8245369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2k_bg43g/_remote_module_non_scriptable.py
2022-05-18T04:47:37.8340285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcawyf136
2022-05-18T04:47:37.8340898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcawyf136/_remote_module_non_scriptable.py
2022-05-18T04:47:37.8358033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv41b_p42
2022-05-18T04:47:37.8358623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv41b_p42/_remote_module_non_scriptable.py
2022-05-18T04:47:38.1976799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:38.2321563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:47:38.2450158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:47:38.2450687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:40.5609524Z ok (5.461s)
2022-05-18T04:47:40.5609786Z 
2022-05-18T04:47:40.5610281Z ----------------------------------------------------------------------
2022-05-18T04:47:40.5610830Z Ran 1 test in 5.461s
2022-05-18T04:47:40.5611019Z 
2022-05-18T04:47:40.5611122Z OK
2022-05-18T04:47:40.5611265Z 
2022-05-18T04:47:40.5612847Z Generating XML reports...
2022-05-18T04:47:40.5656146Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044735.xml
2022-05-18T04:47:41.8192582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2v71dgo
2022-05-18T04:47:41.8193255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2v71dgo/_remote_module_non_scriptable.py
2022-05-18T04:47:42.2377276Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:47:42.2388899Z 
2022-05-18T04:47:42.2389225Z Running tests...
2022-05-18T04:47:42.2389739Z ----------------------------------------------------------------------
2022-05-18T04:47:43.8787896Z   test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:43.9238936Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97294
2022-05-18T04:47:43.9373365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97295
2022-05-18T04:47:43.9505645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97296
2022-05-18T04:47:43.9627323Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97297
2022-05-18T04:47:44.9466261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeq6ludcn
2022-05-18T04:47:44.9466903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeq6ludcn/_remote_module_non_scriptable.py
2022-05-18T04:47:44.9471251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0jkel45
2022-05-18T04:47:44.9472577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0jkel45/_remote_module_non_scriptable.py
2022-05-18T04:47:44.9533215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_857lcr
2022-05-18T04:47:44.9533807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_857lcr/_remote_module_non_scriptable.py
2022-05-18T04:47:44.9593092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8fro88c
2022-05-18T04:47:44.9593699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8fro88c/_remote_module_non_scriptable.py
2022-05-18T04:47:45.3505314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:47:45.3510486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:45.3542768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:45.3604132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:47:50.9802287Z ok (8.741s)
2022-05-18T04:47:50.9802513Z 
2022-05-18T04:47:50.9802956Z ----------------------------------------------------------------------
2022-05-18T04:47:50.9803291Z Ran 1 test in 8.741s
2022-05-18T04:47:50.9803606Z 
2022-05-18T04:47:50.9803712Z OK
2022-05-18T04:47:50.9803862Z 
2022-05-18T04:47:50.9804003Z Generating XML reports...
2022-05-18T04:47:50.9848413Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044742.xml
2022-05-18T04:47:52.2477871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqdfs0vz
2022-05-18T04:47:52.2478495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqdfs0vz/_remote_module_non_scriptable.py
2022-05-18T04:47:52.6796071Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:47:52.6807259Z 
2022-05-18T04:47:52.6807538Z Running tests...
2022-05-18T04:47:52.6808037Z ----------------------------------------------------------------------
2022-05-18T04:47:54.3625588Z   test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:47:54.4089768Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97809
2022-05-18T04:47:54.4212313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97810
2022-05-18T04:47:54.4343098Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97811
2022-05-18T04:47:54.4479243Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97812
2022-05-18T04:47:55.3524070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5rsd_ian
2022-05-18T04:47:55.3524687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5rsd_ian/_remote_module_non_scriptable.py
2022-05-18T04:47:55.3540763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7miilsak
2022-05-18T04:47:55.3541338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7miilsak/_remote_module_non_scriptable.py
2022-05-18T04:47:55.3894739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprofmkc3x
2022-05-18T04:47:55.3895333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprofmkc3x/_remote_module_non_scriptable.py
2022-05-18T04:47:55.4117532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5p0j_2j
2022-05-18T04:47:55.4118203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5p0j_2j/_remote_module_non_scriptable.py
2022-05-18T04:47:55.7545930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:47:55.7554187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:47:55.7891646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:47:55.8118122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:48:01.3663251Z ok (8.685s)
2022-05-18T04:48:01.3663488Z 
2022-05-18T04:48:01.3663910Z ----------------------------------------------------------------------
2022-05-18T04:48:01.3664264Z Ran 1 test in 8.686s
2022-05-18T04:48:01.3664438Z 
2022-05-18T04:48:01.3664536Z OK
2022-05-18T04:48:01.3664675Z 
2022-05-18T04:48:01.3664817Z Generating XML reports...
2022-05-18T04:48:01.3708997Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044752.xml
2022-05-18T04:48:02.6617430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpefdwt7nl
2022-05-18T04:48:02.6618068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpefdwt7nl/_remote_module_non_scriptable.py
2022-05-18T04:48:03.0921108Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:48:03.0932598Z 
2022-05-18T04:48:03.0932903Z Running tests...
2022-05-18T04:48:03.0933387Z ----------------------------------------------------------------------
2022-05-18T04:48:04.7835286Z   test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:04.8295935Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98316
2022-05-18T04:48:04.8434186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98317
2022-05-18T04:48:04.8569641Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98318
2022-05-18T04:48:04.8697842Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98319
2022-05-18T04:48:05.8459256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27d3d7kd
2022-05-18T04:48:05.8459899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27d3d7kd/_remote_module_non_scriptable.py
2022-05-18T04:48:05.8600373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwg3vvh3a
2022-05-18T04:48:05.8600980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwg3vvh3a/_remote_module_non_scriptable.py
2022-05-18T04:48:05.8693395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7gqu9gza
2022-05-18T04:48:05.8694223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7gqu9gza/_remote_module_non_scriptable.py
2022-05-18T04:48:05.8728009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmput2t2z3f
2022-05-18T04:48:05.8728601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmput2t2z3f/_remote_module_non_scriptable.py
2022-05-18T04:48:06.2464655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:48:06.2689093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:48:06.2691302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:06.2885159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:06.4756539Z ok (3.382s)
2022-05-18T04:48:06.4756772Z 
2022-05-18T04:48:06.4757202Z ----------------------------------------------------------------------
2022-05-18T04:48:06.4757547Z Ran 1 test in 3.382s
2022-05-18T04:48:06.4757713Z 
2022-05-18T04:48:06.4757810Z OK
2022-05-18T04:48:06.4757949Z 
2022-05-18T04:48:06.4758117Z Generating XML reports...
2022-05-18T04:48:06.4801649Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044803.xml
2022-05-18T04:48:07.7489643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogj9xmzk
2022-05-18T04:48:07.7490541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogj9xmzk/_remote_module_non_scriptable.py
2022-05-18T04:48:08.1815087Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:48:08.1826690Z 
2022-05-18T04:48:08.1826889Z Running tests...
2022-05-18T04:48:08.1827385Z ----------------------------------------------------------------------
2022-05-18T04:48:09.8607766Z   test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:09.9072897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98647
2022-05-18T04:48:09.9208200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98648
2022-05-18T04:48:09.9349932Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98649
2022-05-18T04:48:09.9490659Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98650
2022-05-18T04:48:10.9112192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppihshxp5
2022-05-18T04:48:10.9113002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppihshxp5/_remote_module_non_scriptable.py
2022-05-18T04:48:10.9435513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcb2qvylt
2022-05-18T04:48:10.9437297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcb2qvylt/_remote_module_non_scriptable.py
2022-05-18T04:48:10.9465748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxq2k5xk1
2022-05-18T04:48:10.9466796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxq2k5xk1/_remote_module_non_scriptable.py
2022-05-18T04:48:10.9503958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5nrbf9p
2022-05-18T04:48:10.9504771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5nrbf9p/_remote_module_non_scriptable.py
2022-05-18T04:48:11.3148723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:11.3548309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:11.3548808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:48:11.3635003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:48:16.9675889Z ok (8.785s)
2022-05-18T04:48:16.9676281Z 
2022-05-18T04:48:16.9676833Z ----------------------------------------------------------------------
2022-05-18T04:48:16.9677207Z Ran 1 test in 8.785s
2022-05-18T04:48:16.9677397Z 
2022-05-18T04:48:16.9677495Z OK
2022-05-18T04:48:16.9677633Z 
2022-05-18T04:48:16.9677753Z Generating XML reports...
2022-05-18T04:48:16.9720597Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044808.xml
2022-05-18T04:48:18.2365873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6ufd39l
2022-05-18T04:48:18.2366516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6ufd39l/_remote_module_non_scriptable.py
2022-05-18T04:48:18.6577515Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:48:18.6588593Z 
2022-05-18T04:48:18.6588970Z Running tests...
2022-05-18T04:48:18.6589931Z ----------------------------------------------------------------------
2022-05-18T04:48:20.3111036Z   test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:20.3577938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99162
2022-05-18T04:48:20.3713891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99163
2022-05-18T04:48:20.3850710Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99164
2022-05-18T04:48:20.3978032Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99165
2022-05-18T04:48:21.3910881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7l5yc3du
2022-05-18T04:48:21.3911697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7l5yc3du/_remote_module_non_scriptable.py
2022-05-18T04:48:21.3974908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3oykdfj
2022-05-18T04:48:21.3975943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3oykdfj/_remote_module_non_scriptable.py
2022-05-18T04:48:21.3998402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngl6tlsu
2022-05-18T04:48:21.3999248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngl6tlsu/_remote_module_non_scriptable.py
2022-05-18T04:48:21.4056513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfa_a9rrl
2022-05-18T04:48:21.4057767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfa_a9rrl/_remote_module_non_scriptable.py
2022-05-18T04:48:21.7986339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:48:21.7986889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:21.8061349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:48:21.8180338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:32.0262446Z ok (13.367s)
2022-05-18T04:48:32.0263004Z 
2022-05-18T04:48:32.0263473Z ----------------------------------------------------------------------
2022-05-18T04:48:32.0263827Z Ran 1 test in 13.367s
2022-05-18T04:48:32.0263997Z 
2022-05-18T04:48:32.0264094Z OK
2022-05-18T04:48:32.0264233Z 
2022-05-18T04:48:32.0264373Z Generating XML reports...
2022-05-18T04:48:32.0308311Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044818.xml
2022-05-18T04:48:33.2946738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0zlmpu5
2022-05-18T04:48:33.2947410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0zlmpu5/_remote_module_non_scriptable.py
2022-05-18T04:48:33.7260232Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:48:33.7271346Z 
2022-05-18T04:48:33.7271670Z Running tests...
2022-05-18T04:48:33.7272151Z ----------------------------------------------------------------------
2022-05-18T04:48:35.3943491Z   test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:35.4406883Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99693
2022-05-18T04:48:35.4541080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99694
2022-05-18T04:48:35.4681073Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99695
2022-05-18T04:48:35.4812396Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99696
2022-05-18T04:48:36.4632555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4om161p3
2022-05-18T04:48:36.4633143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4om161p3/_remote_module_non_scriptable.py
2022-05-18T04:48:36.4666124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkryaxz29
2022-05-18T04:48:36.4666730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkryaxz29/_remote_module_non_scriptable.py
2022-05-18T04:48:36.5125506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxd9r_4j
2022-05-18T04:48:36.5126058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfpu508s
2022-05-18T04:48:36.5126611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxd9r_4j/_remote_module_non_scriptable.py
2022-05-18T04:48:36.5127458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfpu508s/_remote_module_non_scriptable.py
2022-05-18T04:48:36.8662644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:48:36.8683107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:36.9288194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:48:36.9340724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:47.4123226Z ok (13.685s)
2022-05-18T04:48:47.4123472Z 
2022-05-18T04:48:47.4123909Z ----------------------------------------------------------------------
2022-05-18T04:48:47.4124254Z Ran 1 test in 13.685s
2022-05-18T04:48:47.4124403Z 
2022-05-18T04:48:47.4124499Z OK
2022-05-18T04:48:47.4124636Z 
2022-05-18T04:48:47.4124796Z Generating XML reports...
2022-05-18T04:48:47.4168768Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044833.xml
2022-05-18T04:48:48.6752383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcu9crry3
2022-05-18T04:48:48.6753035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcu9crry3/_remote_module_non_scriptable.py
2022-05-18T04:48:49.0987677Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:48:49.0998311Z 
2022-05-18T04:48:49.0998711Z Running tests...
2022-05-18T04:48:49.0999307Z ----------------------------------------------------------------------
2022-05-18T04:48:50.7254392Z   test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:50.7720145Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100208
2022-05-18T04:48:50.7851944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100209
2022-05-18T04:48:50.7977150Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100210
2022-05-18T04:48:50.8115010Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100211
2022-05-18T04:48:51.7912912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5axhdd4
2022-05-18T04:48:51.7913529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8w6ix68h
2022-05-18T04:48:51.7914101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5axhdd4/_remote_module_non_scriptable.py
2022-05-18T04:48:51.7914664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8w6ix68h/_remote_module_non_scriptable.py
2022-05-18T04:48:51.7932904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwy2mat94
2022-05-18T04:48:51.7933503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwy2mat94/_remote_module_non_scriptable.py
2022-05-18T04:48:51.8183576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpon1kqs0w
2022-05-18T04:48:51.8184175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpon1kqs0w/_remote_module_non_scriptable.py
2022-05-18T04:48:52.2018942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:52.2032742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:48:52.2033558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:52.2274853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:48:52.5173692Z ok (3.417s)
2022-05-18T04:48:52.5173928Z 
2022-05-18T04:48:52.5174360Z ----------------------------------------------------------------------
2022-05-18T04:48:52.5175091Z Ran 1 test in 3.418s
2022-05-18T04:48:52.5175269Z 
2022-05-18T04:48:52.5175381Z OK
2022-05-18T04:48:52.5175523Z 
2022-05-18T04:48:52.5175672Z Generating XML reports...
2022-05-18T04:48:52.5218626Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044849.xml
2022-05-18T04:48:53.7729592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjijm354
2022-05-18T04:48:53.7730239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjijm354/_remote_module_non_scriptable.py
2022-05-18T04:48:54.1860836Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:48:54.1871329Z 
2022-05-18T04:48:54.1872100Z Running tests...
2022-05-18T04:48:54.1872585Z ----------------------------------------------------------------------
2022-05-18T04:48:55.8123142Z   test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:48:55.8597591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100551
2022-05-18T04:48:55.8735429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100552
2022-05-18T04:48:55.8875417Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100553
2022-05-18T04:48:55.8999890Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100554
2022-05-18T04:48:56.8613103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4yah11x
2022-05-18T04:48:56.8613800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4yah11x/_remote_module_non_scriptable.py
2022-05-18T04:48:56.8775415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqaxkvb36
2022-05-18T04:48:56.8776010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqaxkvb36/_remote_module_non_scriptable.py
2022-05-18T04:48:56.8786924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kf7_k9g
2022-05-18T04:48:56.8787549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kf7_k9g/_remote_module_non_scriptable.py
2022-05-18T04:48:56.8987335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplf2nbvi_
2022-05-18T04:48:56.8987938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplf2nbvi_/_remote_module_non_scriptable.py
2022-05-18T04:48:57.2605202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:48:57.2792262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:48:57.2850704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:48:57.3091900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:49:00.1416357Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:49:00.1446953Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f4a8ea431bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f4a8ea3eb8e in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc2b (0x7f4a99fddbfb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f4a99fe003f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf7 (0x7f4a99fe1807 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f4a9a1af4cf in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2ca0646 (0x7f4a91b74646 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2ca0766 (0x7f4a91b74766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f4a9aa77f78 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x2bbc355 (0x7f4a9be15355 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x2bbcae9 (0x7f4a9be15ae9 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x173 (0x7f4a9aaa35e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2c3427 (0x7f4aa4aae427 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2c3766 (0x7f4aa4aae766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1bfb9c (0x5567f552bb9c in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x18e1bb (0x5567f54fa1bb in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x18e391 (0x5567f54fa391 in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x3d (0x5567f54a9ffd in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0xe1d (0x5567f55421fd in /opt/conda/bin/python)\nframe #19: _PyFunction_Vectorcall + 0x104 (0x5567f5503284 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x1da (0x5567f54b1a7a in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x2610 (0x5567f55439f0 in /opt/conda/bin/python)\nframe #22: _PyFunction_Vectorcall + 0x104 (0x5567f5503284 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x1da (0x5567f54b1a7a in /opt/conda/bin/python)\nframe #24: <unknown function> + 0x94774a (0x7f4aa513274a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f4aa5130a3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f4aa5133b25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f4aa5137776 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f4a9cf40abc in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f4aa5133915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x3ce0e43 (0x7f4a9cf39e43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f4a9cf3aa38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f4a9cf350b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x3d10b42 (0x7f4a9cf69b42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f4a8ea315eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xc9039 (0x7f4aa8173039 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7f4add7786db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7f4add4a161f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-05-18T04:49:00.1464804Z Traceback (most recent call last):
2022-05-18T04:49:00.1466013Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:49:00.1467016Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:49:00.1468419Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 6267, in _gpu_add_wrong_gpus
2022-05-18T04:49:00.1469362Z     return x.cpu() + y.cuda()
2022-05-18T04:49:00.1470264Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-05-18T04:49:00.1471467Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-05-18T04:49:00.1473411Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f4a8ea431bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:49:00.1475671Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f4a8ea3eb8e in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:49:00.1477752Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc2b (0x7f4a99fddbfb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1479611Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f4a99fe003f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1481626Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf7 (0x7f4a99fe1807 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1483699Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f4a9a1af4cf in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1485366Z frame #6: <unknown function> + 0x2ca0646 (0x7f4a91b74646 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:49:00.1486860Z frame #7: <unknown function> + 0x2ca0766 (0x7f4a91b74766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:49:00.1488723Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f4a9aa77f78 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1490397Z frame #9: <unknown function> + 0x2bbc355 (0x7f4a9be15355 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1491815Z frame #10: <unknown function> + 0x2bbcae9 (0x7f4a9be15ae9 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1493753Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x173 (0x7f4a9aaa35e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1495371Z frame #12: <unknown function> + 0x2c3427 (0x7f4aa4aae427 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1496852Z frame #13: <unknown function> + 0x2c3766 (0x7f4aa4aae766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1497689Z frame #14: <unknown function> + 0x1bfb9c (0x5567f552bb9c in /opt/conda/bin/python)
2022-05-18T04:49:00.1498474Z frame #15: <unknown function> + 0x18e1bb (0x5567f54fa1bb in /opt/conda/bin/python)
2022-05-18T04:49:00.1499260Z frame #16: <unknown function> + 0x18e391 (0x5567f54fa391 in /opt/conda/bin/python)
2022-05-18T04:49:00.1499944Z frame #17: PyNumber_Add + 0x3d (0x5567f54a9ffd in /opt/conda/bin/python)
2022-05-18T04:49:00.1500664Z frame #18: _PyEval_EvalFrameDefault + 0xe1d (0x5567f55421fd in /opt/conda/bin/python)
2022-05-18T04:49:00.1501415Z frame #19: _PyFunction_Vectorcall + 0x104 (0x5567f5503284 in /opt/conda/bin/python)
2022-05-18T04:49:00.1502313Z frame #20: _PyObject_Call + 0x1da (0x5567f54b1a7a in /opt/conda/bin/python)
2022-05-18T04:49:00.1503027Z frame #21: _PyEval_EvalFrameDefault + 0x2610 (0x5567f55439f0 in /opt/conda/bin/python)
2022-05-18T04:49:00.1503809Z frame #22: _PyFunction_Vectorcall + 0x104 (0x5567f5503284 in /opt/conda/bin/python)
2022-05-18T04:49:00.1504704Z frame #23: _PyObject_Call + 0x1da (0x5567f54b1a7a in /opt/conda/bin/python)
2022-05-18T04:49:00.1505998Z frame #24: <unknown function> + 0x94774a (0x7f4aa513274a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1507432Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f4aa5130a3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1509329Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f4aa5133b25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1511355Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f4aa5137776 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1514009Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f4a9cf40abc in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1517075Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f4aa5133915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1519121Z frame #30: <unknown function> + 0x3ce0e43 (0x7f4a9cf39e43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1521279Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f4a9cf3aa38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1523732Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f4a9cf350b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1525415Z frame #33: <unknown function> + 0x3d10b42 (0x7f4a9cf69b42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1527230Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f4a8ea315eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:49:00.1528322Z frame #35: <unknown function> + 0xc9039 (0x7f4aa8173039 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-05-18T04:49:00.1529552Z frame #36: <unknown function> + 0x76db (0x7f4add7786db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-05-18T04:49:00.1530690Z frame #37: clone + 0x3f (0x7f4add4a161f in /lib/x86_64-linux-gnu/libc.so.6)
2022-05-18T04:49:00.1531206Z 
2022-05-18T04:49:00.1531372Z 
2022-05-18T04:49:00.1610635Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:49:00.1624789Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7b40fee1bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f7b40fe9b8e in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc2b (0x7f7b4c588bfb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f7b4c58b03f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf7 (0x7f7b4c58c807 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f7b4c75a4cf in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2ca0646 (0x7f7b4411f646 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2ca0766 (0x7f7b4411f766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f7b4d022f78 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x2bbc355 (0x7f7b4e3c0355 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x2bbcae9 (0x7f7b4e3c0ae9 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x173 (0x7f7b4d04e5e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2c3427 (0x7f7b57059427 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2c3766 (0x7f7b57059766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1bfb9c (0x560e35930b9c in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x18e1bb (0x560e358ff1bb in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x18e391 (0x560e358ff391 in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x3d (0x560e358aeffd in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0xe1d (0x560e359471fd in /opt/conda/bin/python)\nframe #19: _PyFunction_Vectorcall + 0x104 (0x560e35908284 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x1da (0x560e358b6a7a in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x2610 (0x560e359489f0 in /opt/conda/bin/python)\nframe #22: _PyFunction_Vectorcall + 0x104 (0x560e35908284 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x1da (0x560e358b6a7a in /opt/conda/bin/python)\nframe #24: <unknown function> + 0x94774a (0x7f7b576dd74a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7b576dba3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7b576deb25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f7b576e2776 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f7b4f4ebabc in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7b576de915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x3ce0e43 (0x7f7b4f4e4e43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7b4f4e5a38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7b4f4e00b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x3d10b42 (0x7f7b4f514b42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7b40fdc5eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xc9039 (0x7f7b5a71e039 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7f7b8fd236db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7f7b8fa4c61f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-05-18T04:49:00.1632456Z Traceback (most recent call last):
2022-05-18T04:49:00.1633022Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:49:00.1633504Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:49:00.1634166Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 6267, in _gpu_add_wrong_gpus
2022-05-18T04:49:00.1634570Z     return x.cpu() + y.cuda()
2022-05-18T04:49:00.1634990Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-05-18T04:49:00.1635547Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-05-18T04:49:00.1636410Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7b40fee1bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:49:00.1637409Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f7b40fe9b8e in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:49:00.1638399Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc2b (0x7f7b4c588bfb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1639218Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f7b4c58b03f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1640186Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf7 (0x7f7b4c58c807 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1641111Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f7b4c75a4cf in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1641830Z frame #6: <unknown function> + 0x2ca0646 (0x7f7b4411f646 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:49:00.1642481Z frame #7: <unknown function> + 0x2ca0766 (0x7f7b4411f766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:49:00.1643306Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f7b4d022f78 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1644056Z frame #9: <unknown function> + 0x2bbc355 (0x7f7b4e3c0355 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1644724Z frame #10: <unknown function> + 0x2bbcae9 (0x7f7b4e3c0ae9 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1645482Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x173 (0x7f7b4d04e5e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1646203Z frame #12: <unknown function> + 0x2c3427 (0x7f7b57059427 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1646857Z frame #13: <unknown function> + 0x2c3766 (0x7f7b57059766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1647333Z frame #14: <unknown function> + 0x1bfb9c (0x560e35930b9c in /opt/conda/bin/python)
2022-05-18T04:49:00.1647728Z frame #15: <unknown function> + 0x18e1bb (0x560e358ff1bb in /opt/conda/bin/python)
2022-05-18T04:49:00.1648137Z frame #16: <unknown function> + 0x18e391 (0x560e358ff391 in /opt/conda/bin/python)
2022-05-18T04:49:00.1648537Z frame #17: PyNumber_Add + 0x3d (0x560e358aeffd in /opt/conda/bin/python)
2022-05-18T04:49:00.1648961Z frame #18: _PyEval_EvalFrameDefault + 0xe1d (0x560e359471fd in /opt/conda/bin/python)
2022-05-18T04:49:00.1649411Z frame #19: _PyFunction_Vectorcall + 0x104 (0x560e35908284 in /opt/conda/bin/python)
2022-05-18T04:49:00.1649821Z frame #20: _PyObject_Call + 0x1da (0x560e358b6a7a in /opt/conda/bin/python)
2022-05-18T04:49:00.1650244Z frame #21: _PyEval_EvalFrameDefault + 0x2610 (0x560e359489f0 in /opt/conda/bin/python)
2022-05-18T04:49:00.1650645Z frame #22: _PyFunction_Vectorcall + 0x104 (0x560e35908284 in /opt/conda/bin/python)
2022-05-18T04:49:00.1651061Z frame #23: _PyObject_Call + 0x1da (0x560e358b6a7a in /opt/conda/bin/python)
2022-05-18T04:49:00.1651676Z frame #24: <unknown function> + 0x94774a (0x7f7b576dd74a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1652463Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7b576dba3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1653503Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7b576deb25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1654715Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f7b576e2776 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1655991Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f7b4f4ebabc in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1657309Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7b576de915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1658218Z frame #30: <unknown function> + 0x3ce0e43 (0x7f7b4f4e4e43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1659177Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7b4f4e5a38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1660234Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7b4f4e00b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1661037Z frame #33: <unknown function> + 0x3d10b42 (0x7f7b4f514b42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1661730Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7b40fdc5eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:49:00.1662583Z frame #35: <unknown function> + 0xc9039 (0x7f7b5a71e039 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-05-18T04:49:00.1663131Z frame #36: <unknown function> + 0x76db (0x7f7b8fd236db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-05-18T04:49:00.1663653Z frame #37: clone + 0x3f (0x7f7b8fa4c61f in /lib/x86_64-linux-gnu/libc.so.6)
2022-05-18T04:49:00.1663883Z 
2022-05-18T04:49:00.1663905Z 
2022-05-18T04:49:00.1769352Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:49:00.1784660Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7a2ce801bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f7a2ce7bb8e in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc2b (0x7f7a3841abfb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f7a3841d03f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf7 (0x7f7a3841e807 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f7a385ec4cf in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2ca0646 (0x7f7a2ffb1646 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2ca0766 (0x7f7a2ffb1766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f7a38eb4f78 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x2bbc355 (0x7f7a3a252355 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x2bbcae9 (0x7f7a3a252ae9 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x173 (0x7f7a38ee05e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2c3427 (0x7f7a42eeb427 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2c3766 (0x7f7a42eeb766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1bfb9c (0x5635ed906b9c in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x18e1bb (0x5635ed8d51bb in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x18e391 (0x5635ed8d5391 in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x3d (0x5635ed884ffd in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0xe1d (0x5635ed91d1fd in /opt/conda/bin/python)\nframe #19: _PyFunction_Vectorcall + 0x104 (0x5635ed8de284 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x1da (0x5635ed88ca7a in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x2610 (0x5635ed91e9f0 in /opt/conda/bin/python)\nframe #22: _PyFunction_Vectorcall + 0x104 (0x5635ed8de284 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x1da (0x5635ed88ca7a in /opt/conda/bin/python)\nframe #24: <unknown function> + 0x94774a (0x7f7a4356f74a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7a4356da3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7a43570b25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f7a43574776 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f7a3b37dabc in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7a43570915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x3ce0e43 (0x7f7a3b376e43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7a3b377a38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7a3b3720b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x3d10b42 (0x7f7a3b3a6b42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7a2ce6e5eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xc9039 (0x7f7a465b0039 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7f7a7bbb56db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7f7a7b8de61f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-05-18T04:49:00.1792362Z Traceback (most recent call last):
2022-05-18T04:49:00.1792974Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:49:00.1793455Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:49:00.1794134Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 6267, in _gpu_add_wrong_gpus
2022-05-18T04:49:00.1794564Z     return x.cpu() + y.cuda()
2022-05-18T04:49:00.1794951Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-05-18T04:49:00.1795510Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-05-18T04:49:00.1796382Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7a2ce801bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:49:00.1797381Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f7a2ce7bb8e in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:49:00.1798295Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc2b (0x7f7a3841abfb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1799094Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f7a3841d03f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1800012Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf7 (0x7f7a3841e807 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1800925Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f7a385ec4cf in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1801671Z frame #6: <unknown function> + 0x2ca0646 (0x7f7a2ffb1646 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:49:00.1802335Z frame #7: <unknown function> + 0x2ca0766 (0x7f7a2ffb1766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:49:00.1803158Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f7a38eb4f78 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1803913Z frame #9: <unknown function> + 0x2bbc355 (0x7f7a3a252355 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1804565Z frame #10: <unknown function> + 0x2bbcae9 (0x7f7a3a252ae9 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1805347Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x173 (0x7f7a38ee05e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1806058Z frame #12: <unknown function> + 0x2c3427 (0x7f7a42eeb427 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1806720Z frame #13: <unknown function> + 0x2c3766 (0x7f7a42eeb766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1807267Z frame #14: <unknown function> + 0x1bfb9c (0x5635ed906b9c in /opt/conda/bin/python)
2022-05-18T04:49:00.1807686Z frame #15: <unknown function> + 0x18e1bb (0x5635ed8d51bb in /opt/conda/bin/python)
2022-05-18T04:49:00.1808077Z frame #16: <unknown function> + 0x18e391 (0x5635ed8d5391 in /opt/conda/bin/python)
2022-05-18T04:49:00.1808484Z frame #17: PyNumber_Add + 0x3d (0x5635ed884ffd in /opt/conda/bin/python)
2022-05-18T04:49:00.1808916Z frame #18: _PyEval_EvalFrameDefault + 0xe1d (0x5635ed91d1fd in /opt/conda/bin/python)
2022-05-18T04:49:00.1809378Z frame #19: _PyFunction_Vectorcall + 0x104 (0x5635ed8de284 in /opt/conda/bin/python)
2022-05-18T04:49:00.1809804Z frame #20: _PyObject_Call + 0x1da (0x5635ed88ca7a in /opt/conda/bin/python)
2022-05-18T04:49:00.1810231Z frame #21: _PyEval_EvalFrameDefault + 0x2610 (0x5635ed91e9f0 in /opt/conda/bin/python)
2022-05-18T04:49:00.1810662Z frame #22: _PyFunction_Vectorcall + 0x104 (0x5635ed8de284 in /opt/conda/bin/python)
2022-05-18T04:49:00.1811058Z frame #23: _PyObject_Call + 0x1da (0x5635ed88ca7a in /opt/conda/bin/python)
2022-05-18T04:49:00.1811668Z frame #24: <unknown function> + 0x94774a (0x7f7a4356f74a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1812473Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7a4356da3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1813507Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7a43570b25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1814613Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f7a43574776 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1815857Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f7a3b37dabc in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1817168Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7a43570915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1818070Z frame #30: <unknown function> + 0x3ce0e43 (0x7f7a3b376e43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1819028Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7a3b377a38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1820109Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7a3b3720b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1820885Z frame #33: <unknown function> + 0x3d10b42 (0x7f7a3b3a6b42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1821593Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7a2ce6e5eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:49:00.1822412Z frame #35: <unknown function> + 0xc9039 (0x7f7a465b0039 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-05-18T04:49:00.1822976Z frame #36: <unknown function> + 0x76db (0x7f7a7bbb56db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-05-18T04:49:00.1823570Z frame #37: clone + 0x3f (0x7f7a7b8de61f in /lib/x86_64-linux-gnu/libc.so.6)
2022-05-18T04:49:00.1823806Z 
2022-05-18T04:49:00.1823827Z 
2022-05-18T04:49:00.1823974Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:49:00.1836422Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f42e78301bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f42e782bb8e in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc2b (0x7f42f2dcabfb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f42f2dcd03f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf7 (0x7f42f2dce807 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f42f2f9c4cf in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2ca0646 (0x7f42ea961646 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2ca0766 (0x7f42ea961766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f42f3864f78 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x2bbc355 (0x7f42f4c02355 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x2bbcae9 (0x7f42f4c02ae9 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x173 (0x7f42f38905e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2c3427 (0x7f42fd89b427 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2c3766 (0x7f42fd89b766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1bfb9c (0x55d300301b9c in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x18e1bb (0x55d3002d01bb in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x18e391 (0x55d3002d0391 in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x3d (0x55d30027fffd in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0xe1d (0x55d3003181fd in /opt/conda/bin/python)\nframe #19: _PyFunction_Vectorcall + 0x104 (0x55d3002d9284 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x1da (0x55d300287a7a in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x2610 (0x55d3003199f0 in /opt/conda/bin/python)\nframe #22: _PyFunction_Vectorcall + 0x104 (0x55d3002d9284 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x1da (0x55d300287a7a in /opt/conda/bin/python)\nframe #24: <unknown function> + 0x94774a (0x7f42fdf1f74a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f42fdf1da3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f42fdf20b25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f42fdf24776 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f42f5d2dabc in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f42fdf20915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x3ce0e43 (0x7f42f5d26e43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f42f5d27a38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f42f5d220b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x3d10b42 (0x7f42f5d56b42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f42e781e5eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xc9039 (0x7f4300f60039 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7f43365656db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7f433628e61f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-05-18T04:49:00.1843984Z Traceback (most recent call last):
2022-05-18T04:49:00.1844535Z   File "/opt/conda/lib/python3.9/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:49:00.1844983Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:49:00.1845617Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 6267, in _gpu_add_wrong_gpus
2022-05-18T04:49:00.1846046Z     return x.cpu() + y.cuda()
2022-05-18T04:49:00.1846430Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-05-18T04:49:00.1846990Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-05-18T04:49:00.1847858Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f42e78301bb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:49:00.1848842Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f42e782bb8e in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:49:00.1849829Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc2b (0x7f42f2dcabfb in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1850634Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f42f2dcd03f in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1851623Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf7 (0x7f42f2dce807 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1852538Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f42f2f9c4cf in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1853280Z frame #6: <unknown function> + 0x2ca0646 (0x7f42ea961646 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:49:00.1853966Z frame #7: <unknown function> + 0x2ca0766 (0x7f42ea961766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-05-18T04:49:00.1854816Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f42f3864f78 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1855580Z frame #9: <unknown function> + 0x2bbc355 (0x7f42f4c02355 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1856241Z frame #10: <unknown function> + 0x2bbcae9 (0x7f42f4c02ae9 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1857009Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x173 (0x7f42f38905e3 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1857716Z frame #12: <unknown function> + 0x2c3427 (0x7f42fd89b427 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1858365Z frame #13: <unknown function> + 0x2c3766 (0x7f42fd89b766 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1858835Z frame #14: <unknown function> + 0x1bfb9c (0x55d300301b9c in /opt/conda/bin/python)
2022-05-18T04:49:00.1859235Z frame #15: <unknown function> + 0x18e1bb (0x55d3002d01bb in /opt/conda/bin/python)
2022-05-18T04:49:00.1859641Z frame #16: <unknown function> + 0x18e391 (0x55d3002d0391 in /opt/conda/bin/python)
2022-05-18T04:49:00.1860043Z frame #17: PyNumber_Add + 0x3d (0x55d30027fffd in /opt/conda/bin/python)
2022-05-18T04:49:00.1860462Z frame #18: _PyEval_EvalFrameDefault + 0xe1d (0x55d3003181fd in /opt/conda/bin/python)
2022-05-18T04:49:00.1860873Z frame #19: _PyFunction_Vectorcall + 0x104 (0x55d3002d9284 in /opt/conda/bin/python)
2022-05-18T04:49:00.1861275Z frame #20: _PyObject_Call + 0x1da (0x55d300287a7a in /opt/conda/bin/python)
2022-05-18T04:49:00.1861699Z frame #21: _PyEval_EvalFrameDefault + 0x2610 (0x55d3003199f0 in /opt/conda/bin/python)
2022-05-18T04:49:00.1862361Z frame #22: _PyFunction_Vectorcall + 0x104 (0x55d3002d9284 in /opt/conda/bin/python)
2022-05-18T04:49:00.1862774Z frame #23: _PyObject_Call + 0x1da (0x55d300287a7a in /opt/conda/bin/python)
2022-05-18T04:49:00.1863386Z frame #24: <unknown function> + 0x94774a (0x7f42fdf1f74a in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1864199Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f42fdf1da3d in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1865201Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f42fdf20b25 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1866341Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f42fdf24776 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1867578Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f42f5d2dabc in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1868980Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f42fdf20915 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
2022-05-18T04:49:00.1870002Z frame #30: <unknown function> + 0x3ce0e43 (0x7f42f5d26e43 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1870974Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f42f5d27a38 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1872064Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f42f5d220b7 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1872840Z frame #33: <unknown function> + 0x3d10b42 (0x7f42f5d56b42 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T04:49:00.1873526Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f42e781e5eb in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
2022-05-18T04:49:00.1874041Z frame #35: <unknown function> + 0xc9039 (0x7f4300f60039 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-05-18T04:49:00.1874591Z frame #36: <unknown function> + 0x76db (0x7f43365656db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-05-18T04:49:00.1875074Z frame #37: clone + 0x3f (0x7f433628e61f in /lib/x86_64-linux-gnu/libc.so.6)
2022-05-18T04:49:00.1875299Z 
2022-05-18T04:49:00.1875325Z 
2022-05-18T04:49:00.7130536Z ok (6.526s)
2022-05-18T04:49:00.7130926Z 
2022-05-18T04:49:00.7131368Z ----------------------------------------------------------------------
2022-05-18T04:49:00.7131747Z Ran 1 test in 6.526s
2022-05-18T04:49:00.7131921Z 
2022-05-18T04:49:00.7132021Z OK
2022-05-18T04:49:00.7132142Z 
2022-05-18T04:49:00.7132292Z Generating XML reports...
2022-05-18T04:49:00.7175327Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044854.xml
2022-05-18T04:49:02.0032894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpof376d55
2022-05-18T04:49:02.0033569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpof376d55/_remote_module_non_scriptable.py
2022-05-18T04:49:02.4207235Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:49:02.4216592Z 
2022-05-18T04:49:02.4216808Z Running tests...
2022-05-18T04:49:02.4217648Z ----------------------------------------------------------------------
2022-05-18T04:49:04.0404493Z   test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:04.0861370Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101054
2022-05-18T04:49:04.1002553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101055
2022-05-18T04:49:04.1122566Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101056
2022-05-18T04:49:04.1254917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101057
2022-05-18T04:49:05.0270494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavjf2haz
2022-05-18T04:49:05.0275127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavjf2haz/_remote_module_non_scriptable.py
2022-05-18T04:49:05.0412781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeogym6wh
2022-05-18T04:49:05.0413667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeogym6wh/_remote_module_non_scriptable.py
2022-05-18T04:49:05.0762299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9b55pfk
2022-05-18T04:49:05.0762895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9b55pfk/_remote_module_non_scriptable.py
2022-05-18T04:49:05.1102244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbadl724
2022-05-18T04:49:05.1103171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbadl724/_remote_module_non_scriptable.py
2022-05-18T04:49:05.4373579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:05.4597619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:05.4784118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:49:05.5260087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:49:05.7310342Z ok (3.309s)
2022-05-18T04:49:05.7310578Z 
2022-05-18T04:49:05.7311030Z ----------------------------------------------------------------------
2022-05-18T04:49:05.7311413Z Ran 1 test in 3.309s
2022-05-18T04:49:05.7311585Z 
2022-05-18T04:49:05.7311685Z OK
2022-05-18T04:49:05.7311827Z 
2022-05-18T04:49:05.7311946Z Generating XML reports...
2022-05-18T04:49:05.7357003Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044902.xml
2022-05-18T04:49:06.9858457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgb065zyf
2022-05-18T04:49:07.4154030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgb065zyf/_remote_module_non_scriptable.py
2022-05-18T04:49:07.4154908Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:49:07.4164834Z 
2022-05-18T04:49:07.4165210Z Running tests...
2022-05-18T04:49:07.4165674Z ----------------------------------------------------------------------
2022-05-18T04:49:09.0764806Z   test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:09.1220440Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101397
2022-05-18T04:49:09.1355740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101398
2022-05-18T04:49:09.1490279Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101399
2022-05-18T04:49:09.1614079Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101400
2022-05-18T04:49:10.0668027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp9s2d549
2022-05-18T04:49:10.1019047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp9s2d549/_remote_module_non_scriptable.py
2022-05-18T04:49:10.1019676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzikdzbob
2022-05-18T04:49:10.1020242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzikdzbob/_remote_module_non_scriptable.py
2022-05-18T04:49:10.1194616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8gr89jz
2022-05-18T04:49:10.1195235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8gr89jz/_remote_module_non_scriptable.py
2022-05-18T04:49:10.1290309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8po1b2l
2022-05-18T04:49:10.1290915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8po1b2l/_remote_module_non_scriptable.py
2022-05-18T04:49:10.4678077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:10.5022221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:10.5205056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:49:10.5445539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:49:10.7670017Z ok (3.350s)
2022-05-18T04:49:10.7670245Z 
2022-05-18T04:49:10.7670675Z ----------------------------------------------------------------------
2022-05-18T04:49:10.7671054Z Ran 1 test in 3.351s
2022-05-18T04:49:10.7671229Z 
2022-05-18T04:49:10.7671334Z OK
2022-05-18T04:49:10.7671481Z 
2022-05-18T04:49:10.7671628Z Generating XML reports...
2022-05-18T04:49:10.7717658Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044907.xml
2022-05-18T04:49:12.0312828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwt5vpuu
2022-05-18T04:49:12.0313460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwt5vpuu/_remote_module_non_scriptable.py
2022-05-18T04:49:12.4591261Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:49:12.4601083Z 
2022-05-18T04:49:12.4601412Z Running tests...
2022-05-18T04:49:12.4601896Z ----------------------------------------------------------------------
2022-05-18T04:49:14.1204169Z   test_meta_multiple_tensors (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:14.1664015Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101740
2022-05-18T04:49:14.1797210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101741
2022-05-18T04:49:14.1922761Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101742
2022-05-18T04:49:14.2054693Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101743
2022-05-18T04:49:15.1536592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjr4vqhxe
2022-05-18T04:49:15.1537355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjr4vqhxe/_remote_module_non_scriptable.py
2022-05-18T04:49:15.1583062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9banlst
2022-05-18T04:49:15.1585279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9banlst/_remote_module_non_scriptable.py
2022-05-18T04:49:15.1628119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwb7b9csr
2022-05-18T04:49:15.1628700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwb7b9csr/_remote_module_non_scriptable.py
2022-05-18T04:49:15.1699047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpecglhte0
2022-05-18T04:49:15.1699657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpecglhte0/_remote_module_non_scriptable.py
2022-05-18T04:49:15.5538770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:49:15.5688859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:15.5689531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:49:15.5723816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:19.0185036Z ok (6.558s)
2022-05-18T04:49:19.0185293Z 
2022-05-18T04:49:19.0185712Z ----------------------------------------------------------------------
2022-05-18T04:49:19.0186145Z Ran 1 test in 6.558s
2022-05-18T04:49:19.0186347Z 
2022-05-18T04:49:19.0186449Z OK
2022-05-18T04:49:19.0186591Z 
2022-05-18T04:49:19.0186734Z Generating XML reports...
2022-05-18T04:49:19.0231360Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044912.xml
2022-05-18T04:49:20.2657878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_4yelom
2022-05-18T04:49:20.2658777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_4yelom/_remote_module_non_scriptable.py
2022-05-18T04:49:20.6802732Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:49:20.6813170Z 
2022-05-18T04:49:20.6813647Z Running tests...
2022-05-18T04:49:20.6814589Z ----------------------------------------------------------------------
2022-05-18T04:49:22.3132607Z   test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:22.3594594Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102243
2022-05-18T04:49:22.3723945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102244
2022-05-18T04:49:22.3863564Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102245
2022-05-18T04:49:22.3987557Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102246
2022-05-18T04:49:23.3320321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpim0172gy
2022-05-18T04:49:23.3320972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpim0172gy/_remote_module_non_scriptable.py
2022-05-18T04:49:23.3832144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79hmjhjz
2022-05-18T04:49:23.3832760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79hmjhjz/_remote_module_non_scriptable.py
2022-05-18T04:49:23.3838832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9ralln7
2022-05-18T04:49:23.3839436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9ralln7/_remote_module_non_scriptable.py
2022-05-18T04:49:23.3851094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6i2egqtm
2022-05-18T04:49:23.3851685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6i2egqtm/_remote_module_non_scriptable.py
2022-05-18T04:49:23.7362197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:23.7862461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:23.7863106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:49:23.7939553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:49:28.7143142Z ok (8.033s)
2022-05-18T04:49:28.7143393Z 
2022-05-18T04:49:28.7143797Z ----------------------------------------------------------------------
2022-05-18T04:49:28.7144169Z Ran 1 test in 8.033s
2022-05-18T04:49:28.7144346Z 
2022-05-18T04:49:28.7145081Z OK
2022-05-18T04:49:28.7145231Z 
2022-05-18T04:49:28.7145391Z Generating XML reports...
2022-05-18T04:49:28.7188142Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044920.xml
2022-05-18T04:49:29.9842236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdsbwuvso
2022-05-18T04:49:30.4090622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdsbwuvso/_remote_module_non_scriptable.py
2022-05-18T04:49:30.4091509Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:49:30.4101437Z 
2022-05-18T04:49:30.4101751Z Running tests...
2022-05-18T04:49:30.4102720Z ----------------------------------------------------------------------
2022-05-18T04:49:32.0607423Z   test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:32.1060214Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102617
2022-05-18T04:49:32.1194940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102618
2022-05-18T04:49:32.1327715Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102619
2022-05-18T04:49:32.1452056Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102620
2022-05-18T04:49:33.1147418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexdrnvep
2022-05-18T04:49:33.1148036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexdrnvep/_remote_module_non_scriptable.py
2022-05-18T04:49:33.1247612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_4b2ah_
2022-05-18T04:49:33.1248205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_4b2ah_/_remote_module_non_scriptable.py
2022-05-18T04:49:33.1477207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptov16nip
2022-05-18T04:49:33.1477817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptov16nip/_remote_module_non_scriptable.py
2022-05-18T04:49:33.1753650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsy78b164
2022-05-18T04:49:33.1754260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsy78b164/_remote_module_non_scriptable.py
2022-05-18T04:49:33.5156177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:33.5266330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:49:33.5527195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:33.5852092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:49:40.0639202Z ok (9.653s)
2022-05-18T04:49:40.0639534Z 
2022-05-18T04:49:40.0640073Z ----------------------------------------------------------------------
2022-05-18T04:49:40.0640438Z Ran 1 test in 9.654s
2022-05-18T04:49:40.0640625Z 
2022-05-18T04:49:40.0640737Z OK
2022-05-18T04:49:40.0640881Z 
2022-05-18T04:49:40.0641024Z Generating XML reports...
2022-05-18T04:49:40.0683054Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044930.xml
2022-05-18T04:49:41.3081560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mcgopyt
2022-05-18T04:49:41.3082197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mcgopyt/_remote_module_non_scriptable.py
2022-05-18T04:49:41.7202920Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:49:41.7212423Z 
2022-05-18T04:49:41.7212623Z Running tests...
2022-05-18T04:49:41.7213149Z ----------------------------------------------------------------------
2022-05-18T04:49:43.3299950Z   test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:43.3760264Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102992
2022-05-18T04:49:43.3899472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102993
2022-05-18T04:49:43.4036231Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102994
2022-05-18T04:49:43.4161175Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102995
2022-05-18T04:49:44.3591463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl78_brxv
2022-05-18T04:49:44.3592101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl78_brxv/_remote_module_non_scriptable.py
2022-05-18T04:49:44.3944907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7bbq_7_d
2022-05-18T04:49:44.3945898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7bbq_7_d/_remote_module_non_scriptable.py
2022-05-18T04:49:44.4152889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4r9wzr0
2022-05-18T04:49:44.4153498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4r9wzr0/_remote_module_non_scriptable.py
2022-05-18T04:49:44.4166236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_w24vyo9
2022-05-18T04:49:44.4166808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_w24vyo9/_remote_module_non_scriptable.py
2022-05-18T04:49:44.7800033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:44.7970347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:49:44.8222670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:49:44.8279326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:49:51.2341946Z ok (9.513s)
2022-05-18T04:49:51.2342695Z 
2022-05-18T04:49:51.2343140Z ----------------------------------------------------------------------
2022-05-18T04:49:51.2343516Z Ran 1 test in 9.513s
2022-05-18T04:49:51.2343692Z 
2022-05-18T04:49:51.2343811Z OK
2022-05-18T04:49:51.2343957Z 
2022-05-18T04:49:51.2344076Z Generating XML reports...
2022-05-18T04:49:51.2388682Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044941.xml
2022-05-18T04:49:52.5062677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6txcbcb7
2022-05-18T04:49:52.5063333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6txcbcb7/_remote_module_non_scriptable.py
2022-05-18T04:49:52.9386903Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:49:52.9398982Z 
2022-05-18T04:49:52.9399272Z Running tests...
2022-05-18T04:49:52.9399766Z ----------------------------------------------------------------------
2022-05-18T04:49:54.5872767Z   test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:49:54.6344524Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103367
2022-05-18T04:49:54.6484399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103368
2022-05-18T04:49:54.6639338Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103369
2022-05-18T04:49:54.6767666Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103370
2022-05-18T04:49:55.6819094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08_7w5dp
2022-05-18T04:49:55.6819725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08_7w5dp/_remote_module_non_scriptable.py
2022-05-18T04:49:55.6842569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsrnim81
2022-05-18T04:49:55.6843179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsrnim81/_remote_module_non_scriptable.py
2022-05-18T04:49:55.6865815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcn011a_e
2022-05-18T04:49:55.6867604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcn011a_e/_remote_module_non_scriptable.py
2022-05-18T04:49:55.6897638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfehs2ih
2022-05-18T04:49:55.6898255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfehs2ih/_remote_module_non_scriptable.py
2022-05-18T04:49:56.0867246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:49:56.0867822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:49:56.1024958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:49:56.1040434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:00.9920581Z ok (8.052s)
2022-05-18T04:50:00.9920791Z 
2022-05-18T04:50:00.9921256Z ----------------------------------------------------------------------
2022-05-18T04:50:00.9922062Z Ran 1 test in 8.052s
2022-05-18T04:50:00.9922235Z 
2022-05-18T04:50:00.9922341Z OK
2022-05-18T04:50:00.9922487Z 
2022-05-18T04:50:00.9922616Z Generating XML reports...
2022-05-18T04:50:00.9967867Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044952.xml
2022-05-18T04:50:02.2795195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5sm9kaaq
2022-05-18T04:50:02.2795820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5sm9kaaq/_remote_module_non_scriptable.py
2022-05-18T04:50:02.7018336Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:50:02.7027916Z 
2022-05-18T04:50:02.7028198Z Running tests...
2022-05-18T04:50:02.7028664Z ----------------------------------------------------------------------
2022-05-18T04:50:04.3394453Z   test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:04.3824100Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103741
2022-05-18T04:50:04.3963005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103742
2022-05-18T04:50:04.4109989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103743
2022-05-18T04:50:04.4257155Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103744
2022-05-18T04:50:05.3465936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw09ig6u_
2022-05-18T04:50:05.3467165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw09ig6u_/_remote_module_non_scriptable.py
2022-05-18T04:50:05.3929229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzwbt5cf
2022-05-18T04:50:05.3930339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzwbt5cf/_remote_module_non_scriptable.py
2022-05-18T04:50:05.3938819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yh4dbb3
2022-05-18T04:50:05.3939944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yh4dbb3/_remote_module_non_scriptable.py
2022-05-18T04:50:05.4089086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_wc3yrtm
2022-05-18T04:50:05.4090202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_wc3yrtm/_remote_module_non_scriptable.py
2022-05-18T04:50:05.7460506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:50:05.7926148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:05.8003727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:05.8159586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:50:18.6576673Z ok (15.955s)
2022-05-18T04:50:18.6576965Z 
2022-05-18T04:50:18.6577418Z ----------------------------------------------------------------------
2022-05-18T04:50:18.6577779Z Ran 1 test in 15.955s
2022-05-18T04:50:18.6577954Z 
2022-05-18T04:50:18.6578057Z OK
2022-05-18T04:50:18.6581160Z 
2022-05-18T04:50:18.6581577Z Generating XML reports...
2022-05-18T04:50:18.6625078Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045002.xml
2022-05-18T04:50:19.9407309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpko7rfzzl
2022-05-18T04:50:19.9407970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpko7rfzzl/_remote_module_non_scriptable.py
2022-05-18T04:50:20.3735872Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:50:20.3746668Z 
2022-05-18T04:50:20.3746887Z Running tests...
2022-05-18T04:50:20.3747590Z ----------------------------------------------------------------------
2022-05-18T04:50:22.0325266Z   test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:22.0794908Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104244
2022-05-18T04:50:22.0929889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104245
2022-05-18T04:50:22.1082461Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104246
2022-05-18T04:50:22.1208306Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104247
2022-05-18T04:50:23.0411373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82iu35ca
2022-05-18T04:50:23.0412548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82iu35ca/_remote_module_non_scriptable.py
2022-05-18T04:50:23.0510373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusaf3_kn
2022-05-18T04:50:23.0511548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusaf3_kn/_remote_module_non_scriptable.py
2022-05-18T04:50:23.1271569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2h_du0h
2022-05-18T04:50:23.1272697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2h_du0h/_remote_module_non_scriptable.py
2022-05-18T04:50:23.1381842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpczywgzs6
2022-05-18T04:50:23.1383357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpczywgzs6/_remote_module_non_scriptable.py
2022-05-18T04:50:23.4529174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:23.4548260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:50:23.5418386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:23.5550086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:50:39.1588945Z ok (18.784s)
2022-05-18T04:50:39.1589165Z 
2022-05-18T04:50:39.1589598Z ----------------------------------------------------------------------
2022-05-18T04:50:39.1592407Z Ran 1 test in 18.784s
2022-05-18T04:50:39.1592994Z 
2022-05-18T04:50:39.1593184Z OK
2022-05-18T04:50:39.1593345Z 
2022-05-18T04:50:39.1593500Z Generating XML reports...
2022-05-18T04:50:39.1634665Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045020.xml
2022-05-18T04:50:40.4334409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82mlsbd9
2022-05-18T04:50:40.4335055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82mlsbd9/_remote_module_non_scriptable.py
2022-05-18T04:50:40.8600519Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:50:40.8611648Z 
2022-05-18T04:50:40.8611959Z Running tests...
2022-05-18T04:50:40.8612422Z ----------------------------------------------------------------------
2022-05-18T04:50:42.5293545Z   test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:50:42.5740728Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104753
2022-05-18T04:50:42.5867109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104754
2022-05-18T04:50:42.5997731Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104755
2022-05-18T04:50:42.6120121Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104756
2022-05-18T04:50:43.5313730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu46ityl3
2022-05-18T04:50:43.5314891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu46ityl3/_remote_module_non_scriptable.py
2022-05-18T04:50:43.5410421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnu87iatm
2022-05-18T04:50:43.5411518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnu87iatm/_remote_module_non_scriptable.py
2022-05-18T04:50:43.5459065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphyn8ymj4
2022-05-18T04:50:43.5460256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphyn8ymj4/_remote_module_non_scriptable.py
2022-05-18T04:50:43.5461377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpho46cd3z
2022-05-18T04:50:43.5466783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpho46cd3z/_remote_module_non_scriptable.py
2022-05-18T04:50:43.9417975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:50:43.9461071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:50:43.9545840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:50:43.9623478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:50:57.0440143Z ok (16.183s)
2022-05-18T04:50:57.0440383Z 
2022-05-18T04:50:57.0440844Z ----------------------------------------------------------------------
2022-05-18T04:50:57.0441194Z Ran 1 test in 16.183s
2022-05-18T04:50:57.0441338Z 
2022-05-18T04:50:57.0444103Z OK
2022-05-18T04:50:57.0444296Z 
2022-05-18T04:50:57.0444568Z Generating XML reports...
2022-05-18T04:50:57.0484705Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045040.xml
2022-05-18T04:50:58.3123710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshec9vs4
2022-05-18T04:50:58.3124330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshec9vs4/_remote_module_non_scriptable.py
2022-05-18T04:50:58.7418184Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:50:58.7429782Z 
2022-05-18T04:50:58.7430115Z Running tests...
2022-05-18T04:50:58.7430596Z ----------------------------------------------------------------------
2022-05-18T04:51:00.4108265Z   test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:00.4578917Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105256
2022-05-18T04:51:00.4717891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105257
2022-05-18T04:51:00.4844357Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105258
2022-05-18T04:51:00.4973783Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105259
2022-05-18T04:51:01.4329942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bt2qaew
2022-05-18T04:51:01.4330630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bt2qaew/_remote_module_non_scriptable.py
2022-05-18T04:51:01.4517753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mfbyrfr
2022-05-18T04:51:01.4518351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mfbyrfr/_remote_module_non_scriptable.py
2022-05-18T04:51:01.4535171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5h3fikyk
2022-05-18T04:51:01.4535750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5h3fikyk/_remote_module_non_scriptable.py
2022-05-18T04:51:01.4781074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaz4_yqy_
2022-05-18T04:51:01.4781678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaz4_yqy_/_remote_module_non_scriptable.py
2022-05-18T04:51:01.8383742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:51:01.8619138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:51:01.8619694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:01.8814078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:17.2357804Z ok (18.492s)
2022-05-18T04:51:17.2358045Z 
2022-05-18T04:51:17.2358476Z ----------------------------------------------------------------------
2022-05-18T04:51:17.2358835Z Ran 1 test in 18.493s
2022-05-18T04:51:17.2358983Z 
2022-05-18T04:51:17.2363288Z OK
2022-05-18T04:51:17.2363445Z 
2022-05-18T04:51:17.2364011Z Generating XML reports...
2022-05-18T04:51:17.2402228Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045058.xml
2022-05-18T04:51:18.4793320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvh5wi6u
2022-05-18T04:51:18.4793964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvh5wi6u/_remote_module_non_scriptable.py
2022-05-18T04:51:18.9097318Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:51:18.9110095Z 
2022-05-18T04:51:18.9110390Z Running tests...
2022-05-18T04:51:18.9110867Z ----------------------------------------------------------------------
2022-05-18T04:51:20.6037422Z   test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:20.6502986Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105765
2022-05-18T04:51:20.6640020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105766
2022-05-18T04:51:20.6784212Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105767
2022-05-18T04:51:20.6914683Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105768
2022-05-18T04:51:21.6091561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpef6r_0je
2022-05-18T04:51:21.6092242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpef6r_0je/_remote_module_non_scriptable.py
2022-05-18T04:51:21.6516652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppq98v_06
2022-05-18T04:51:21.6517264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppq98v_06/_remote_module_non_scriptable.py
2022-05-18T04:51:21.6807245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq0j3brza
2022-05-18T04:51:21.6807881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq0j3brza/_remote_module_non_scriptable.py
2022-05-18T04:51:21.6945332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4uccmkym
2022-05-18T04:51:21.6945928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4uccmkym/_remote_module_non_scriptable.py
2022-05-18T04:51:22.0126644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:51:22.0565731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:51:22.0979218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:22.1055466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:35.1233846Z ok (16.212s)
2022-05-18T04:51:35.1234179Z 
2022-05-18T04:51:35.1234622Z ----------------------------------------------------------------------
2022-05-18T04:51:35.1234973Z Ran 1 test in 16.212s
2022-05-18T04:51:35.1235169Z 
2022-05-18T04:51:35.1235271Z OK
2022-05-18T04:51:35.1238406Z 
2022-05-18T04:51:35.1238948Z Generating XML reports...
2022-05-18T04:51:35.1279156Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045118.xml
2022-05-18T04:51:36.4334994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3d0yva6
2022-05-18T04:51:36.4335597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3d0yva6/_remote_module_non_scriptable.py
2022-05-18T04:51:36.8654404Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:51:36.8663640Z 
2022-05-18T04:51:36.8663799Z Running tests...
2022-05-18T04:51:36.8664327Z ----------------------------------------------------------------------
2022-05-18T04:51:38.5295062Z   test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:38.5761416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106268
2022-05-18T04:51:38.5893010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106269
2022-05-18T04:51:38.6039318Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106270
2022-05-18T04:51:38.6165867Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106271
2022-05-18T04:51:39.5733555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8s37rysv
2022-05-18T04:51:39.5734194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8s37rysv/_remote_module_non_scriptable.py
2022-05-18T04:51:39.5744437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4qq1033
2022-05-18T04:51:39.5745400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4qq1033/_remote_module_non_scriptable.py
2022-05-18T04:51:39.5960635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdafvjsy9
2022-05-18T04:51:39.5961254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdafvjsy9/_remote_module_non_scriptable.py
2022-05-18T04:51:39.5961787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qgrmay3
2022-05-18T04:51:39.5965406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qgrmay3/_remote_module_non_scriptable.py
2022-05-18T04:51:39.9789262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:51:39.9812958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:39.9969857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:51:39.9979307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:51:51.4460249Z ok (14.579s)
2022-05-18T04:51:51.4460672Z 
2022-05-18T04:51:51.4461383Z ----------------------------------------------------------------------
2022-05-18T04:51:51.4462445Z Ran 1 test in 14.580s
2022-05-18T04:51:51.4462781Z 
2022-05-18T04:51:51.4462974Z OK
2022-05-18T04:51:51.4463325Z 
2022-05-18T04:51:51.4463591Z Generating XML reports...
2022-05-18T04:51:51.4508075Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045136.xml
2022-05-18T04:51:52.7188483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pvdmnk1
2022-05-18T04:51:52.7189337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pvdmnk1/_remote_module_non_scriptable.py
2022-05-18T04:51:53.1366188Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:51:53.1375575Z 
2022-05-18T04:51:53.1376551Z Running tests...
2022-05-18T04:51:53.1378300Z ----------------------------------------------------------------------
2022-05-18T04:51:54.7644193Z   test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:51:54.8090978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106770
2022-05-18T04:51:54.8223340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106771
2022-05-18T04:51:54.8348603Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106772
2022-05-18T04:51:54.8478281Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106773
2022-05-18T04:51:55.7416278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwu3yt13z
2022-05-18T04:51:55.7417442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwu3yt13z/_remote_module_non_scriptable.py
2022-05-18T04:51:55.7734082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxpwgdjv
2022-05-18T04:51:55.7735521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxpwgdjv/_remote_module_non_scriptable.py
2022-05-18T04:51:55.7842896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdhd7pa2
2022-05-18T04:51:55.7847008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdhd7pa2/_remote_module_non_scriptable.py
2022-05-18T04:51:55.7889746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkv64c5b3
2022-05-18T04:51:55.7890870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkv64c5b3/_remote_module_non_scriptable.py
2022-05-18T04:51:56.1607502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:51:56.1765090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:51:56.1883435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:51:56.1975262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:08.1772059Z ok (15.039s)
2022-05-18T04:52:08.1772273Z 
2022-05-18T04:52:08.1772796Z ----------------------------------------------------------------------
2022-05-18T04:52:08.1773173Z Ran 1 test in 15.040s
2022-05-18T04:52:08.1773348Z 
2022-05-18T04:52:08.1773451Z OK
2022-05-18T04:52:08.1773612Z 
2022-05-18T04:52:08.1773733Z Generating XML reports...
2022-05-18T04:52:08.1817892Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045153.xml
2022-05-18T04:52:09.4342871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdrd7yeao
2022-05-18T04:52:09.4343524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdrd7yeao/_remote_module_non_scriptable.py
2022-05-18T04:52:09.8514217Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:52:09.8524806Z 
2022-05-18T04:52:09.8525158Z Running tests...
2022-05-18T04:52:09.8525630Z ----------------------------------------------------------------------
2022-05-18T04:52:11.5123459Z   test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:11.5599163Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107275
2022-05-18T04:52:11.5735774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107276
2022-05-18T04:52:11.5884686Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107277
2022-05-18T04:52:11.6010924Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107278
2022-05-18T04:52:12.5180979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7otwghe
2022-05-18T04:52:12.5181587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7otwghe/_remote_module_non_scriptable.py
2022-05-18T04:52:12.5205101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfiwf_db0
2022-05-18T04:52:12.5205676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfiwf_db0/_remote_module_non_scriptable.py
2022-05-18T04:52:12.5583831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pmlopiw
2022-05-18T04:52:12.5584743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pmlopiw/_remote_module_non_scriptable.py
2022-05-18T04:52:12.5691473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7ycb7ip
2022-05-18T04:52:12.5692074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7ycb7ip/_remote_module_non_scriptable.py
2022-05-18T04:52:12.9263060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:52:12.9324016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:12.9617072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:12.9887708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:52:24.8300725Z ok (14.977s)
2022-05-18T04:52:24.8300975Z 
2022-05-18T04:52:24.8301418Z ----------------------------------------------------------------------
2022-05-18T04:52:24.8301788Z Ran 1 test in 14.978s
2022-05-18T04:52:24.8302316Z 
2022-05-18T04:52:24.8302404Z OK
2022-05-18T04:52:24.8302545Z 
2022-05-18T04:52:24.8305502Z Generating XML reports...
2022-05-18T04:52:24.8346455Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045209.xml
2022-05-18T04:52:26.0951526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67_vo0sd
2022-05-18T04:52:26.0952145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67_vo0sd/_remote_module_non_scriptable.py
2022-05-18T04:52:26.5129375Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:52:26.5139150Z 
2022-05-18T04:52:26.5139574Z Running tests...
2022-05-18T04:52:26.5140204Z ----------------------------------------------------------------------
2022-05-18T04:52:28.1424136Z   test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:28.1899346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107780
2022-05-18T04:52:28.2035281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107781
2022-05-18T04:52:28.2181170Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107782
2022-05-18T04:52:28.2307701Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107783
2022-05-18T04:52:29.2181055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ucb58gm
2022-05-18T04:52:29.2181732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ucb58gm/_remote_module_non_scriptable.py
2022-05-18T04:52:29.2344399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3xvc6a5
2022-05-18T04:52:29.2344986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3xvc6a5/_remote_module_non_scriptable.py
2022-05-18T04:52:29.2518855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmu3vzbhl
2022-05-18T04:52:29.2519451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmu3vzbhl/_remote_module_non_scriptable.py
2022-05-18T04:52:29.3014954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyoce3o9
2022-05-18T04:52:29.3015630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyoce3o9/_remote_module_non_scriptable.py
2022-05-18T04:52:29.6318962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:29.6409192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:29.6542785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:52:29.7185298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:52:41.1597873Z ok (14.646s)
2022-05-18T04:52:41.1598554Z 
2022-05-18T04:52:41.1598988Z ----------------------------------------------------------------------
2022-05-18T04:52:41.1599335Z Ran 1 test in 14.646s
2022-05-18T04:52:41.1599500Z 
2022-05-18T04:52:41.1599595Z OK
2022-05-18T04:52:41.1599713Z 
2022-05-18T04:52:41.1601503Z Generating XML reports...
2022-05-18T04:52:41.1644253Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045226.xml
2022-05-18T04:52:42.4130082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_51ll8a
2022-05-18T04:52:42.4131042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_51ll8a/_remote_module_non_scriptable.py
2022-05-18T04:52:42.8225644Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:52:42.8236856Z 
2022-05-18T04:52:42.8237045Z Running tests...
2022-05-18T04:52:42.8237528Z ----------------------------------------------------------------------
2022-05-18T04:52:44.4405709Z   test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:52:44.4866625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108282
2022-05-18T04:52:44.5010339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108283
2022-05-18T04:52:44.5144870Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108284
2022-05-18T04:52:44.5285135Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108285
2022-05-18T04:52:45.4885791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpmluqz7
2022-05-18T04:52:45.4886435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpmluqz7/_remote_module_non_scriptable.py
2022-05-18T04:52:45.5352024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtvirm70
2022-05-18T04:52:45.5352637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtvirm70/_remote_module_non_scriptable.py
2022-05-18T04:52:45.5395240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6n8zenkq
2022-05-18T04:52:45.5395850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6n8zenkq/_remote_module_non_scriptable.py
2022-05-18T04:52:45.5446343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmu37s0i
2022-05-18T04:52:45.5446942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmu37s0i/_remote_module_non_scriptable.py
2022-05-18T04:52:45.8878390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:52:45.9350884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:52:45.9389694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:52:45.9523418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:52:59.1611581Z ok (16.337s)
2022-05-18T04:52:59.1612128Z 
2022-05-18T04:52:59.1612890Z ----------------------------------------------------------------------
2022-05-18T04:52:59.1613244Z Ran 1 test in 16.337s
2022-05-18T04:52:59.1613485Z 
2022-05-18T04:52:59.1613577Z OK
2022-05-18T04:52:59.1613693Z 
2022-05-18T04:52:59.1613831Z Generating XML reports...
2022-05-18T04:52:59.1658215Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045242.xml
2022-05-18T04:53:00.4499795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwj80_4hg
2022-05-18T04:53:00.4500511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwj80_4hg/_remote_module_non_scriptable.py
2022-05-18T04:53:00.8871530Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:53:00.8881765Z 
2022-05-18T04:53:00.8882284Z Running tests...
2022-05-18T04:53:00.8882791Z ----------------------------------------------------------------------
2022-05-18T04:53:02.5855590Z   test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:02.6329961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108785
2022-05-18T04:53:02.6474820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108786
2022-05-18T04:53:02.6628519Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108787
2022-05-18T04:53:02.6755108Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108788
2022-05-18T04:53:03.6189512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpid6i_ntu
2022-05-18T04:53:03.6190182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpid6i_ntu/_remote_module_non_scriptable.py
2022-05-18T04:53:03.6405766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmid0xc2l
2022-05-18T04:53:03.6406380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmid0xc2l/_remote_module_non_scriptable.py
2022-05-18T04:53:03.6424617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwo3537y
2022-05-18T04:53:03.6425524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwo3537y/_remote_module_non_scriptable.py
2022-05-18T04:53:03.6624271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpii74l9se
2022-05-18T04:53:03.6624897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpii74l9se/_remote_module_non_scriptable.py
2022-05-18T04:53:04.0222294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:04.0488439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:04.0494516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:53:04.0717403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:53:19.1135670Z ok (18.225s)
2022-05-18T04:53:19.1135904Z 
2022-05-18T04:53:19.1136336Z ----------------------------------------------------------------------
2022-05-18T04:53:19.1136689Z Ran 1 test in 18.225s
2022-05-18T04:53:19.1136837Z 
2022-05-18T04:53:19.1136940Z OK
2022-05-18T04:53:19.1137084Z 
2022-05-18T04:53:19.1137222Z Generating XML reports...
2022-05-18T04:53:19.1180309Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045300.xml
2022-05-18T04:53:20.3873765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6n223zx
2022-05-18T04:53:20.3874423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6n223zx/_remote_module_non_scriptable.py
2022-05-18T04:53:20.8035450Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:53:20.8045761Z 
2022-05-18T04:53:20.8046277Z Running tests...
2022-05-18T04:53:20.8046915Z ----------------------------------------------------------------------
2022-05-18T04:53:22.4318745Z   test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:22.4773365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109294
2022-05-18T04:53:22.4913926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109295
2022-05-18T04:53:22.5051474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109296
2022-05-18T04:53:22.5175390Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109297
2022-05-18T04:53:23.4664779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcs5q7v0
2022-05-18T04:53:23.4665684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcs5q7v0/_remote_module_non_scriptable.py
2022-05-18T04:53:23.4691435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu54hvmqa
2022-05-18T04:53:23.4692012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu54hvmqa/_remote_module_non_scriptable.py
2022-05-18T04:53:23.4845316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33t0lwmk
2022-05-18T04:53:23.4845914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33t0lwmk/_remote_module_non_scriptable.py
2022-05-18T04:53:23.5122181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsij_mwcu
2022-05-18T04:53:23.5122783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsij_mwcu/_remote_module_non_scriptable.py
2022-05-18T04:53:23.8667563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:23.8820585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:23.8923247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:53:23.9263813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:53:36.9491495Z ok (16.144s)
2022-05-18T04:53:36.9491809Z 
2022-05-18T04:53:36.9492254Z ----------------------------------------------------------------------
2022-05-18T04:53:36.9492617Z Ran 1 test in 16.145s
2022-05-18T04:53:36.9492792Z 
2022-05-18T04:53:36.9492867Z OK
2022-05-18T04:53:36.9496217Z 
2022-05-18T04:53:36.9496665Z Generating XML reports...
2022-05-18T04:53:36.9535362Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045320.xml
2022-05-18T04:53:38.2161914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfs562jmz
2022-05-18T04:53:38.2162557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfs562jmz/_remote_module_non_scriptable.py
2022-05-18T04:53:38.6470359Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:53:38.6480084Z 
2022-05-18T04:53:38.6480556Z Running tests...
2022-05-18T04:53:38.6481079Z ----------------------------------------------------------------------
2022-05-18T04:53:40.3224046Z   test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:53:40.3684078Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109797
2022-05-18T04:53:40.3824015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109798
2022-05-18T04:53:40.3964308Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109799
2022-05-18T04:53:40.4086980Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109800
2022-05-18T04:53:41.3075813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfhvkf11
2022-05-18T04:53:41.3077018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfhvkf11/_remote_module_non_scriptable.py
2022-05-18T04:53:41.3268936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps7vfryhv
2022-05-18T04:53:41.3270025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps7vfryhv/_remote_module_non_scriptable.py
2022-05-18T04:53:41.3681646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeko9qm23
2022-05-18T04:53:41.3682785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeko9qm23/_remote_module_non_scriptable.py
2022-05-18T04:53:41.3703478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw32t6nub
2022-05-18T04:53:41.3704590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw32t6nub/_remote_module_non_scriptable.py
2022-05-18T04:53:41.7173248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:53:41.7466551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:53:41.7748297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:53:41.7787682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:53:57.2447036Z ok (18.596s)
2022-05-18T04:53:57.2447274Z 
2022-05-18T04:53:57.2447710Z ----------------------------------------------------------------------
2022-05-18T04:53:57.2448067Z Ran 1 test in 18.597s
2022-05-18T04:53:57.2452218Z 
2022-05-18T04:53:57.2453080Z OK
2022-05-18T04:53:57.2453402Z 
2022-05-18T04:53:57.2453654Z Generating XML reports...
2022-05-18T04:53:57.2491469Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045338.xml
2022-05-18T04:53:58.5234871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpun5xqr_1
2022-05-18T04:53:58.5235505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpun5xqr_1/_remote_module_non_scriptable.py
2022-05-18T04:53:58.9556885Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:53:58.9567480Z 
2022-05-18T04:53:58.9567685Z Running tests...
2022-05-18T04:53:58.9568403Z ----------------------------------------------------------------------
2022-05-18T04:54:00.6433157Z   test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:00.6900113Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110306
2022-05-18T04:54:00.7034025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110307
2022-05-18T04:54:00.7171642Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110308
2022-05-18T04:54:00.7304497Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110309
2022-05-18T04:54:01.7059072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08llgcq9
2022-05-18T04:54:01.7059708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08llgcq9/_remote_module_non_scriptable.py
2022-05-18T04:54:01.7067060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3xtdpb3g
2022-05-18T04:54:01.7067716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3xtdpb3g/_remote_module_non_scriptable.py
2022-05-18T04:54:01.7093328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpza919nvb
2022-05-18T04:54:01.7093940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpza919nvb/_remote_module_non_scriptable.py
2022-05-18T04:54:01.7813462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxczkr15w
2022-05-18T04:54:01.7814088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxczkr15w/_remote_module_non_scriptable.py
2022-05-18T04:54:02.1201847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:02.1202693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:02.1217329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:54:02.2014058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:54:05.5438766Z ok (6.587s)
2022-05-18T04:54:05.5439022Z 
2022-05-18T04:54:05.5439469Z ----------------------------------------------------------------------
2022-05-18T04:54:05.5439873Z Ran 1 test in 6.587s
2022-05-18T04:54:05.5440051Z 
2022-05-18T04:54:05.5440145Z OK
2022-05-18T04:54:05.5440262Z 
2022-05-18T04:54:05.5440398Z Generating XML reports...
2022-05-18T04:54:05.5484918Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045358.xml
2022-05-18T04:54:06.8263710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7kub6few
2022-05-18T04:54:06.8264368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7kub6few/_remote_module_non_scriptable.py
2022-05-18T04:54:07.2619187Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:54:07.2630937Z 
2022-05-18T04:54:07.2631273Z Running tests...
2022-05-18T04:54:07.2631749Z ----------------------------------------------------------------------
2022-05-18T04:54:08.9217118Z   test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:08.9694021Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110813
2022-05-18T04:54:08.9832529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110814
2022-05-18T04:54:08.9977232Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110815
2022-05-18T04:54:09.0103351Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110816
2022-05-18T04:54:10.0109330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpco_v4jje
2022-05-18T04:54:10.0109950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpco_v4jje/_remote_module_non_scriptable.py
2022-05-18T04:54:10.0111646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yg1tqgw
2022-05-18T04:54:10.0114074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yg1tqgw/_remote_module_non_scriptable.py
2022-05-18T04:54:10.0128123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8biy_3x
2022-05-18T04:54:10.0129238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8biy_3x/_remote_module_non_scriptable.py
2022-05-18T04:54:10.0151621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44uapa4w
2022-05-18T04:54:10.0152225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44uapa4w/_remote_module_non_scriptable.py
2022-05-18T04:54:10.4189741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:10.4197159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:10.4201360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:54:10.4241279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:54:16.0277455Z ok (8.764s)
2022-05-18T04:54:16.0277679Z 
2022-05-18T04:54:16.0278126Z ----------------------------------------------------------------------
2022-05-18T04:54:16.0278494Z Ran 1 test in 8.765s
2022-05-18T04:54:16.0278659Z 
2022-05-18T04:54:16.0278736Z OK
2022-05-18T04:54:16.0278877Z 
2022-05-18T04:54:16.0279023Z Generating XML reports...
2022-05-18T04:54:16.0324570Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045407.xml
2022-05-18T04:54:17.3243557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpksg5qz0t
2022-05-18T04:54:17.3244196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpksg5qz0t/_remote_module_non_scriptable.py
2022-05-18T04:54:17.7387951Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:54:17.7399389Z 
2022-05-18T04:54:17.7399577Z Running tests...
2022-05-18T04:54:17.7400079Z ----------------------------------------------------------------------
2022-05-18T04:54:19.3582807Z   test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:19.4032455Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111936
2022-05-18T04:54:19.4174102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111937
2022-05-18T04:54:19.4306279Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 111938
2022-05-18T04:54:19.4441699Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 111939
2022-05-18T04:54:20.3776373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp100ckq19
2022-05-18T04:54:20.3776990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp100ckq19/_remote_module_non_scriptable.py
2022-05-18T04:54:20.4104796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94ft413r
2022-05-18T04:54:20.4105409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94ft413r/_remote_module_non_scriptable.py
2022-05-18T04:54:20.4223666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vrfrd7r
2022-05-18T04:54:20.4224253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vrfrd7r/_remote_module_non_scriptable.py
2022-05-18T04:54:20.4816858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyq1czjk8
2022-05-18T04:54:20.4817446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyq1czjk8/_remote_module_non_scriptable.py
2022-05-18T04:54:20.7839675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:20.8102378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:20.8219867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:54:20.8846258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:54:24.3573590Z ok (6.617s)
2022-05-18T04:54:24.3573833Z 
2022-05-18T04:54:24.3574278Z ----------------------------------------------------------------------
2022-05-18T04:54:24.3574604Z Ran 1 test in 6.617s
2022-05-18T04:54:24.3577447Z 
2022-05-18T04:54:24.3577723Z OK
2022-05-18T04:54:24.3577910Z 
2022-05-18T04:54:24.3578056Z Generating XML reports...
2022-05-18T04:54:24.3618904Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220518045417.xml
2022-05-18T04:54:25.6211621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx7tc_jaw
2022-05-18T04:54:25.6212261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx7tc_jaw/_remote_module_non_scriptable.py
2022-05-18T04:54:26.0488441Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:54:26.0498667Z 
2022-05-18T04:54:26.0498931Z Running tests...
2022-05-18T04:54:26.0499424Z ----------------------------------------------------------------------
2022-05-18T04:54:27.7130467Z   test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:27.7583818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112591
2022-05-18T04:54:27.7715982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112592
2022-05-18T04:54:27.7857391Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 112593
2022-05-18T04:54:27.7980712Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 112594
2022-05-18T04:54:28.7299952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt76meonx
2022-05-18T04:54:28.7301180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt76meonx/_remote_module_non_scriptable.py
2022-05-18T04:54:28.7588491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6x_wrpc8
2022-05-18T04:54:28.7590431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6x_wrpc8/_remote_module_non_scriptable.py
2022-05-18T04:54:28.7601814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhfq8rf4
2022-05-18T04:54:28.7602704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhfq8rf4/_remote_module_non_scriptable.py
2022-05-18T04:54:28.7704640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf57zox2v
2022-05-18T04:54:28.7705238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf57zox2v/_remote_module_non_scriptable.py
2022-05-18T04:54:29.1331792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:29.1724429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:54:29.1751300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:29.1776722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:54:33.7132501Z ok (7.663s)
2022-05-18T04:54:33.7132776Z 
2022-05-18T04:54:33.7133238Z ----------------------------------------------------------------------
2022-05-18T04:54:33.7133606Z Ran 1 test in 7.663s
2022-05-18T04:54:33.7133752Z 
2022-05-18T04:54:33.7133867Z OK
2022-05-18T04:54:33.7134047Z 
2022-05-18T04:54:33.7134192Z Generating XML reports...
2022-05-18T04:54:33.7179726Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220518045426.xml
2022-05-18T04:54:34.9995459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_za9as4y
2022-05-18T04:54:34.9996113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_za9as4y/_remote_module_non_scriptable.py
2022-05-18T04:54:35.4307913Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T04:54:35.4319644Z 
2022-05-18T04:54:35.4319966Z Running tests...
2022-05-18T04:54:35.4320435Z ----------------------------------------------------------------------
2022-05-18T04:54:37.0932136Z   test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-05-18T04:54:37.1410565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113246
2022-05-18T04:54:37.1551481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113247
2022-05-18T04:54:37.1703359Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 113248
2022-05-18T04:54:37.1853015Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 113249
2022-05-18T04:54:38.1027057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcw_ervpc
2022-05-18T04:54:38.1027666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcw_ervpc/_remote_module_non_scriptable.py
2022-05-18T04:54:38.1143361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhgg7zky
2022-05-18T04:54:38.1145623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhgg7zky/_remote_module_non_scriptable.py
2022-05-18T04:54:38.1560496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmyx70kmk
2022-05-18T04:54:38.1690910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmyx70kmk/_remote_module_non_scriptable.py
2022-05-18T04:54:38.1691487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3o6b2fu
2022-05-18T04:54:38.1692045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3o6b2fu/_remote_module_non_scriptable.py
2022-05-18T04:54:38.5210535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:54:38.5300298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:54:38.5590461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:54:38.5704551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:54:43.1016217Z ok (7.669s)
2022-05-18T04:54:43.1016530Z 
2022-05-18T04:54:43.1016997Z ----------------------------------------------------------------------
2022-05-18T04:54:43.1017358Z Ran 1 test in 7.670s
2022-05-18T04:54:43.1017538Z 
2022-05-18T04:54:43.1017635Z OK
2022-05-18T04:54:43.1017752Z 
2022-05-18T04:54:43.1017901Z Generating XML reports...
2022-05-18T04:54:43.1062252Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220518045435.xml
2022-05-18T04:54:44.9945217Z 
2022-05-18T04:54:44.9945759Z real	18m13.976s
2022-05-18T04:54:44.9946477Z user	42m6.698s
2022-05-18T04:54:44.9946756Z sys	47m53.151s
2022-05-18T04:54:44.9946992Z + assert_git_not_dirty
2022-05-18T04:54:44.9947567Z + [[ linux-bionic-cuda10.2-py3.9-gcc7 != *rocm* ]]
2022-05-18T04:54:44.9948004Z + [[ linux-bionic-cuda10.2-py3.9-gcc7 != *xla* ]]
2022-05-18T04:54:44.9950827Z ++ git status --porcelain
2022-05-18T04:54:45.8733546Z + git_status=
2022-05-18T04:54:45.8734071Z + [[ -n '' ]]
2022-05-18T04:54:45.8734310Z + cleanup
2022-05-18T04:54:45.8734574Z + retcode=0
2022-05-18T04:54:45.8734808Z + set +x
2022-05-18T04:54:45.8735095Z EXITED_USER_LAND
2022-05-18T04:54:45.8826838Z ##[group]Run pytorch/pytorch/.github/actions/get-workflow-job-id@master
2022-05-18T04:54:45.8827215Z with:
2022-05-18T04:54:45.8827784Z   github-token: ***
2022-05-18T04:54:45.8828031Z env:
2022-05-18T04:54:45.8828233Z   IN_CI: 1
2022-05-18T04:54:45.8828459Z   IS_GHA: 1
2022-05-18T04:54:45.8828711Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:54:45.8828964Z   GPU_FLAG: --gpus all
2022-05-18T04:54:45.8829212Z ##[endgroup]
2022-05-18T04:54:45.8860565Z ##[group]Run nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a
2022-05-18T04:54:45.8860881Z with:
2022-05-18T04:54:45.8861084Z   shell: bash
2022-05-18T04:54:45.8861326Z   timeout_minutes: 10
2022-05-18T04:54:45.8861574Z   max_attempts: 5
2022-05-18T04:54:45.8861807Z   retry_wait_seconds: 30
2022-05-18T04:54:45.8862727Z   command: set -x
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}"

2022-05-18T04:54:45.8863236Z   polling_interval_seconds: 1
2022-05-18T04:54:45.8863509Z   warning_on_retry: true
2022-05-18T04:54:45.8863770Z   continue_on_error: false
2022-05-18T04:54:45.8863995Z env:
2022-05-18T04:54:45.8864210Z   IN_CI: 1
2022-05-18T04:54:45.8864437Z   IS_GHA: 1
2022-05-18T04:54:45.8864666Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:54:45.8864933Z   GPU_FLAG: --gpus all
2022-05-18T04:54:45.8865324Z   GITHUB_TOKEN: ***
2022-05-18T04:54:45.8865555Z ##[endgroup]
2022-05-18T04:54:45.9328389Z 
2022-05-18T04:54:45.9404702Z + python3 -m pip install requests==2.26.0
2022-05-18T04:54:47.2183355Z Defaulting to user installation because normal site-packages is not writeable
2022-05-18T04:54:47.3795403Z Collecting requests==2.26.0
2022-05-18T04:54:47.3991963Z   Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
2022-05-18T04:54:47.4688860Z Collecting idna<4,>=2.5; python_version >= "3"
2022-05-18T04:54:47.4737425Z   Downloading idna-3.3-py3-none-any.whl (61 kB)
2022-05-18T04:54:47.5312376Z Collecting charset-normalizer~=2.0.0; python_version >= "3"
2022-05-18T04:54:47.5429190Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2022-05-18T04:54:47.6420973Z Collecting urllib3<1.27,>=1.21.1
2022-05-18T04:54:47.6471569Z   Downloading urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
2022-05-18T04:54:47.7355332Z Collecting certifi>=2017.4.17
2022-05-18T04:54:47.7709415Z   Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
2022-05-18T04:54:47.8595229Z Installing collected packages: idna, charset-normalizer, urllib3, certifi, requests
2022-05-18T04:54:47.9772973Z   WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-05-18T04:54:47.9773777Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-05-18T04:54:48.1205459Z Successfully installed certifi-2021.10.8 charset-normalizer-2.0.12 idna-3.3 requests-2.26.0 urllib3-1.26.9
2022-05-18T04:54:48.3131094Z ++ python3 .github/scripts/get_workflow_job_id.py 2342799949 i-0d4a316768328dd7a
2022-05-18T04:54:49.7733456Z + GHA_WORKFLOW_JOB_ID=6482671504
2022-05-18T04:54:49.7734210Z + echo '::set-output name=job-id::6482671504'
2022-05-18T04:54:49.9429405Z Command completed after 1 attempt(s).
2022-05-18T04:54:49.9429626Z 
2022-05-18T04:54:49.9598741Z Prepare all required actions
2022-05-18T04:54:49.9599227Z Getting action download info
2022-05-18T04:54:50.2633057Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2)
2022-05-18T04:54:50.4049728Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-05-18T04:54:50.4050026Z with:
2022-05-18T04:54:50.4050388Z   file-suffix: test-multigpu-1-1-linux.16xlarge.nvidia.gpu_6482671504
2022-05-18T04:54:50.4050755Z env:
2022-05-18T04:54:50.4050987Z   IN_CI: 1
2022-05-18T04:54:50.4051201Z   IS_GHA: 1
2022-05-18T04:54:50.4051463Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:54:50.4051742Z   GPU_FLAG: --gpus all
2022-05-18T04:54:50.4051981Z ##[endgroup]
2022-05-18T04:54:50.4081713Z ##[group]Run # Remove any previous test jsons if they exist
2022-05-18T04:54:50.4082107Z # Remove any previous test jsons if they exist
2022-05-18T04:54:50.4082435Z rm -f test-jsons-*.zip
2022-05-18T04:54:50.4082812Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-05-18T04:54:50.4094739Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:54:50.4095048Z env:
2022-05-18T04:54:50.4095287Z   IN_CI: 1
2022-05-18T04:54:50.4095502Z   IS_GHA: 1
2022-05-18T04:54:50.4095770Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:54:50.4096053Z   GPU_FLAG: --gpus all
2022-05-18T04:54:50.4096406Z   FILE_SUFFIX: test-multigpu-1-1-linux.16xlarge.nvidia.gpu_6482671504
2022-05-18T04:54:50.4096782Z ##[endgroup]
2022-05-18T04:54:50.4260368Z   adding: test/allowlist_for_publicAPI.json (deflated 82%)
2022-05-18T04:54:50.4295223Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-05-18T04:54:50.4295791Z   adding: test/.pytorch-slow-tests.json (deflated 71%)
2022-05-18T04:54:50.4300715Z   adding: test/.pytorch-disabled-tests.json (deflated 83%)
2022-05-18T04:54:50.4324359Z ##[group]Run # Remove any previous test reports if they exist
2022-05-18T04:54:50.4324829Z # Remove any previous test reports if they exist
2022-05-18T04:54:50.4325144Z rm -f test-reports-*.zip
2022-05-18T04:54:50.4325488Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml'
2022-05-18T04:54:50.4339716Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:54:50.4339996Z env:
2022-05-18T04:54:50.4340215Z   IN_CI: 1
2022-05-18T04:54:50.4340435Z   IS_GHA: 1
2022-05-18T04:54:50.4340664Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:54:50.4340929Z   GPU_FLAG: --gpus all
2022-05-18T04:54:50.4341296Z   FILE_SUFFIX: test-multigpu-1-1-linux.16xlarge.nvidia.gpu_6482671504
2022-05-18T04:54:50.4341629Z ##[endgroup]
2022-05-18T04:54:50.4466371Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220518041012.xml (deflated 38%)
2022-05-18T04:54:50.4467207Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041016.xml (deflated 41%)
2022-05-18T04:54:50.4468050Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041019.xml (deflated 40%)
2022-05-18T04:54:50.4468862Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041022.xml (deflated 40%)
2022-05-18T04:54:50.4469692Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041025.xml (deflated 41%)
2022-05-18T04:54:50.4470567Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041028.xml (deflated 42%)
2022-05-18T04:54:50.4471681Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041032.xml (deflated 41%)
2022-05-18T04:54:50.4472516Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041039.xml (deflated 41%)
2022-05-18T04:54:50.4473481Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041043.xml (deflated 41%)
2022-05-18T04:54:50.4474253Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041102.xml (deflated 39%)
2022-05-18T04:54:50.4474940Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041106.xml (deflated 38%)
2022-05-18T04:54:50.4475599Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041112.xml (deflated 38%)
2022-05-18T04:54:50.4476274Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041116.xml (deflated 38%)
2022-05-18T04:54:50.4476977Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041122.xml (deflated 38%)
2022-05-18T04:54:50.4477650Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041127.xml (deflated 38%)
2022-05-18T04:54:50.4478329Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041131.xml (deflated 38%)
2022-05-18T04:54:50.4478980Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041136.xml (deflated 38%)
2022-05-18T04:54:50.4479734Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041140.xml (deflated 45%)
2022-05-18T04:54:50.4480552Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041146.xml (deflated 44%)
2022-05-18T04:54:50.4481370Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041152.xml (deflated 42%)
2022-05-18T04:54:50.4482159Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041158.xml (deflated 43%)
2022-05-18T04:54:50.4482961Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041204.xml (deflated 45%)
2022-05-18T04:54:50.4483771Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041210.xml (deflated 45%)
2022-05-18T04:54:50.4484584Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041216.xml (deflated 46%)
2022-05-18T04:54:50.4485367Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041222.xml (deflated 46%)
2022-05-18T04:54:50.4486166Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041228.xml (deflated 44%)
2022-05-18T04:54:50.4486961Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041233.xml (deflated 46%)
2022-05-18T04:54:50.4487806Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041239.xml (deflated 46%)
2022-05-18T04:54:50.4488593Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041245.xml (deflated 43%)
2022-05-18T04:54:50.4489393Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041251.xml (deflated 43%)
2022-05-18T04:54:50.4490191Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041257.xml (deflated 43%)
2022-05-18T04:54:50.4491070Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041302.xml (deflated 44%)
2022-05-18T04:54:50.4491847Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041307.xml (deflated 45%)
2022-05-18T04:54:50.4492720Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041312.xml (deflated 44%)
2022-05-18T04:54:50.4493531Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041316.xml (deflated 45%)
2022-05-18T04:54:50.4494326Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041320.xml (deflated 45%)
2022-05-18T04:54:50.4495227Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041324.xml (deflated 50%)
2022-05-18T04:54:50.4496039Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041330.xml (deflated 42%)
2022-05-18T04:54:50.4496833Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041335.xml (deflated 41%)
2022-05-18T04:54:50.4497631Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041341.xml (deflated 41%)
2022-05-18T04:54:50.4498426Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041346.xml (deflated 41%)
2022-05-18T04:54:50.4499208Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041352.xml (deflated 41%)
2022-05-18T04:54:50.4500000Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041358.xml (deflated 42%)
2022-05-18T04:54:50.4500793Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041405.xml (deflated 42%)
2022-05-18T04:54:50.4501587Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041409.xml (deflated 41%)
2022-05-18T04:54:50.4502698Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041413.xml (deflated 41%)
2022-05-18T04:54:50.4503504Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041418.xml (deflated 44%)
2022-05-18T04:54:50.4504305Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041422.xml (deflated 45%)
2022-05-18T04:54:50.4505096Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041426.xml (deflated 41%)
2022-05-18T04:54:50.4505873Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041430.xml (deflated 41%)
2022-05-18T04:54:50.4506659Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041437.xml (deflated 41%)
2022-05-18T04:54:50.4507518Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041441.xml (deflated 42%)
2022-05-18T04:54:50.4508327Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041445.xml (deflated 41%)
2022-05-18T04:54:50.4509117Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041452.xml (deflated 41%)
2022-05-18T04:54:50.4509878Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041458.xml (deflated 40%)
2022-05-18T04:54:50.4510777Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041502.xml (deflated 39%)
2022-05-18T04:54:50.4511538Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041508.xml (deflated 39%)
2022-05-18T04:54:50.4512300Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041513.xml (deflated 40%)
2022-05-18T04:54:50.4513105Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041517.xml (deflated 39%)
2022-05-18T04:54:50.4513874Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041521.xml (deflated 39%)
2022-05-18T04:54:50.4514621Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041526.xml (deflated 39%)
2022-05-18T04:54:50.4515368Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041530.xml (deflated 39%)
2022-05-18T04:54:50.4516112Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041538.xml (deflated 40%)
2022-05-18T04:54:50.4516866Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041542.xml (deflated 40%)
2022-05-18T04:54:50.4517611Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041548.xml (deflated 39%)
2022-05-18T04:54:50.4518366Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041554.xml (deflated 39%)
2022-05-18T04:54:50.4519095Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041559.xml (deflated 39%)
2022-05-18T04:54:50.4519847Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041603.xml (deflated 39%)
2022-05-18T04:54:50.4520606Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041607.xml (deflated 39%)
2022-05-18T04:54:50.4521346Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041612.xml (deflated 39%)
2022-05-18T04:54:50.4522077Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041616.xml (deflated 39%)
2022-05-18T04:54:50.4522830Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041622.xml (deflated 40%)
2022-05-18T04:54:50.4523576Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041626.xml (deflated 40%)
2022-05-18T04:54:50.4524320Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041631.xml (deflated 40%)
2022-05-18T04:54:50.4525054Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041637.xml (deflated 39%)
2022-05-18T04:54:50.4525799Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041641.xml (deflated 40%)
2022-05-18T04:54:50.4526547Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041646.xml (deflated 39%)
2022-05-18T04:54:50.4527289Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041651.xml (deflated 40%)
2022-05-18T04:54:50.4528018Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041656.xml (deflated 40%)
2022-05-18T04:54:50.4528771Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041700.xml (deflated 40%)
2022-05-18T04:54:50.4529516Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041707.xml (deflated 40%)
2022-05-18T04:54:50.4530339Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041711.xml (deflated 39%)
2022-05-18T04:54:50.4531066Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041715.xml (deflated 39%)
2022-05-18T04:54:50.4531816Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041721.xml (deflated 39%)
2022-05-18T04:54:50.4532614Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041725.xml (deflated 39%)
2022-05-18T04:54:50.4533384Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041730.xml (deflated 39%)
2022-05-18T04:54:50.4534117Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041734.xml (deflated 40%)
2022-05-18T04:54:50.4534864Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041742.xml (deflated 39%)
2022-05-18T04:54:50.4535617Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041747.xml (deflated 39%)
2022-05-18T04:54:50.4536361Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041751.xml (deflated 39%)
2022-05-18T04:54:50.4537097Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041757.xml (deflated 39%)
2022-05-18T04:54:50.4537843Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041801.xml (deflated 39%)
2022-05-18T04:54:50.4538596Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041806.xml (deflated 39%)
2022-05-18T04:54:50.4539343Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041812.xml (deflated 40%)
2022-05-18T04:54:50.4540087Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041817.xml (deflated 40%)
2022-05-18T04:54:50.4540833Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041821.xml (deflated 40%)
2022-05-18T04:54:50.4541585Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041826.xml (deflated 39%)
2022-05-18T04:54:50.4542556Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041831.xml (deflated 40%)
2022-05-18T04:54:50.4543389Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041836.xml (deflated 39%)
2022-05-18T04:54:50.4544129Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041841.xml (deflated 41%)
2022-05-18T04:54:50.4544879Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041842.xml (deflated 39%)
2022-05-18T04:54:50.4545624Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041846.xml (deflated 40%)
2022-05-18T04:54:50.4546374Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041847.xml (deflated 39%)
2022-05-18T04:54:50.4547112Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041853.xml (deflated 40%)
2022-05-18T04:54:50.4547826Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041858.xml (deflated 39%)
2022-05-18T04:54:50.4548525Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041859.xml (deflated 39%)
2022-05-18T04:54:50.4549196Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041900.xml (deflated 39%)
2022-05-18T04:54:50.4549976Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041901.xml (deflated 39%)
2022-05-18T04:54:50.4550654Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041902.xml (deflated 38%)
2022-05-18T04:54:50.4551333Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041903.xml (deflated 39%)
2022-05-18T04:54:50.4552127Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220518041904.xml (deflated 39%)
2022-05-18T04:54:50.4552828Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220518041907.xml (deflated 41%)
2022-05-18T04:54:50.4553506Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041922.xml (deflated 38%)
2022-05-18T04:54:50.4554172Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041928.xml (deflated 38%)
2022-05-18T04:54:50.4554844Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041934.xml (deflated 38%)
2022-05-18T04:54:50.4555491Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041939.xml (deflated 38%)
2022-05-18T04:54:50.4556163Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041945.xml (deflated 38%)
2022-05-18T04:54:50.4556824Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041949.xml (deflated 38%)
2022-05-18T04:54:50.4557486Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042003.xml (deflated 38%)
2022-05-18T04:54:50.4558136Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042012.xml (deflated 38%)
2022-05-18T04:54:50.4558798Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042021.xml (deflated 38%)
2022-05-18T04:54:50.4559456Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042027.xml (deflated 38%)
2022-05-18T04:54:50.4560127Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042033.xml (deflated 37%)
2022-05-18T04:54:50.4560780Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042038.xml (deflated 38%)
2022-05-18T04:54:50.4561440Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042044.xml (deflated 38%)
2022-05-18T04:54:50.4562099Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042050.xml (deflated 39%)
2022-05-18T04:54:50.4562757Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042054.xml (deflated 38%)
2022-05-18T04:54:50.4563407Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518042059.xml (deflated 38%)
2022-05-18T04:54:50.4564157Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042105.xml (deflated 42%)
2022-05-18T04:54:50.4564974Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042111.xml (deflated 41%)
2022-05-18T04:54:50.4565784Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042117.xml (deflated 41%)
2022-05-18T04:54:50.4566576Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042123.xml (deflated 41%)
2022-05-18T04:54:50.4567367Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042129.xml (deflated 41%)
2022-05-18T04:54:50.4568178Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042130.xml (deflated 42%)
2022-05-18T04:54:50.4569069Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042131.xml (deflated 41%)
2022-05-18T04:54:50.4569854Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042137.xml (deflated 41%)
2022-05-18T04:54:50.4570693Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042142.xml (deflated 44%)
2022-05-18T04:54:50.4571506Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042148.xml (deflated 45%)
2022-05-18T04:54:50.4572299Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042154.xml (deflated 43%)
2022-05-18T04:54:50.4573068Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042200.xml (deflated 43%)
2022-05-18T04:54:50.4573863Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042206.xml (deflated 45%)
2022-05-18T04:54:50.4574666Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042212.xml (deflated 45%)
2022-05-18T04:54:50.4575464Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042218.xml (deflated 46%)
2022-05-18T04:54:50.4576253Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042224.xml (deflated 46%)
2022-05-18T04:54:50.4577042Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042230.xml (deflated 44%)
2022-05-18T04:54:50.4577841Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042236.xml (deflated 45%)
2022-05-18T04:54:50.4578635Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042242.xml (deflated 46%)
2022-05-18T04:54:50.4579412Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042248.xml (deflated 44%)
2022-05-18T04:54:50.4580209Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042254.xml (deflated 44%)
2022-05-18T04:54:50.4581003Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042300.xml (deflated 42%)
2022-05-18T04:54:50.4581788Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042306.xml (deflated 41%)
2022-05-18T04:54:50.4582773Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042311.xml (deflated 42%)
2022-05-18T04:54:50.4583582Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042317.xml (deflated 44%)
2022-05-18T04:54:50.4584376Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042323.xml (deflated 44%)
2022-05-18T04:54:50.4585162Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042328.xml (deflated 41%)
2022-05-18T04:54:50.4585942Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042335.xml (deflated 41%)
2022-05-18T04:54:50.4586742Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042341.xml (deflated 41%)
2022-05-18T04:54:50.4587534Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042345.xml (deflated 41%)
2022-05-18T04:54:50.4588438Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042351.xml (deflated 41%)
2022-05-18T04:54:50.4589218Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042357.xml (deflated 42%)
2022-05-18T04:54:50.4590017Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042403.xml (deflated 41%)
2022-05-18T04:54:50.4590877Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042409.xml (deflated 40%)
2022-05-18T04:54:50.4591704Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042415.xml (deflated 41%)
2022-05-18T04:54:50.4592478Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042420.xml (deflated 41%)
2022-05-18T04:54:50.4593278Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042426.xml (deflated 41%)
2022-05-18T04:54:50.4594067Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042432.xml (deflated 41%)
2022-05-18T04:54:50.4594958Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042438.xml (deflated 41%)
2022-05-18T04:54:50.4595764Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042444.xml (deflated 41%)
2022-05-18T04:54:50.4596537Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042450.xml (deflated 42%)
2022-05-18T04:54:50.4597325Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042455.xml (deflated 41%)
2022-05-18T04:54:50.4598133Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042501.xml (deflated 41%)
2022-05-18T04:54:50.4598923Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042508.xml (deflated 41%)
2022-05-18T04:54:50.4599711Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042517.xml (deflated 41%)
2022-05-18T04:54:50.4600501Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042521.xml (deflated 42%)
2022-05-18T04:54:50.4601296Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042527.xml (deflated 42%)
2022-05-18T04:54:50.4602083Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042533.xml (deflated 41%)
2022-05-18T04:54:50.4602865Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042538.xml (deflated 42%)
2022-05-18T04:54:50.4603667Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042544.xml (deflated 41%)
2022-05-18T04:54:50.4604452Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042551.xml (deflated 42%)
2022-05-18T04:54:50.4605246Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042556.xml (deflated 41%)
2022-05-18T04:54:50.4606024Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042601.xml (deflated 42%)
2022-05-18T04:54:50.4606820Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042609.xml (deflated 42%)
2022-05-18T04:54:50.4607747Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042614.xml (deflated 42%)
2022-05-18T04:54:50.4608620Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042620.xml (deflated 43%)
2022-05-18T04:54:50.4609402Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042642.xml (deflated 44%)
2022-05-18T04:54:50.4610259Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042648.xml (deflated 42%)
2022-05-18T04:54:50.4611046Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042653.xml (deflated 41%)
2022-05-18T04:54:50.4611841Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042658.xml (deflated 41%)
2022-05-18T04:54:50.4612628Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042703.xml (deflated 40%)
2022-05-18T04:54:50.4613438Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042709.xml (deflated 41%)
2022-05-18T04:54:50.4614209Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518042716.xml (deflated 41%)
2022-05-18T04:54:50.4614993Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042722.xml (deflated 40%)
2022-05-18T04:54:50.4615748Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042726.xml (deflated 40%)
2022-05-18T04:54:50.4616512Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042743.xml (deflated 42%)
2022-05-18T04:54:50.4617249Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042744.xml (deflated 40%)
2022-05-18T04:54:50.4618011Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042802.xml (deflated 41%)
2022-05-18T04:54:50.4618767Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042818.xml (deflated 41%)
2022-05-18T04:54:50.4619523Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042836.xml (deflated 41%)
2022-05-18T04:54:50.4620257Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042853.xml (deflated 42%)
2022-05-18T04:54:50.4621019Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518042854.xml (deflated 41%)
2022-05-18T04:54:50.4621792Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220518042911.xml (deflated 41%)
2022-05-18T04:54:50.4622788Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042912.xml (deflated 38%)
2022-05-18T04:54:50.4623544Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042917.xml (deflated 38%)
2022-05-18T04:54:50.4624285Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042923.xml (deflated 38%)
2022-05-18T04:54:50.4625054Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042928.xml (deflated 39%)
2022-05-18T04:54:50.4625797Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042934.xml (deflated 38%)
2022-05-18T04:54:50.4626552Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042940.xml (deflated 39%)
2022-05-18T04:54:50.4627283Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042946.xml (deflated 39%)
2022-05-18T04:54:50.4628143Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042952.xml (deflated 38%)
2022-05-18T04:54:50.4628901Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518042958.xml (deflated 39%)
2022-05-18T04:54:50.4629705Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043004.xml (deflated 39%)
2022-05-18T04:54:50.4630455Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043012.xml (deflated 38%)
2022-05-18T04:54:50.4631206Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043018.xml (deflated 39%)
2022-05-18T04:54:50.4631951Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043023.xml (deflated 39%)
2022-05-18T04:54:50.4632700Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043029.xml (deflated 39%)
2022-05-18T04:54:50.4633429Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043035.xml (deflated 38%)
2022-05-18T04:54:50.4634237Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043040.xml (deflated 39%)
2022-05-18T04:54:50.4635017Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518043046.xml (deflated 39%)
2022-05-18T04:54:50.4635758Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220518043054.xml (deflated 39%)
2022-05-18T04:54:50.4636498Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220518043057.xml (deflated 39%)
2022-05-18T04:54:50.4637336Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220518043118.xml (deflated 43%)
2022-05-18T04:54:50.4638285Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220518043121.xml (deflated 43%)
2022-05-18T04:54:50.4639208Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220518043124.xml (deflated 43%)
2022-05-18T04:54:50.4640077Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043128.xml (deflated 41%)
2022-05-18T04:54:50.4640927Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043135.xml (deflated 42%)
2022-05-18T04:54:50.4641779Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043143.xml (deflated 42%)
2022-05-18T04:54:50.4642635Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043150.xml (deflated 41%)
2022-05-18T04:54:50.4643477Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043157.xml (deflated 42%)
2022-05-18T04:54:50.4644298Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043204.xml (deflated 41%)
2022-05-18T04:54:50.4645146Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043211.xml (deflated 41%)
2022-05-18T04:54:50.4645998Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518043218.xml (deflated 41%)
2022-05-18T04:54:50.4646836Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043239.xml (deflated 42%)
2022-05-18T04:54:50.4647739Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043247.xml (deflated 42%)
2022-05-18T04:54:50.4648588Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043254.xml (deflated 42%)
2022-05-18T04:54:50.4649483Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043301.xml (deflated 42%)
2022-05-18T04:54:50.4650336Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043308.xml (deflated 42%)
2022-05-18T04:54:50.4651158Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043315.xml (deflated 42%)
2022-05-18T04:54:50.4651998Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518043322.xml (deflated 42%)
2022-05-18T04:54:50.4652763Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518043340.xml (deflated 39%)
2022-05-18T04:54:50.4653455Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518043343.xml (deflated 39%)
2022-05-18T04:54:50.4654129Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518043346.xml (deflated 39%)
2022-05-18T04:54:50.4654826Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518043349.xml (deflated 39%)
2022-05-18T04:54:50.4655542Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518043352.xml (deflated 40%)
2022-05-18T04:54:50.4656273Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518043354.xml (deflated 40%)
2022-05-18T04:54:50.4656992Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518043357.xml (deflated 39%)
2022-05-18T04:54:50.4657724Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518043400.xml (deflated 39%)
2022-05-18T04:54:50.4658437Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220518043403.xml (deflated 39%)
2022-05-18T04:54:50.4659149Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220518043406.xml (deflated 39%)
2022-05-18T04:54:50.4659863Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518043409.xml (deflated 39%)
2022-05-18T04:54:50.4660576Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518043412.xml (deflated 39%)
2022-05-18T04:54:50.4661293Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043414.xml (deflated 39%)
2022-05-18T04:54:50.4662211Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043417.xml (deflated 39%)
2022-05-18T04:54:50.4662919Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043420.xml (deflated 39%)
2022-05-18T04:54:50.4663635Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043423.xml (deflated 40%)
2022-05-18T04:54:50.4664342Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220518043436.xml (deflated 38%)
2022-05-18T04:54:50.4665035Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043439.xml (deflated 38%)
2022-05-18T04:54:50.4665702Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043442.xml (deflated 38%)
2022-05-18T04:54:50.4666388Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043445.xml (deflated 38%)
2022-05-18T04:54:50.4667179Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043448.xml (deflated 37%)
2022-05-18T04:54:50.4667854Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043451.xml (deflated 38%)
2022-05-18T04:54:50.4668513Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043453.xml (deflated 38%)
2022-05-18T04:54:50.4669257Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043456.xml (deflated 38%)
2022-05-18T04:54:50.4669954Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043501.xml (deflated 38%)
2022-05-18T04:54:50.4670719Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043516.xml (deflated 40%)
2022-05-18T04:54:50.4671524Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043521.xml (deflated 40%)
2022-05-18T04:54:50.4672357Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043525.xml (deflated 40%)
2022-05-18T04:54:50.4673177Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043531.xml (deflated 40%)
2022-05-18T04:54:50.4673988Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043537.xml (deflated 40%)
2022-05-18T04:54:50.4674779Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043541.xml (deflated 40%)
2022-05-18T04:54:50.4675600Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043546.xml (deflated 40%)
2022-05-18T04:54:50.4676409Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043552.xml (deflated 40%)
2022-05-18T04:54:50.4677221Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043558.xml (deflated 40%)
2022-05-18T04:54:50.4678012Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043603.xml (deflated 39%)
2022-05-18T04:54:50.4678836Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043607.xml (deflated 39%)
2022-05-18T04:54:50.4679651Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043613.xml (deflated 39%)
2022-05-18T04:54:50.4680457Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043618.xml (deflated 39%)
2022-05-18T04:54:50.4681245Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043624.xml (deflated 39%)
2022-05-18T04:54:50.4682135Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20220518043643.xml (deflated 41%)
2022-05-18T04:54:50.4683053Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220518043650.xml (deflated 41%)
2022-05-18T04:54:50.4683971Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220518043657.xml (deflated 41%)
2022-05-18T04:54:50.4684893Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220518043704.xml (deflated 41%)
2022-05-18T04:54:50.4685788Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518043712.xml (deflated 40%)
2022-05-18T04:54:50.4686690Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518043718.xml (deflated 40%)
2022-05-18T04:54:50.4687668Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518043725.xml (deflated 41%)
2022-05-18T04:54:50.4688568Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518043730.xml (deflated 41%)
2022-05-18T04:54:50.4689472Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20220518043737.xml (deflated 40%)
2022-05-18T04:54:50.4690357Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043746.xml (deflated 39%)
2022-05-18T04:54:50.4691245Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043754.xml (deflated 39%)
2022-05-18T04:54:50.4692129Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043802.xml (deflated 40%)
2022-05-18T04:54:50.4693087Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043810.xml (deflated 39%)
2022-05-18T04:54:50.4693983Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043818.xml (deflated 39%)
2022-05-18T04:54:50.4694926Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043826.xml (deflated 40%)
2022-05-18T04:54:50.4695841Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043835.xml (deflated 40%)
2022-05-18T04:54:50.4696701Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518043843.xml (deflated 39%)
2022-05-18T04:54:50.4697643Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043851.xml (deflated 42%)
2022-05-18T04:54:50.4698613Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043901.xml (deflated 42%)
2022-05-18T04:54:50.4699584Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043913.xml (deflated 42%)
2022-05-18T04:54:50.4700558Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043923.xml (deflated 43%)
2022-05-18T04:54:50.4701506Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043933.xml (deflated 43%)
2022-05-18T04:54:50.4702686Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043943.xml (deflated 43%)
2022-05-18T04:54:50.4703667Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518043954.xml (deflated 43%)
2022-05-18T04:54:50.4704643Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044004.xml (deflated 43%)
2022-05-18T04:54:50.4705591Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044014.xml (deflated 43%)
2022-05-18T04:54:50.4706561Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044024.xml (deflated 43%)
2022-05-18T04:54:50.4707530Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044029.xml (deflated 43%)
2022-05-18T04:54:50.4708615Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044035.xml (deflated 43%)
2022-05-18T04:54:50.4709548Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044040.xml (deflated 43%)
2022-05-18T04:54:50.4710588Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044045.xml (deflated 43%)
2022-05-18T04:54:50.4711576Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044051.xml (deflated 43%)
2022-05-18T04:54:50.4712554Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044058.xml (deflated 42%)
2022-05-18T04:54:50.4713524Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044109.xml (deflated 43%)
2022-05-18T04:54:50.4714464Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044122.xml (deflated 42%)
2022-05-18T04:54:50.4715424Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044141.xml (deflated 43%)
2022-05-18T04:54:50.4716393Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044155.xml (deflated 43%)
2022-05-18T04:54:50.4717357Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044207.xml (deflated 43%)
2022-05-18T04:54:50.4718300Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044212.xml (deflated 42%)
2022-05-18T04:54:50.4719256Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044221.xml (deflated 42%)
2022-05-18T04:54:50.4720223Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044229.xml (deflated 42%)
2022-05-18T04:54:50.4721197Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044237.xml (deflated 43%)
2022-05-18T04:54:50.4722162Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044247.xml (deflated 42%)
2022-05-18T04:54:50.4723104Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044258.xml (deflated 42%)
2022-05-18T04:54:50.4724075Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044308.xml (deflated 42%)
2022-05-18T04:54:50.4725042Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044319.xml (deflated 42%)
2022-05-18T04:54:50.4726010Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044329.xml (deflated 42%)
2022-05-18T04:54:50.4726943Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044340.xml (deflated 42%)
2022-05-18T04:54:50.4727898Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044351.xml (deflated 42%)
2022-05-18T04:54:50.4728933Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044401.xml (deflated 43%)
2022-05-18T04:54:50.4729900Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044412.xml (deflated 42%)
2022-05-18T04:54:50.4730911Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044422.xml (deflated 43%)
2022-05-18T04:54:50.4731867Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044433.xml (deflated 42%)
2022-05-18T04:54:50.4732835Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044443.xml (deflated 42%)
2022-05-18T04:54:50.4733799Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044454.xml (deflated 42%)
2022-05-18T04:54:50.4734764Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044504.xml (deflated 42%)
2022-05-18T04:54:50.4735705Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044514.xml (deflated 42%)
2022-05-18T04:54:50.4736731Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044525.xml (deflated 42%)
2022-05-18T04:54:50.4737686Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044535.xml (deflated 42%)
2022-05-18T04:54:50.4738643Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044543.xml (deflated 43%)
2022-05-18T04:54:50.4739604Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044554.xml (deflated 42%)
2022-05-18T04:54:50.4740540Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044602.xml (deflated 42%)
2022-05-18T04:54:50.4741503Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044611.xml (deflated 42%)
2022-05-18T04:54:50.4742685Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044621.xml (deflated 42%)
2022-05-18T04:54:50.4743649Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044631.xml (deflated 43%)
2022-05-18T04:54:50.4744610Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044637.xml (deflated 43%)
2022-05-18T04:54:50.4745550Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044642.xml (deflated 43%)
2022-05-18T04:54:50.4746517Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044647.xml (deflated 42%)
2022-05-18T04:54:50.4747487Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044652.xml (deflated 42%)
2022-05-18T04:54:50.4748445Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044659.xml (deflated 42%)
2022-05-18T04:54:50.4749489Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044706.xml (deflated 42%)
2022-05-18T04:54:50.4750449Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044713.xml (deflated 42%)
2022-05-18T04:54:50.4751491Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044720.xml (deflated 42%)
2022-05-18T04:54:50.4752469Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044727.xml (deflated 42%)
2022-05-18T04:54:50.4753426Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044735.xml (deflated 42%)
2022-05-18T04:54:50.4754362Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044742.xml (deflated 42%)
2022-05-18T04:54:50.4755336Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044752.xml (deflated 42%)
2022-05-18T04:54:50.4756299Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044803.xml (deflated 43%)
2022-05-18T04:54:50.4757257Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044808.xml (deflated 43%)
2022-05-18T04:54:50.4758197Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044818.xml (deflated 42%)
2022-05-18T04:54:50.4759158Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044833.xml (deflated 42%)
2022-05-18T04:54:50.4760122Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044849.xml (deflated 42%)
2022-05-18T04:54:50.4761077Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044854.xml (deflated 43%)
2022-05-18T04:54:50.4762022Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044902.xml (deflated 42%)
2022-05-18T04:54:50.4762988Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044907.xml (deflated 42%)
2022-05-18T04:54:50.4763947Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044912.xml (deflated 43%)
2022-05-18T04:54:50.4764917Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044920.xml (deflated 42%)
2022-05-18T04:54:50.4765880Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044930.xml (deflated 41%)
2022-05-18T04:54:50.4766820Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044941.xml (deflated 42%)
2022-05-18T04:54:50.4767793Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518044952.xml (deflated 42%)
2022-05-18T04:54:50.4768747Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045002.xml (deflated 42%)
2022-05-18T04:54:50.4769709Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045020.xml (deflated 42%)
2022-05-18T04:54:50.4770724Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045040.xml (deflated 42%)
2022-05-18T04:54:50.4771688Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045058.xml (deflated 42%)
2022-05-18T04:54:50.4772709Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045118.xml (deflated 42%)
2022-05-18T04:54:50.4773683Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045136.xml (deflated 42%)
2022-05-18T04:54:50.4774723Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045153.xml (deflated 41%)
2022-05-18T04:54:50.4775669Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045209.xml (deflated 42%)
2022-05-18T04:54:50.4776630Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045226.xml (deflated 42%)
2022-05-18T04:54:50.4777593Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045242.xml (deflated 41%)
2022-05-18T04:54:50.4778555Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045300.xml (deflated 42%)
2022-05-18T04:54:50.4779496Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045320.xml (deflated 42%)
2022-05-18T04:54:50.4780467Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045338.xml (deflated 41%)
2022-05-18T04:54:50.4781430Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045358.xml (deflated 42%)
2022-05-18T04:54:50.4782598Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518045407.xml (deflated 43%)
2022-05-18T04:54:50.4783594Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220518045417.xml (deflated 43%)
2022-05-18T04:54:50.4784573Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220518045426.xml (deflated 43%)
2022-05-18T04:54:50.4785574Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220518045435.xml (deflated 42%)
2022-05-18T04:54:50.4826233Z ##[group]Run seemethere/upload-artifact-s3@v4
2022-05-18T04:54:50.4826534Z with:
2022-05-18T04:54:50.4826778Z   retention-days: 14
2022-05-18T04:54:50.4827032Z   if-no-files-found: warn
2022-05-18T04:54:50.4827310Z   path: test-jsons-*.zip
2022-05-18T04:54:50.4827563Z   name: artifact
2022-05-18T04:54:50.4827796Z   s3-bucket: gha-artifacts
2022-05-18T04:54:50.4828064Z   region: us-east-1
2022-05-18T04:54:50.4828295Z env:
2022-05-18T04:54:50.4828490Z   IN_CI: 1
2022-05-18T04:54:50.4828735Z   IS_GHA: 1
2022-05-18T04:54:50.4828981Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:54:50.4829227Z   GPU_FLAG: --gpus all
2022-05-18T04:54:50.4829471Z ##[endgroup]
2022-05-18T04:54:50.9167482Z With the provided path, there will be 1 file uploaded
2022-05-18T04:54:50.9167906Z Uploading to s3 prefix: pytorch/pytorch/2342799949/1/artifact
2022-05-18T04:54:50.9178527Z Starting upload of test-jsons-test-multigpu-1-1-linux.16xlarge.nvidia.gpu_6482671504.zip
2022-05-18T04:54:51.0851781Z Finished upload of test-jsons-test-multigpu-1-1-linux.16xlarge.nvidia.gpu_6482671504.zip
2022-05-18T04:54:51.1051863Z ##[group]Run seemethere/upload-artifact-s3@v4
2022-05-18T04:54:51.1052295Z with:
2022-05-18T04:54:51.1052524Z   retention-days: 14
2022-05-18T04:54:51.1052785Z   if-no-files-found: error
2022-05-18T04:54:51.1053043Z   path: test-reports-*.zip
2022-05-18T04:54:51.1053428Z   name: artifact
2022-05-18T04:54:51.1053690Z   s3-bucket: gha-artifacts
2022-05-18T04:54:51.1053927Z   region: us-east-1
2022-05-18T04:54:51.1054151Z env:
2022-05-18T04:54:51.1054361Z   IN_CI: 1
2022-05-18T04:54:51.1054557Z   IS_GHA: 1
2022-05-18T04:54:51.1054792Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:54:51.1055048Z   GPU_FLAG: --gpus all
2022-05-18T04:54:51.1055270Z ##[endgroup]
2022-05-18T04:54:51.5452099Z With the provided path, there will be 1 file uploaded
2022-05-18T04:54:51.5452542Z Uploading to s3 prefix: pytorch/pytorch/2342799949/1/artifact
2022-05-18T04:54:51.5463356Z Starting upload of test-reports-test-multigpu-1-1-linux.16xlarge.nvidia.gpu_6482671504.zip
2022-05-18T04:54:51.6923723Z Finished upload of test-reports-test-multigpu-1-1-linux.16xlarge.nvidia.gpu_6482671504.zip
2022-05-18T04:54:51.7086109Z ##[group]Run set -x
2022-05-18T04:54:51.7086409Z set -x
2022-05-18T04:54:51.7086723Z python3 -m pip install -r requirements.txt
2022-05-18T04:54:51.7087077Z python3 -m pip install boto3==1.19.12
2022-05-18T04:54:51.7087492Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-05-18T04:54:51.7103686Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:54:51.7103992Z env:
2022-05-18T04:54:51.7104219Z   IN_CI: 1
2022-05-18T04:54:51.7104426Z   IS_GHA: 1
2022-05-18T04:54:51.7104681Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:54:51.7104958Z   GPU_FLAG: --gpus all
2022-05-18T04:54:51.7105227Z   AWS_DEFAULT_REGION: us-east-1
2022-05-18T04:54:51.7105652Z   BRANCH: master
2022-05-18T04:54:51.7105967Z   JOB_BASE_NAME: linux-bionic-cuda10.2-py3.9-gcc7-test
2022-05-18T04:54:51.7106267Z   TEST_CONFIG: multigpu
2022-05-18T04:54:51.7106514Z   SHARD_NUMBER: 1
2022-05-18T04:54:51.7106870Z   BUILD_ENVIRONMENT: linux-bionic-cuda10.2-py3.9-gcc7
2022-05-18T04:54:51.7107157Z   PR_NUMBER: 
2022-05-18T04:54:51.7107431Z   SHA1: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T04:54:51.7107703Z   TAG: 
2022-05-18T04:54:51.7107917Z   WORKFLOW_ID: 2342799949
2022-05-18T04:54:51.7108354Z   GITHUB_TOKEN: ***
2022-05-18T04:54:51.7108617Z   GHA_WORKFLOW_JOB_ID: 6482671504
2022-05-18T04:54:51.7109048Z ##[endgroup]
2022-05-18T04:54:51.7145340Z + python3 -m pip install -r requirements.txt
2022-05-18T04:54:52.0138029Z Defaulting to user installation because normal site-packages is not writeable
2022-05-18T04:54:52.0455000Z Ignoring dataclasses: markers 'python_version < "3.7"' don't match your environment
2022-05-18T04:54:52.0970156Z Collecting astunparse
2022-05-18T04:54:52.1133168Z   Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
2022-05-18T04:54:52.1432596Z Collecting expecttest
2022-05-18T04:54:52.1482168Z   Downloading expecttest-0.1.3-py3-none-any.whl (6.5 kB)
2022-05-18T04:54:52.1921619Z Collecting future
2022-05-18T04:54:52.1963942Z   Downloading future-0.18.2.tar.gz (829 kB)
2022-05-18T04:54:53.4900018Z Collecting numpy
2022-05-18T04:54:53.4959830Z   Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
2022-05-18T04:54:54.1486130Z Collecting psutil
2022-05-18T04:54:54.1546052Z   Downloading psutil-5.9.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)
2022-05-18T04:54:54.2903323Z Collecting pyyaml
2022-05-18T04:54:54.2953542Z   Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
2022-05-18T04:54:54.3186706Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (2.26.0)
2022-05-18T04:54:54.3362617Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (49.1.3)
2022-05-18T04:54:54.3992807Z Collecting six
2022-05-18T04:54:54.4034904Z   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
2022-05-18T04:54:54.4380526Z Collecting types-dataclasses
2022-05-18T04:54:54.4424121Z   Downloading types_dataclasses-0.6.5-py3-none-any.whl (2.8 kB)
2022-05-18T04:54:54.4830888Z Collecting typing_extensions
2022-05-18T04:54:54.4872342Z   Downloading typing_extensions-4.2.0-py3-none-any.whl (24 kB)
2022-05-18T04:54:54.5737205Z Collecting wheel<1.0,>=0.23.0
2022-05-18T04:54:54.5779564Z   Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
2022-05-18T04:54:54.5907245Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (3.3)
2022-05-18T04:54:54.5922527Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (2021.10.8)
2022-05-18T04:54:54.5932467Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (1.26.9)
2022-05-18T04:54:54.6149006Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (2.0.12)
2022-05-18T04:54:54.6176897Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
2022-05-18T04:54:54.6551325Z Installing collected packages: six, wheel, astunparse, expecttest, future, numpy, psutil, pyyaml, types-dataclasses, typing-extensions
2022-05-18T04:54:54.6988761Z   WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-05-18T04:54:54.6989421Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-05-18T04:54:54.7318769Z     Running setup.py install for future: started
2022-05-18T04:54:55.4435730Z     Running setup.py install for future: finished with status 'done'
2022-05-18T04:54:57.5290285Z   WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-05-18T04:54:57.5291013Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-05-18T04:54:57.8138115Z Successfully installed astunparse-1.6.3 expecttest-0.1.3 future-0.18.2 numpy-1.21.6 psutil-5.9.0 pyyaml-6.0 six-1.16.0 types-dataclasses-0.6.5 typing-extensions-4.2.0 wheel-0.37.1
2022-05-18T04:54:57.8881933Z + python3 -m pip install boto3==1.19.12
2022-05-18T04:54:58.2000120Z Defaulting to user installation because normal site-packages is not writeable
2022-05-18T04:54:59.0662859Z Collecting boto3==1.19.12
2022-05-18T04:54:59.0839126Z   Downloading boto3-1.19.12-py3-none-any.whl (131 kB)
2022-05-18T04:55:00.1227493Z Collecting botocore<1.23.0,>=1.22.12
2022-05-18T04:55:00.1291254Z   Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB)
2022-05-18T04:55:00.3756315Z Collecting s3transfer<0.6.0,>=0.5.0
2022-05-18T04:55:00.3802881Z   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
2022-05-18T04:55:00.4278428Z Collecting jmespath<1.0.0,>=0.7.1
2022-05-18T04:55:00.4323230Z   Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
2022-05-18T04:55:00.4439289Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.9)
2022-05-18T04:55:00.5117507Z Collecting python-dateutil<3.0.0,>=2.1
2022-05-18T04:55:00.5165082Z   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
2022-05-18T04:55:00.5338146Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-05-18T04:55:00.6203488Z Installing collected packages: jmespath, python-dateutil, botocore, s3transfer, boto3
2022-05-18T04:55:01.5948880Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2
2022-05-18T04:55:01.6541005Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-05-18T04:55:06.6542600Z [scribe] Scribe access token not provided, sending report via boto3...
2022-05-18T04:55:06.6542955Z 
2022-05-18T04:55:06.6543380Z ----- Historic stats comparison result ------
2022-05-18T04:55:06.6543599Z 
2022-05-18T04:55:06.6543990Z     job: linux-bionic-cuda10.2-py3.9-gcc7-test
2022-05-18T04:55:06.6544354Z     commit: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T04:55:06.6544573Z 
2022-05-18T04:55:06.6544786Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-05-18T04:55:06.6545043Z 
2022-05-18T04:55:06.6545182Z     : (master)
2022-05-18T04:55:06.6545417Z     |
2022-05-18T04:55:06.6545667Z     * 3b2375291a (HEAD)              total time  2021.90s
2022-05-18T04:55:06.6546590Z     * 6e3391a7c3 (base)   3 reports, total time  1180.02s ±  895.86s
2022-05-18T04:55:06.6547478Z     * 48581d74ad          4 reports, total time  1369.92s ±  835.10s
2022-05-18T04:55:06.6547944Z     * c35bd8d423          4 reports, total time  1202.39s ±  728.97s
2022-05-18T04:55:06.6548389Z     * f6beda89c6          6 reports, total time  1161.10s ± 1359.05s
2022-05-18T04:55:06.6548825Z     * ee080918df          9 reports, total time  2710.43s ± 2712.08s
2022-05-18T04:55:06.6549145Z     * bbaefdf6b5          0 reports
2022-05-18T04:55:06.6549401Z     * 7c52f204e0          0 reports
2022-05-18T04:55:06.6549685Z     * e0451d8022          0 reports
2022-05-18T04:55:06.6550079Z     * 4e2f5507d0          9 reports, total time  2696.76s ± 2644.11s
2022-05-18T04:55:06.6550491Z     * b64845eb18          9 reports, total time  2712.94s ± 2654.45s
2022-05-18T04:55:06.6550784Z     |
2022-05-18T04:55:06.6551009Z     :
2022-05-18T04:55:06.6551153Z 
2022-05-18T04:55:06.6551301Z Removed  (across  562 suites)     0 tests, totaling     0.00s
2022-05-18T04:55:06.6551667Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-05-18T04:55:06.6552043Z Added    (across   32 suites)   385 tests, totaling +2021.90s
2022-05-18T04:55:06.7099869Z Prepare all required actions
2022-05-18T04:55:06.7124522Z ##[group]Run ./.github/actions/teardown-linux
2022-05-18T04:55:06.7124816Z with:
2022-05-18T04:55:06.7125016Z env:
2022-05-18T04:55:06.7125246Z   IN_CI: 1
2022-05-18T04:55:06.7125484Z   IS_GHA: 1
2022-05-18T04:55:06.7125717Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:55:06.7125996Z   GPU_FLAG: --gpus all
2022-05-18T04:55:06.7126251Z ##[endgroup]
2022-05-18T04:55:06.7144373Z ##[group]Run .github/scripts/wait_for_ssh_to_drain.sh
2022-05-18T04:55:06.7144735Z .github/scripts/wait_for_ssh_to_drain.sh
2022-05-18T04:55:06.7159307Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:55:06.7159762Z env:
2022-05-18T04:55:06.7159981Z   IN_CI: 1
2022-05-18T04:55:06.7160202Z   IS_GHA: 1
2022-05-18T04:55:06.7160428Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:55:06.7160699Z   GPU_FLAG: --gpus all
2022-05-18T04:55:06.7160948Z ##[endgroup]
2022-05-18T04:55:06.7214613Z Holding runner for 2 hours until all ssh sessions have logged out
2022-05-18T04:55:06.7285190Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-05-18T04:55:06.7285661Z # ignore expansion of "docker ps -q" since it could be empty
2022-05-18T04:55:06.7286015Z # shellcheck disable=SC2046
2022-05-18T04:55:06.7286311Z docker stop $(docker ps -q) || true
2022-05-18T04:55:06.7286628Z # Prune all of the docker images
2022-05-18T04:55:06.7286933Z docker system prune -af
2022-05-18T04:55:06.7298581Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:55:06.7298894Z env:
2022-05-18T04:55:06.7299252Z   IN_CI: 1
2022-05-18T04:55:06.7299463Z   IS_GHA: 1
2022-05-18T04:55:06.7299720Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:55:06.7299996Z   GPU_FLAG: --gpus all
2022-05-18T04:55:06.7300252Z ##[endgroup]
2022-05-18T04:55:07.8056894Z 3bd9d49d70bf
2022-05-18T04:55:08.5635215Z Deleted Containers:
2022-05-18T04:55:08.5635655Z 3bd9d49d70bf00640e887a904f809d68654d815afb203dfacb3a7962baa0db74
2022-05-18T04:55:08.5635906Z 
2022-05-18T04:55:12.8485726Z Deleted Images:
2022-05-18T04:55:12.8486699Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:55:12.8487675Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7@sha256:9737b662edb86afcd12a9367db6178a57889543632c0b710c5058abe14dc048f
2022-05-18T04:55:12.8488292Z deleted: sha256:914b650c5e1ee0f842697bbae2306dd6d831a4fa7fb861ca07bf056998b8539a
2022-05-18T04:55:12.8488780Z deleted: sha256:1034dda927c8a98e2c5d65a336554b89dbbe1e12c28d4d48b88e54f147a2e4e0
2022-05-18T04:55:12.8489244Z deleted: sha256:9daaaebd2559405012ffcc55a915a07af3fd8dfffccf3a4095f52a8d3a2a0808
2022-05-18T04:55:12.8489688Z deleted: sha256:62633a2457311070c286784502f87ac7817442880550ef46ef31086f62f63bd8
2022-05-18T04:55:12.8490074Z deleted: sha256:b40174881876c17fba8e4416c64d2b2065ba27f412e978c62446c6bd9975f43d
2022-05-18T04:55:12.8490511Z deleted: sha256:2b9f3cf2c41f5277698e8a3507d610f08552eb7289a4388f78a18f4934288b8c
2022-05-18T04:55:12.8490935Z deleted: sha256:bd03b60328b2f30ca7a665b612f2cc06f82974a2523f37e690f2eb32b20e23b1
2022-05-18T04:55:12.8491386Z deleted: sha256:9ead2207e8271970850e6a2fd7eacfc78f81c37d45b383107a12ba34b33a0068
2022-05-18T04:55:12.8491812Z deleted: sha256:03fe25e910ef9c726eef212a600805ba6fdd2cba133eec3a76ae6a62e71c50a8
2022-05-18T04:55:12.8492252Z deleted: sha256:42e9502eca4ade58460a090e6049a4c886d6667dc476a43c122110e9970e0504
2022-05-18T04:55:12.8492674Z deleted: sha256:3e18692fe2820772fe2b383c23571e3871b1e76e6ed758ca077a24e1fdae6a28
2022-05-18T04:55:12.8493095Z deleted: sha256:a9c1ea768838d14bfbdde1eb39006e75c504ef0e289e20b1cf1a0960ad20d993
2022-05-18T04:55:12.8493542Z deleted: sha256:653ed47cee104744163b9185cfc53ab6e751d141965b21a2f8bff4fb24acfd37
2022-05-18T04:55:12.8493984Z deleted: sha256:2ff0727ba124b0079c011424c629c2a5e27c5d7afb7b950b5513d4ab4f5e958d
2022-05-18T04:55:12.8494696Z deleted: sha256:4c3c43891ad25595b7374a30159f60ec584375dbc3820ecb30f5ad0374e5e86e
2022-05-18T04:55:12.8495146Z deleted: sha256:fa7d613a19e64cdd36a0c27fc6a2a50dd27c841da90bfae85e542064284ab2fd
2022-05-18T04:55:12.8495611Z deleted: sha256:22ec6f7d0cdf47c266dd9f601a0c98bd88bbd7e4ce3d21c9f7e00349cf7a0f8d
2022-05-18T04:55:12.8496096Z deleted: sha256:3ddedefb6de6867b92dc64bef9ed3206b098bcec87336ba702a4eec81de23bdf
2022-05-18T04:55:12.8496532Z deleted: sha256:6d2243fa3601d3ad6f7187388ef2f63d2eb318689d897e70fafdf33f22667537
2022-05-18T04:55:12.8496958Z deleted: sha256:8d2732c0f78444380cf8b5381c7b649a2e38315a0c11b8f03c7aab8f436d5390
2022-05-18T04:55:12.8497393Z deleted: sha256:85365c4faa86a33743f2107ccd2057705ec1aba1968cfeafbd737362b5499158
2022-05-18T04:55:12.8497840Z deleted: sha256:1aa2e018ba9609d32285b9d5ae5d41d884801742d27f0cbfcd249ab14b4bd4dc
2022-05-18T04:55:12.8498236Z deleted: sha256:3e096c567269719a45cda64f50eb9814c8bb7049822811461314641c8eb96c61
2022-05-18T04:55:12.8498669Z deleted: sha256:6c5ba201ed4d2056c53645f53d30efb8e4ba80fbea2c45042319090bd48d473c
2022-05-18T04:55:12.8499105Z deleted: sha256:64928ee816f9ae39d46f7dd36a5e45302562fd147967f6ab287a487c354b6b6c
2022-05-18T04:55:12.8499498Z deleted: sha256:a57b906a61d609815d662f4f4b65996a46514b07aea462793fd4143718ffc840
2022-05-18T04:55:12.8499907Z deleted: sha256:494612c761757956fbc4227e61a4a1e63e0f9b3372cf2430e2ee002ab523cfde
2022-05-18T04:55:12.8500333Z deleted: sha256:ff2c733048c22f423a6b20c35ff08bcbb6fe1bc76306464e654ef1ee28c3d861
2022-05-18T04:55:12.8500776Z deleted: sha256:4661c25de76163d8c2e45ca688f5b819c61c6c9f8e49ed83df44db353263f033
2022-05-18T04:55:12.8507449Z deleted: sha256:e23913a427ab6a1d96fc5ac9b9916776209427c2ef8eb9f44a4d16735f8c8494
2022-05-18T04:55:12.8508579Z deleted: sha256:832b3ad6407fa37ec6d8fd8f9d28172fe3bd5f6280fad98472d09eb0bc252ae0
2022-05-18T04:55:12.8509181Z deleted: sha256:a2b9dd02872fa4e35324d54aba02a6f1f21cb993714948cb709f94a2d85029f9
2022-05-18T04:55:12.8509659Z deleted: sha256:cb96bd5b78d181c6c2779f27e47036a8e9c3e1bcf09da94039148abd1c7d05ee
2022-05-18T04:55:12.8510107Z deleted: sha256:ea52cef0d0fe0c5edd5d235153b16fb0ce71bd0120ad33ed45f75bbfa3d9eadf
2022-05-18T04:55:12.8510588Z deleted: sha256:4fb97c7eb8955725be2bae74694a3af51e36e515a6c92a1aa75965cc09864f99
2022-05-18T04:55:12.8511027Z deleted: sha256:b2537994f751dde0a341c1f0d09a833be0150eb5a1cd60c7e65874442f6475a3
2022-05-18T04:55:12.8511454Z deleted: sha256:412f35baea526807361ea20e8f0e18576bdf2c6c40bdec402e94d86222a2b56e
2022-05-18T04:55:12.8511871Z deleted: sha256:cf621551bc4ed287124425a3d232f6c751dff14e9986bf7b7a697634d2f599bc
2022-05-18T04:55:12.8512293Z deleted: sha256:8003ff14feede16807731ad20c8151882bb62d724eb628e4c99ceaa2eea2a479
2022-05-18T04:55:12.8512739Z deleted: sha256:a1270a733ee0912cf66cd39d15f2ceace3789554b56647c5a5638b6ba73e3dab
2022-05-18T04:55:12.8513178Z deleted: sha256:a2811bdab35ec13d2eb84fdf4de75cbd29c5f6e227e4f11e9e8a9de714b7e132
2022-05-18T04:55:12.8513623Z deleted: sha256:f80e00922ecb54c1458a8c92d41e262173286ff550ed7468674de42de539714b
2022-05-18T04:55:12.8514046Z deleted: sha256:eb265251ed90e139bb4bfd41d9fa6a2cc6275eab106538fead323171069af9c9
2022-05-18T04:55:12.8514500Z deleted: sha256:fbee4dd8d443dcf0791e3965ee624b8ecc7b15d503ffbf8f2912d4d1d0a0cb47
2022-05-18T04:55:12.8514942Z deleted: sha256:e2f7d8e2982218fbc16adfe64b71e1839795e7a3ea82f5ff65336d58ae4cea0b
2022-05-18T04:55:12.8516032Z deleted: sha256:275df7d7943e762bf0a85fc2a9cd297c01ecb5d87ae4d86466c3f7f704d1c778
2022-05-18T04:55:12.8516477Z deleted: sha256:c2c5293df593b2d991852fe08e5db0f8c5d3c06b64247dc508084e747e64a42e
2022-05-18T04:55:12.8516889Z deleted: sha256:986cd2e7c143559516bc8388d5dd603eec6a1be4855c777c7e7f16bf22b9fa23
2022-05-18T04:55:12.8517330Z deleted: sha256:9d6787a516e72b7ed9422c8df1a4b298d82982bdf80ee1e198eedf1e1a010d76
2022-05-18T04:55:12.8517578Z 
2022-05-18T04:55:12.8638048Z Total reclaimed space: 12.14GB
2022-05-18T04:55:12.8707926Z Post job cleanup.
2022-05-18T04:55:12.8743423Z Post job cleanup.
2022-05-18T04:55:13.0104142Z [command]/usr/bin/git version
2022-05-18T04:55:13.0154499Z git version 2.32.0
2022-05-18T04:55:13.0218885Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/dddb3124-8dda-4c3b-a123-fc7977c23663' before making global git config changes
2022-05-18T04:55:13.0219496Z Adding repository directory to the temporary git global config as a safe directory
2022-05-18T04:55:13.0228676Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T04:55:13.0278448Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-05-18T04:55:13.0317598Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-05-18T04:55:13.0691065Z Entering 'android/libs/fbjni'
2022-05-18T04:55:13.0736589Z Entering 'third_party/FP16'
2022-05-18T04:55:13.0783902Z Entering 'third_party/FXdiv'
2022-05-18T04:55:13.0835143Z Entering 'third_party/NNPACK'
2022-05-18T04:55:13.0888989Z Entering 'third_party/QNNPACK'
2022-05-18T04:55:13.0940208Z Entering 'third_party/XNNPACK'
2022-05-18T04:55:13.1004488Z Entering 'third_party/benchmark'
2022-05-18T04:55:13.1055631Z Entering 'third_party/cpuinfo'
2022-05-18T04:55:13.1108030Z Entering 'third_party/cub'
2022-05-18T04:55:13.1160892Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:55:13.1217530Z Entering 'third_party/eigen'
2022-05-18T04:55:13.1275139Z Entering 'third_party/fbgemm'
2022-05-18T04:55:13.1325256Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:55:13.1372973Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:55:13.1425030Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:55:13.1478659Z Entering 'third_party/flatbuffers'
2022-05-18T04:55:13.1532599Z Entering 'third_party/fmt'
2022-05-18T04:55:13.1579367Z Entering 'third_party/foxi'
2022-05-18T04:55:13.1630324Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:55:13.1681603Z Entering 'third_party/gloo'
2022-05-18T04:55:13.1732321Z Entering 'third_party/googletest'
2022-05-18T04:55:13.1781503Z Entering 'third_party/ideep'
2022-05-18T04:55:13.1831799Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:55:13.1884219Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:55:13.1942482Z Entering 'third_party/ios-cmake'
2022-05-18T04:55:13.1993946Z Entering 'third_party/kineto'
2022-05-18T04:55:13.2043792Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:55:13.2093351Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:55:13.2146428Z Entering 'third_party/nccl/nccl'
2022-05-18T04:55:13.2197184Z Entering 'third_party/neon2sse'
2022-05-18T04:55:13.2246626Z Entering 'third_party/onnx'
2022-05-18T04:55:13.2309128Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:55:13.2358330Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:55:13.2412060Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:55:13.2462659Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:55:13.2515563Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:55:13.2566882Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:55:13.2617027Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:55:13.2674179Z Entering 'third_party/pocketfft'
2022-05-18T04:55:13.2722595Z Entering 'third_party/protobuf'
2022-05-18T04:55:13.2777728Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:55:13.2830735Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:55:13.2883218Z Entering 'third_party/psimd'
2022-05-18T04:55:13.2933625Z Entering 'third_party/pthreadpool'
2022-05-18T04:55:13.2982534Z Entering 'third_party/pybind11'
2022-05-18T04:55:13.3033910Z Entering 'third_party/python-enum'
2022-05-18T04:55:13.3082804Z Entering 'third_party/python-peachpy'
2022-05-18T04:55:13.3133521Z Entering 'third_party/python-six'
2022-05-18T04:55:13.3182796Z Entering 'third_party/sleef'
2022-05-18T04:55:13.3234307Z Entering 'third_party/tbb'
2022-05-18T04:55:13.3285671Z Entering 'third_party/tensorpipe'
2022-05-18T04:55:13.3335840Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:55:13.3387308Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:55:13.3435384Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:55:13.3486745Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:55:13.3532804Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:55:13.3585636Z Entering 'third_party/zstd'
2022-05-18T04:55:13.3663343Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-05-18T04:55:13.3703390Z http.https://github.com/.extraheader
2022-05-18T04:55:13.3715926Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-05-18T04:55:13.3763731Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-05-18T04:55:13.4157299Z Entering 'android/libs/fbjni'
2022-05-18T04:55:13.4184052Z http.https://github.com/.extraheader
2022-05-18T04:55:13.4223174Z Entering 'third_party/FP16'
2022-05-18T04:55:13.4253206Z http.https://github.com/.extraheader
2022-05-18T04:55:13.4294177Z Entering 'third_party/FXdiv'
2022-05-18T04:55:13.4321993Z http.https://github.com/.extraheader
2022-05-18T04:55:13.4359955Z Entering 'third_party/NNPACK'
2022-05-18T04:55:13.4388265Z http.https://github.com/.extraheader
2022-05-18T04:55:13.4427435Z Entering 'third_party/QNNPACK'
2022-05-18T04:55:13.4455713Z http.https://github.com/.extraheader
2022-05-18T04:55:13.4497042Z Entering 'third_party/XNNPACK'
2022-05-18T04:55:13.4525850Z http.https://github.com/.extraheader
2022-05-18T04:55:13.4576527Z Entering 'third_party/benchmark'
2022-05-18T04:55:13.4606537Z http.https://github.com/.extraheader
2022-05-18T04:55:13.4646519Z Entering 'third_party/cpuinfo'
2022-05-18T04:55:13.4675460Z http.https://github.com/.extraheader
2022-05-18T04:55:13.4715870Z Entering 'third_party/cub'
2022-05-18T04:55:13.4746812Z http.https://github.com/.extraheader
2022-05-18T04:55:13.4785638Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:55:13.4815196Z http.https://github.com/.extraheader
2022-05-18T04:55:13.4863434Z Entering 'third_party/eigen'
2022-05-18T04:55:13.4891225Z http.https://github.com/.extraheader
2022-05-18T04:55:13.4934414Z Entering 'third_party/fbgemm'
2022-05-18T04:55:13.4963892Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5003849Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:55:13.5032919Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5073526Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:55:13.5101683Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5141602Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:55:13.5169817Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5209663Z Entering 'third_party/flatbuffers'
2022-05-18T04:55:13.5236967Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5277637Z Entering 'third_party/fmt'
2022-05-18T04:55:13.5305739Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5345195Z Entering 'third_party/foxi'
2022-05-18T04:55:13.5374499Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5413311Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:55:13.5441401Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5480388Z Entering 'third_party/gloo'
2022-05-18T04:55:13.5510177Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5548552Z Entering 'third_party/googletest'
2022-05-18T04:55:13.5577425Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5615499Z Entering 'third_party/ideep'
2022-05-18T04:55:13.5645149Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5682093Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:55:13.5708959Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5749021Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:55:13.5777875Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5825527Z Entering 'third_party/ios-cmake'
2022-05-18T04:55:13.5852570Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5892814Z Entering 'third_party/kineto'
2022-05-18T04:55:13.5921795Z http.https://github.com/.extraheader
2022-05-18T04:55:13.5959530Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:55:13.5987806Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6026263Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:55:13.6054295Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6094979Z Entering 'third_party/nccl/nccl'
2022-05-18T04:55:13.6124565Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6162904Z Entering 'third_party/neon2sse'
2022-05-18T04:55:13.6189786Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6227284Z Entering 'third_party/onnx'
2022-05-18T04:55:13.6257077Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6310179Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:55:13.6339562Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6375347Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:55:13.6405601Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6449782Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:55:13.6477604Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6515206Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:55:13.6542842Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6588169Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:55:13.6615809Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6655218Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:55:13.6684225Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6722365Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:55:13.6750434Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6795618Z Entering 'third_party/pocketfft'
2022-05-18T04:55:13.6824015Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6860693Z Entering 'third_party/protobuf'
2022-05-18T04:55:13.6889593Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6932113Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:55:13.6958645Z http.https://github.com/.extraheader
2022-05-18T04:55:13.6996479Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:55:13.7025031Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7068695Z Entering 'third_party/psimd'
2022-05-18T04:55:13.7097085Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7135510Z Entering 'third_party/pthreadpool'
2022-05-18T04:55:13.7164936Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7203422Z Entering 'third_party/pybind11'
2022-05-18T04:55:13.7231970Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7269428Z Entering 'third_party/python-enum'
2022-05-18T04:55:13.7297356Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7338716Z Entering 'third_party/python-peachpy'
2022-05-18T04:55:13.7368051Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7408050Z Entering 'third_party/python-six'
2022-05-18T04:55:13.7436251Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7474201Z Entering 'third_party/sleef'
2022-05-18T04:55:13.7501436Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7541519Z Entering 'third_party/tbb'
2022-05-18T04:55:13.7570118Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7613304Z Entering 'third_party/tensorpipe'
2022-05-18T04:55:13.7641499Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7679271Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:55:13.7708784Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7747348Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:55:13.7776786Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7814731Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:55:13.7843223Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7881945Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:55:13.7911452Z http.https://github.com/.extraheader
2022-05-18T04:55:13.7948683Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:55:13.7979199Z http.https://github.com/.extraheader
2022-05-18T04:55:13.8024044Z Entering 'third_party/zstd'
2022-05-18T04:55:13.8054222Z http.https://github.com/.extraheader
2022-05-18T04:55:13.8392918Z Cleaning up orphan processes